-- Logs begin at Mon 2022-05-23 11:58:30 UTC, end at Mon 2022-05-23 13:30:25 UTC. -- May 23 12:39:44.982406 dv-cinder-master-3-iscsi systemd[1]: Starting Devstack devstack@keystone.service... May 23 12:39:44.991500 dv-cinder-master-3-iscsi devstack@keystone.service[92045]: [uWSGI] getting INI configuration from /etc/keystone/keystone-uwsgi-public.ini May 23 12:39:44.993482 dv-cinder-master-3-iscsi devstack@keystone.service[92045]: *** Starting uWSGI 2.0.18-debian (64bit) on [Mon May 23 12:39:44 2022] *** May 23 12:39:44.993482 dv-cinder-master-3-iscsi devstack@keystone.service[92045]: compiled with version: 10.0.1 20200405 (experimental) [master revision 0be9efad938:fcb98e4978a:705510a708d3642c9c962beb663c476167e4e8a4] on 11 April 2020 11:15:55 May 23 12:39:44.993482 dv-cinder-master-3-iscsi devstack@keystone.service[92045]: os: Linux-5.4.0-72-generic #80-Ubuntu SMP Mon Apr 12 17:35:00 UTC 2021 May 23 12:39:44.993482 dv-cinder-master-3-iscsi devstack@keystone.service[92045]: nodename: dv-cinder-master-3-iscsi May 23 12:39:44.993482 dv-cinder-master-3-iscsi devstack@keystone.service[92045]: machine: x86_64 May 23 12:39:44.993482 dv-cinder-master-3-iscsi devstack@keystone.service[92045]: clock source: unix May 23 12:39:44.993482 dv-cinder-master-3-iscsi devstack@keystone.service[92045]: pcre jit disabled May 23 12:39:44.993482 dv-cinder-master-3-iscsi devstack@keystone.service[92045]: detected number of CPU cores: 4 May 23 12:39:44.993482 dv-cinder-master-3-iscsi devstack@keystone.service[92045]: current working directory: / May 23 12:39:44.993482 dv-cinder-master-3-iscsi devstack@keystone.service[92045]: detected binary path: /usr/bin/uwsgi-core May 23 12:39:44.993482 dv-cinder-master-3-iscsi devstack@keystone.service[92045]: your processes number limit is 47680 May 23 12:39:44.993482 dv-cinder-master-3-iscsi devstack@keystone.service[92045]: your memory page size is 4096 bytes May 23 12:39:44.993482 dv-cinder-master-3-iscsi devstack@keystone.service[92045]: detected max file descriptor number: 2048 May 23 12:39:44.993482 dv-cinder-master-3-iscsi devstack@keystone.service[92045]: lock engine: pthread robust mutexes May 23 12:39:44.994426 dv-cinder-master-3-iscsi devstack@keystone.service[92045]: thunder lock: enabled May 23 12:39:44.994426 dv-cinder-master-3-iscsi devstack@keystone.service[92045]: uwsgi socket 0 bound to UNIX address /var/run/uwsgi/keystone-wsgi-public.socket fd 4 May 23 12:39:44.994426 dv-cinder-master-3-iscsi devstack@keystone.service[92045]: Python version: 3.8.10 (default, Mar 15 2022, 12:22:08) [GCC 9.4.0] May 23 12:39:45.038745 dv-cinder-master-3-iscsi devstack@keystone.service[92045]: Python main interpreter initialized at 0x558e29d869c0 May 23 12:39:45.038745 dv-cinder-master-3-iscsi devstack@keystone.service[92045]: python threads support enabled May 23 12:39:45.038745 dv-cinder-master-3-iscsi devstack@keystone.service[92045]: your server socket listen backlog is limited to 100 connections May 23 12:39:45.038745 dv-cinder-master-3-iscsi devstack@keystone.service[92045]: your mercy for graceful operations on workers is 90 seconds May 23 12:39:45.039059 dv-cinder-master-3-iscsi devstack@keystone.service[92045]: mapped 403077 bytes (393 KB) for 2 cores May 23 12:39:45.039158 dv-cinder-master-3-iscsi devstack@keystone.service[92045]: *** Operational MODE: preforking *** May 23 12:39:45.039440 dv-cinder-master-3-iscsi devstack@keystone.service[92045]: *** uWSGI is running in multiple interpreter mode *** May 23 12:39:45.039440 dv-cinder-master-3-iscsi devstack@keystone.service[92045]: spawned uWSGI master process (pid: 92045) May 23 12:39:45.039638 dv-cinder-master-3-iscsi systemd[1]: Started Devstack devstack@keystone.service. May 23 12:39:45.041692 dv-cinder-master-3-iscsi devstack@keystone.service[92045]: spawned uWSGI worker 1 (pid: 92056, cores: 1) May 23 12:39:45.041692 dv-cinder-master-3-iscsi devstack@keystone.service[92045]: spawned uWSGI worker 2 (pid: 92057, cores: 1) May 23 12:39:45.041692 dv-cinder-master-3-iscsi devstack@keystone.service[92045]: running "unix_signal:15 gracefully_kill_them_all" (master-start)... May 23 12:39:50.788084 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: /usr/lib/python3/dist-packages/secretstorage/dhcrypto.py:15: CryptographyDeprecationWarning: int_from_bytes is deprecated, use int.from_bytes instead May 23 12:39:50.788084 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: from cryptography.utils import int_from_bytes May 23 12:39:50.788882 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: /usr/lib/python3/dist-packages/secretstorage/util.py:19: CryptographyDeprecationWarning: int_from_bytes is deprecated, use int.from_bytes instead May 23 12:39:50.788882 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: from cryptography.utils import int_from_bytes May 23 12:39:50.838490 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG uwsgi [-] ******************************************************************************** {{(pid=92056) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2589}} May 23 12:39:50.838686 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG uwsgi [-] Configuration options gathered from: {{(pid=92056) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2590}} May 23 12:39:50.838882 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG uwsgi [-] command line args: [] {{(pid=92056) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2591}} May 23 12:39:50.839156 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG uwsgi [-] config files: ['/etc/keystone/keystone.conf'] {{(pid=92056) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2592}} May 23 12:39:50.839336 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG uwsgi [-] ================================================================================ {{(pid=92056) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2594}} May 23 12:39:50.839591 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG uwsgi [-] admin_token = **** {{(pid=92056) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 23 12:39:50.839842 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG uwsgi [-] config_dir = [] {{(pid=92056) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 23 12:39:50.840032 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG uwsgi [-] config_file = ['/etc/keystone/keystone.conf'] {{(pid=92056) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 23 12:39:50.840217 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG uwsgi [-] config_source = [] {{(pid=92056) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 23 12:39:50.840399 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG uwsgi [-] debug = True {{(pid=92056) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 23 12:39:50.840599 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG uwsgi [-] default_log_levels = ['amqp=WARN', 'amqplib=WARN', 'boto=WARN', 'qpid=WARN', 'sqlalchemy=WARN', 'suds=INFO', 'oslo.messaging=INFO', 'oslo_messaging=INFO', 'iso8601=WARN', 'requests.packages.urllib3.connectionpool=WARN', 'urllib3.connectionpool=WARN', 'websocket=WARN', 'requests.packages.urllib3.util.retry=WARN', 'urllib3.util.retry=WARN', 'keystonemiddleware=WARN', 'routes.middleware=WARN', 'stevedore=WARN', 'taskflow=WARN', 'keystoneauth=WARN', 'oslo.cache=INFO', 'oslo_policy=INFO', 'dogpile.core.dogpile=INFO', 'dogpile=INFO', 'routes=INFO'] {{(pid=92056) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 23 12:39:50.841074 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG uwsgi [-] default_publisher_id = None {{(pid=92056) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 23 12:39:50.841166 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG uwsgi [-] insecure_debug = False {{(pid=92056) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 23 12:39:50.841375 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG uwsgi [-] instance_format = [instance: %(uuid)s] {{(pid=92056) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 23 12:39:50.841628 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG uwsgi [-] instance_uuid_format = [instance: %(uuid)s] {{(pid=92056) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 23 12:39:50.841861 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG uwsgi [-] list_limit = None {{(pid=92056) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 23 12:39:50.842609 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG uwsgi [-] log_config_append = None {{(pid=92056) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 23 12:39:50.842946 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG uwsgi [-] log_date_format = %Y-%m-%d %H:%M:%S {{(pid=92056) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 23 12:39:50.843081 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG uwsgi [-] log_dir = None {{(pid=92056) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 23 12:39:50.843263 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG uwsgi [-] log_file = None {{(pid=92056) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 23 12:39:50.843504 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG uwsgi [-] log_rotate_interval = 1 {{(pid=92056) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 23 12:39:50.843756 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG uwsgi [-] log_rotate_interval_type = days {{(pid=92056) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 23 12:39:50.844004 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG uwsgi [-] log_rotation_type = none {{(pid=92056) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 23 12:39:50.844254 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG uwsgi [-] logging_context_format_string = %(color)s%(levelname)s %(name)s [%(global_request_id)s %(request_id)s %(project_name)s %(user_name)s%(color)s] %(instance)s%(color)s%(message)s {{(pid=92056) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 23 12:39:50.844437 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG uwsgi [-] logging_debug_format_suffix = {{(pid=%(process)d) %(funcName)s %(pathname)s:%(lineno)d}} {{(pid=92056) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 23 12:39:50.844617 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG uwsgi [-] logging_default_format_string = %(color)s%(levelname)s %(name)s [-%(color)s] %(instance)s%(color)s%(message)s {{(pid=92056) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 23 12:39:50.844880 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG uwsgi [-] logging_exception_prefix = ERROR %(name)s %(instance)s {{(pid=92056) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 23 12:39:50.845112 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG uwsgi [-] logging_user_identity_format = %(user)s %(project)s %(domain)s %(system_scope)s %(user_domain)s %(project_domain)s {{(pid=92056) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 23 12:39:50.845349 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG uwsgi [-] max_logfile_count = 30 {{(pid=92056) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 23 12:39:50.845587 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG uwsgi [-] max_logfile_size_mb = 200 {{(pid=92056) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 23 12:39:50.845821 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG uwsgi [-] max_param_size = 64 {{(pid=92056) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 23 12:39:50.846098 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG uwsgi [-] max_project_tree_depth = 5 {{(pid=92056) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 23 12:39:50.846347 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG uwsgi [-] max_token_size = 16384 {{(pid=92056) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 23 12:39:50.846598 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG uwsgi [-] notification_format = cadf {{(pid=92056) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 23 12:39:50.846878 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG uwsgi [-] notification_opt_out = ['identity.authenticate.success', 'identity.authenticate.pending', 'identity.authenticate.failed'] {{(pid=92056) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 23 12:39:50.847363 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG uwsgi [-] public_endpoint = http://10.222.0.30/identity {{(pid=92056) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 23 12:39:50.847548 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG uwsgi [-] publish_errors = False {{(pid=92056) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 23 12:39:50.847782 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG uwsgi [-] pydev_debug_host = None {{(pid=92056) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 23 12:39:50.848014 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG uwsgi [-] pydev_debug_port = None {{(pid=92056) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 23 12:39:50.848194 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG uwsgi [-] rate_limit_burst = 0 {{(pid=92056) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 23 12:39:50.848443 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG uwsgi [-] rate_limit_except_level = CRITICAL {{(pid=92056) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 23 12:39:50.848684 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG uwsgi [-] rate_limit_interval = 0 {{(pid=92056) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 23 12:39:50.848919 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG uwsgi [-] standard_threads = False {{(pid=92056) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 23 12:39:50.849153 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG uwsgi [-] strict_password_check = False {{(pid=92056) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 23 12:39:50.849395 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG uwsgi [-] syslog_log_facility = LOG_USER {{(pid=92056) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 23 12:39:50.849574 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG uwsgi [-] use_eventlog = False {{(pid=92056) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 23 12:39:50.849752 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG uwsgi [-] use_journal = False {{(pid=92056) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 23 12:39:50.849961 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG uwsgi [-] use_json = False {{(pid=92056) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 23 12:39:50.850142 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG uwsgi [-] use_stderr = False {{(pid=92056) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 23 12:39:50.850320 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG uwsgi [-] use_syslog = False {{(pid=92056) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 23 12:39:50.850552 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG uwsgi [-] watch_log_file = False {{(pid=92056) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 23 12:39:50.850809 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG uwsgi [-] oslo_concurrency.disable_process_locking = False {{(pid=92056) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:50.851050 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG uwsgi [-] oslo_concurrency.lock_path = None {{(pid=92056) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:50.851309 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG uwsgi [-] profiler.connection_string = messaging:// {{(pid=92056) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:50.851549 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG uwsgi [-] profiler.enabled = False {{(pid=92056) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:50.851799 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG uwsgi [-] profiler.es_doc_type = notification {{(pid=92056) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:50.852037 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG uwsgi [-] profiler.es_scroll_size = 10000 {{(pid=92056) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:50.852286 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG uwsgi [-] profiler.es_scroll_time = 2m {{(pid=92056) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:50.852524 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG uwsgi [-] profiler.filter_error_trace = False {{(pid=92056) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:50.852774 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG uwsgi [-] profiler.hmac_keys = SECRET_KEY {{(pid=92056) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:50.853021 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG uwsgi [-] profiler.sentinel_service_name = mymaster {{(pid=92056) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:50.853274 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG uwsgi [-] profiler.socket_timeout = 0.1 {{(pid=92056) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:50.853509 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG uwsgi [-] profiler.trace_sqlalchemy = False {{(pid=92056) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:50.853757 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_buffer_size = 1000 {{(pid=92056) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:50.854023 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_enabled = False {{(pid=92056) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:50.854273 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_process_name = {{(pid=92056) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:50.854522 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_socket_file = /var/tmp/metrics_collector.sock {{(pid=92056) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:50.854761 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_thread_stop_timeout = 10 {{(pid=92056) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:50.855016 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG uwsgi [-] oslo_middleware.http_basic_auth_user_file = /etc/htpasswd {{(pid=92056) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:50.855271 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG uwsgi [-] audit.namespace = openstack {{(pid=92056) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:50.855517 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG uwsgi [-] oslo_policy.enforce_new_defaults = False {{(pid=92056) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:50.855756 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG uwsgi [-] oslo_policy.enforce_scope = False {{(pid=92056) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:50.856008 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG uwsgi [-] oslo_policy.policy_default_rule = default {{(pid=92056) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:50.856270 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG uwsgi [-] oslo_policy.policy_dirs = ['policy.d'] {{(pid=92056) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:50.856524 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG uwsgi [-] oslo_policy.policy_file = policy.yaml {{(pid=92056) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:50.856777 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG uwsgi [-] oslo_policy.remote_content_type = application/x-www-form-urlencoded {{(pid=92056) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:50.857011 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG uwsgi [-] oslo_policy.remote_ssl_ca_crt_file = None {{(pid=92056) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:50.857244 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG uwsgi [-] oslo_policy.remote_ssl_client_crt_file = None {{(pid=92056) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:50.857475 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG uwsgi [-] oslo_policy.remote_ssl_client_key_file = None {{(pid=92056) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:50.858340 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG uwsgi [-] oslo_policy.remote_ssl_verify_server_crt = False {{(pid=92056) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:50.858600 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG uwsgi [-] keystone_authtoken.admin_password = **** {{(pid=92056) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:50.858860 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG uwsgi [-] keystone_authtoken.admin_tenant_name = admin {{(pid=92056) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:50.859097 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG uwsgi [-] keystone_authtoken.admin_token = **** {{(pid=92056) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:50.859334 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG uwsgi [-] keystone_authtoken.admin_user = None {{(pid=92056) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:50.859585 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG uwsgi [-] keystone_authtoken.auth_admin_prefix = {{(pid=92056) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:50.859835 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG uwsgi [-] keystone_authtoken.auth_host = 127.0.0.1 {{(pid=92056) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:50.860074 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG uwsgi [-] keystone_authtoken.auth_port = 35357 {{(pid=92056) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:50.860326 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG uwsgi [-] keystone_authtoken.auth_protocol = https {{(pid=92056) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:50.860562 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG uwsgi [-] keystone_authtoken.auth_section = None {{(pid=92056) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:50.860802 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG uwsgi [-] keystone_authtoken.auth_type = None {{(pid=92056) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:50.861039 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG uwsgi [-] keystone_authtoken.auth_uri = None {{(pid=92056) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:50.861272 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG uwsgi [-] keystone_authtoken.auth_version = None {{(pid=92056) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:50.861505 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG uwsgi [-] keystone_authtoken.cache = None {{(pid=92056) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:50.861739 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG uwsgi [-] keystone_authtoken.cafile = None {{(pid=92056) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:50.861995 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG uwsgi [-] keystone_authtoken.certfile = None {{(pid=92056) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:50.862237 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG uwsgi [-] keystone_authtoken.delay_auth_decision = False {{(pid=92056) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:50.862487 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG uwsgi [-] keystone_authtoken.enforce_token_bind = permissive {{(pid=92056) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:50.862721 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG uwsgi [-] keystone_authtoken.http_connect_timeout = None {{(pid=92056) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:50.862960 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG uwsgi [-] keystone_authtoken.http_request_max_retries = 3 {{(pid=92056) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:50.863193 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG uwsgi [-] keystone_authtoken.identity_uri = None {{(pid=92056) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:50.863431 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG uwsgi [-] keystone_authtoken.include_service_catalog = True {{(pid=92056) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:50.863668 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG uwsgi [-] keystone_authtoken.insecure = False {{(pid=92056) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:50.863916 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG uwsgi [-] keystone_authtoken.interface = internal {{(pid=92056) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:50.864148 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG uwsgi [-] keystone_authtoken.keyfile = None {{(pid=92056) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:50.864387 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_conn_get_timeout = 10 {{(pid=92056) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:50.864624 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_dead_retry = 300 {{(pid=92056) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:50.864861 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_maxsize = 10 {{(pid=92056) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:50.865098 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_socket_timeout = 3 {{(pid=92056) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:50.865334 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_unused_timeout = 60 {{(pid=92056) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:50.865564 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG uwsgi [-] keystone_authtoken.memcache_secret_key = **** {{(pid=92056) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:50.865811 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG uwsgi [-] keystone_authtoken.memcache_security_strategy = None {{(pid=92056) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:50.866076 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG uwsgi [-] keystone_authtoken.memcache_use_advanced_pool = True {{(pid=92056) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:50.866314 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG uwsgi [-] keystone_authtoken.memcached_servers = None {{(pid=92056) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:50.866548 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG uwsgi [-] keystone_authtoken.region_name = None {{(pid=92056) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:50.866807 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG uwsgi [-] keystone_authtoken.service_token_roles = ['service'] {{(pid=92056) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:50.867047 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG uwsgi [-] keystone_authtoken.service_token_roles_required = False {{(pid=92056) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:50.867279 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG uwsgi [-] keystone_authtoken.service_type = None {{(pid=92056) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:50.867517 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG uwsgi [-] keystone_authtoken.token_cache_time = 300 {{(pid=92056) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:50.867754 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG uwsgi [-] keystone_authtoken.www_authenticate_uri = None {{(pid=92056) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:50.868026 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG uwsgi [-] cache.backend = dogpile.cache.memcached {{(pid=92056) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:50.868266 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG uwsgi [-] cache.backend_argument = **** {{(pid=92056) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:50.868518 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG uwsgi [-] cache.config_prefix = cache.oslo {{(pid=92056) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:50.868772 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG uwsgi [-] cache.dead_timeout = 60.0 {{(pid=92056) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:50.869012 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG uwsgi [-] cache.debug_cache_backend = False {{(pid=92056) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:50.869250 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG uwsgi [-] cache.enable_retry_client = False {{(pid=92056) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:50.869488 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG uwsgi [-] cache.enable_socket_keepalive = False {{(pid=92056) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:50.869737 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG uwsgi [-] cache.enabled = True {{(pid=92056) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:50.869999 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG uwsgi [-] cache.expiration_time = 600 {{(pid=92056) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:50.870239 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG uwsgi [-] cache.hashclient_retry_attempts = 2 {{(pid=92056) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:50.870484 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG uwsgi [-] cache.hashclient_retry_delay = 1.0 {{(pid=92056) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:50.870723 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG uwsgi [-] cache.memcache_dead_retry = 300 {{(pid=92056) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:50.870959 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG uwsgi [-] cache.memcache_pool_connection_get_timeout = 10 {{(pid=92056) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:50.871195 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG uwsgi [-] cache.memcache_pool_flush_on_reconnect = False {{(pid=92056) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:50.871431 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG uwsgi [-] cache.memcache_pool_maxsize = 10 {{(pid=92056) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:50.871667 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG uwsgi [-] cache.memcache_pool_unused_timeout = 60 {{(pid=92056) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:50.871929 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG uwsgi [-] cache.memcache_servers = ['localhost:11211'] {{(pid=92056) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:50.872171 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG uwsgi [-] cache.memcache_socket_timeout = 1.0 {{(pid=92056) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:50.872412 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG uwsgi [-] cache.proxies = [] {{(pid=92056) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:50.872651 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG uwsgi [-] cache.retry_attempts = 2 {{(pid=92056) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:50.872894 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG uwsgi [-] cache.retry_delay = 0.0 {{(pid=92056) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:50.873131 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG uwsgi [-] cache.socket_keepalive_count = 1 {{(pid=92056) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:50.873408 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG uwsgi [-] cache.socket_keepalive_idle = 1 {{(pid=92056) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:50.873644 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG uwsgi [-] cache.socket_keepalive_interval = 1 {{(pid=92056) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:50.873892 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG uwsgi [-] cache.tls_allowed_ciphers = None {{(pid=92056) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:50.874130 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG uwsgi [-] cache.tls_cafile = None {{(pid=92056) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:50.874362 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG uwsgi [-] cache.tls_certfile = None {{(pid=92056) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:50.874599 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG uwsgi [-] cache.tls_enabled = False {{(pid=92056) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:50.874830 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG uwsgi [-] cache.tls_keyfile = None {{(pid=92056) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:50.875071 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG uwsgi [-] application_credential.cache_time = None {{(pid=92056) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:50.875309 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG uwsgi [-] application_credential.caching = True {{(pid=92056) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:50.875557 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG uwsgi [-] application_credential.driver = sql {{(pid=92056) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:50.875793 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG uwsgi [-] application_credential.user_limit = -1 {{(pid=92056) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:50.876049 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG uwsgi [-] assignment.driver = sql {{(pid=92056) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:50.876310 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG uwsgi [-] assignment.prohibited_implied_role = ['admin'] {{(pid=92056) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:50.876553 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG uwsgi [-] auth.application_credential = None {{(pid=92056) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:50.876784 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG uwsgi [-] auth.external = None {{(pid=92056) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:50.877017 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG uwsgi [-] auth.mapped = None {{(pid=92056) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:50.877311 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG uwsgi [-] auth.methods = ['external', 'password', 'token', 'oauth1', 'mapped', 'application_credential'] {{(pid=92056) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:50.877546 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG uwsgi [-] auth.oauth1 = None {{(pid=92056) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:50.877778 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG uwsgi [-] auth.password = None {{(pid=92056) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:50.878033 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG uwsgi [-] auth.token = None {{(pid=92056) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:50.878273 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG uwsgi [-] catalog.cache_time = None {{(pid=92056) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:50.878510 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG uwsgi [-] catalog.caching = True {{(pid=92056) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:50.878762 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG uwsgi [-] catalog.driver = sql {{(pid=92056) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:50.878993 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG uwsgi [-] catalog.list_limit = None {{(pid=92056) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:50.879241 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG uwsgi [-] catalog.template_file = default_catalog.templates {{(pid=92056) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:50.879486 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG uwsgi [-] credential.auth_ttl = 15 {{(pid=92056) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:50.879719 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG uwsgi [-] credential.cache_time = None {{(pid=92056) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:50.879962 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG uwsgi [-] credential.caching = True {{(pid=92056) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:50.880210 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG uwsgi [-] credential.driver = sql {{(pid=92056) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:50.880462 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG uwsgi [-] credential.key_repository = /etc/keystone/credential-keys/ {{(pid=92056) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:50.880749 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG uwsgi [-] credential.provider = fernet {{(pid=92056) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:50.880951 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG uwsgi [-] credential.user_limit = -1 {{(pid=92056) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:50.881193 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG uwsgi [-] domain_config.cache_time = 300 {{(pid=92056) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:50.881432 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG uwsgi [-] domain_config.caching = True {{(pid=92056) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:50.881686 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG uwsgi [-] domain_config.driver = sql {{(pid=92056) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:50.881964 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG uwsgi [-] endpoint_filter.driver = sql {{(pid=92056) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:50.882208 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG uwsgi [-] endpoint_filter.return_all_endpoints_if_no_filter = True {{(pid=92056) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:50.882464 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG uwsgi [-] endpoint_policy.driver = sql {{(pid=92056) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:50.882749 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG uwsgi [-] eventlet_server.admin_bind_host = 0.0.0.0 {{(pid=92056) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:50.882996 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG uwsgi [-] eventlet_server.admin_port = 35357 {{(pid=92056) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:50.883261 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG uwsgi [-] eventlet_server.public_bind_host = 0.0.0.0 {{(pid=92056) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:50.883508 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG uwsgi [-] eventlet_server.public_port = 5000 {{(pid=92056) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:50.883767 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG uwsgi [-] federation.assertion_prefix = {{(pid=92056) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:50.884006 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG uwsgi [-] federation.caching = True {{(pid=92056) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:50.884248 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG uwsgi [-] federation.default_authorization_ttl = 0 {{(pid=92056) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:50.884498 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG uwsgi [-] federation.driver = sql {{(pid=92056) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:50.884747 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG uwsgi [-] federation.federated_domain_name = Federated {{(pid=92056) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:50.884980 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG uwsgi [-] federation.remote_id_attribute = None {{(pid=92056) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:50.885230 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG uwsgi [-] federation.sso_callback_template = /etc/keystone/sso_callback_template.html {{(pid=92056) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:50.885470 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG uwsgi [-] federation.trusted_dashboard = [] {{(pid=92056) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:50.885724 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG uwsgi [-] fernet_receipts.key_repository = /etc/keystone/fernet-keys/ {{(pid=92056) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:50.885987 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG uwsgi [-] fernet_receipts.max_active_keys = 3 {{(pid=92056) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:50.886244 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG uwsgi [-] fernet_tokens.key_repository = /etc/keystone/fernet-keys/ {{(pid=92056) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:50.886482 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG uwsgi [-] fernet_tokens.max_active_keys = 3 {{(pid=92056) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:50.886729 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG uwsgi [-] identity.cache_time = 600 {{(pid=92056) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:50.886966 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG uwsgi [-] identity.caching = True {{(pid=92056) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:50.887215 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG uwsgi [-] identity.default_domain_id = default {{(pid=92056) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:50.887466 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG uwsgi [-] identity.domain_config_dir = /etc/keystone/domains {{(pid=92056) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:50.887705 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG uwsgi [-] identity.domain_configurations_from_database = False {{(pid=92056) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:50.887944 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG uwsgi [-] identity.domain_specific_drivers_enabled = False {{(pid=92056) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:50.888193 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG uwsgi [-] identity.driver = sql {{(pid=92056) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:50.888424 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG uwsgi [-] identity.list_limit = None {{(pid=92056) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:50.888668 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG uwsgi [-] identity.max_password_length = 4096 {{(pid=92056) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:50.889491 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG uwsgi [-] identity.password_hash_algorithm = bcrypt {{(pid=92056) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:50.889794 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG uwsgi [-] identity.password_hash_rounds = 4 {{(pid=92056) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:50.890055 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG uwsgi [-] identity.salt_bytesize = None {{(pid=92056) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:50.890291 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG uwsgi [-] identity.scrypt_block_size = None {{(pid=92056) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:50.890527 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG uwsgi [-] identity.scrypt_parallelism = None {{(pid=92056) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:50.890775 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG uwsgi [-] identity_mapping.backward_compatible_ids = True {{(pid=92056) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:50.891024 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG uwsgi [-] identity_mapping.driver = sql {{(pid=92056) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:50.891273 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG uwsgi [-] identity_mapping.generator = sha256 {{(pid=92056) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:50.891531 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG uwsgi [-] jwt_tokens.jws_private_key_repository = /etc/keystone/jws-keys/private {{(pid=92056) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:50.891780 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG uwsgi [-] jwt_tokens.jws_public_key_repository = /etc/keystone/jws-keys/public {{(pid=92056) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:50.892063 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG uwsgi [-] ldap.alias_dereferencing = default {{(pid=92056) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:50.892300 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG uwsgi [-] ldap.auth_pool_connection_lifetime = 60 {{(pid=92056) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:50.892538 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG uwsgi [-] ldap.auth_pool_size = 100 {{(pid=92056) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:50.892771 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG uwsgi [-] ldap.chase_referrals = None {{(pid=92056) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:50.893009 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG uwsgi [-] ldap.connection_timeout = -1 {{(pid=92056) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:50.893241 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG uwsgi [-] ldap.debug_level = None {{(pid=92056) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:50.893477 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG uwsgi [-] ldap.group_ad_nesting = False {{(pid=92056) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:50.893720 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG uwsgi [-] ldap.group_additional_attribute_mapping = [] {{(pid=92056) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:50.893987 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG uwsgi [-] ldap.group_attribute_ignore = [] {{(pid=92056) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:50.894237 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG uwsgi [-] ldap.group_desc_attribute = description {{(pid=92056) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:50.894469 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG uwsgi [-] ldap.group_filter = None {{(pid=92056) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:50.894720 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG uwsgi [-] ldap.group_id_attribute = cn {{(pid=92056) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:50.894968 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG uwsgi [-] ldap.group_member_attribute = member {{(pid=92056) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:50.895205 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG uwsgi [-] ldap.group_members_are_ids = False {{(pid=92056) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:50.895452 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG uwsgi [-] ldap.group_name_attribute = ou {{(pid=92056) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:50.895699 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG uwsgi [-] ldap.group_objectclass = groupOfNames {{(pid=92056) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:50.895998 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG uwsgi [-] ldap.group_tree_dn = None {{(pid=92056) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:50.896258 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG uwsgi [-] ldap.page_size = 0 {{(pid=92056) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:50.896490 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG uwsgi [-] ldap.password = **** {{(pid=92056) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:50.896730 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG uwsgi [-] ldap.pool_connection_lifetime = 600 {{(pid=92056) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:50.896970 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG uwsgi [-] ldap.pool_connection_timeout = -1 {{(pid=92056) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:50.897217 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG uwsgi [-] ldap.pool_retry_delay = 0.1 {{(pid=92056) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:50.897459 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG uwsgi [-] ldap.pool_retry_max = 3 {{(pid=92056) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:50.897697 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG uwsgi [-] ldap.pool_size = 10 {{(pid=92056) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:50.897975 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG uwsgi [-] ldap.query_scope = one {{(pid=92056) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:50.898230 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG uwsgi [-] ldap.suffix = cn=example,cn=com {{(pid=92056) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:50.898465 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG uwsgi [-] ldap.tls_cacertdir = None {{(pid=92056) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:50.898699 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG uwsgi [-] ldap.tls_cacertfile = None {{(pid=92056) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:50.898967 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG uwsgi [-] ldap.tls_req_cert = demand {{(pid=92056) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:50.899216 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG uwsgi [-] ldap.url = ldap://localhost {{(pid=92056) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:50.899456 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG uwsgi [-] ldap.use_auth_pool = True {{(pid=92056) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:50.899696 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG uwsgi [-] ldap.use_pool = True {{(pid=92056) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:50.899934 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG uwsgi [-] ldap.use_tls = False {{(pid=92056) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:50.900167 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG uwsgi [-] ldap.user = None {{(pid=92056) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:50.900409 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG uwsgi [-] ldap.user_additional_attribute_mapping = [] {{(pid=92056) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:50.900685 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG uwsgi [-] ldap.user_attribute_ignore = ['default_project_id'] {{(pid=92056) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:50.901077 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG uwsgi [-] ldap.user_default_project_id_attribute = None {{(pid=92056) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:50.901523 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG uwsgi [-] ldap.user_description_attribute = description {{(pid=92056) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:50.901996 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG uwsgi [-] ldap.user_enabled_attribute = enabled {{(pid=92056) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:50.902441 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG uwsgi [-] ldap.user_enabled_default = True {{(pid=92056) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:50.902868 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG uwsgi [-] ldap.user_enabled_emulation = False {{(pid=92056) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:50.903282 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG uwsgi [-] ldap.user_enabled_emulation_dn = None {{(pid=92056) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:50.903705 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG uwsgi [-] ldap.user_enabled_emulation_use_group_config = False {{(pid=92056) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:50.904128 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG uwsgi [-] ldap.user_enabled_invert = False {{(pid=92056) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:50.905139 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG uwsgi [-] ldap.user_enabled_mask = 0 {{(pid=92056) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:50.905541 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG uwsgi [-] ldap.user_filter = None {{(pid=92056) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:50.906289 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG uwsgi [-] ldap.user_id_attribute = cn {{(pid=92056) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:50.906743 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG uwsgi [-] ldap.user_mail_attribute = mail {{(pid=92056) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:50.907165 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG uwsgi [-] ldap.user_name_attribute = sn {{(pid=92056) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:50.907499 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG uwsgi [-] ldap.user_objectclass = inetOrgPerson {{(pid=92056) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:50.907762 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG uwsgi [-] ldap.user_pass_attribute = userPassword {{(pid=92056) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:50.908003 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG uwsgi [-] ldap.user_tree_dn = None {{(pid=92056) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:50.908262 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG uwsgi [-] memcache.dead_retry = 300 {{(pid=92056) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:50.908505 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG uwsgi [-] memcache.pool_connection_get_timeout = 10 {{(pid=92056) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:50.908747 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG uwsgi [-] memcache.pool_maxsize = 10 {{(pid=92056) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:50.908995 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG uwsgi [-] memcache.pool_unused_timeout = 60 {{(pid=92056) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:50.909235 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG uwsgi [-] memcache.socket_timeout = 3 {{(pid=92056) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:50.909487 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG uwsgi [-] oauth1.access_token_duration = 86400 {{(pid=92056) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:50.909738 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG uwsgi [-] oauth1.driver = sql {{(pid=92056) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:50.910007 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG uwsgi [-] oauth1.request_token_duration = 28800 {{(pid=92056) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:50.910269 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG uwsgi [-] policy.driver = sql {{(pid=92056) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:50.910504 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG uwsgi [-] policy.list_limit = None {{(pid=92056) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:50.910753 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG uwsgi [-] receipt.cache_on_issue = True {{(pid=92056) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:50.910992 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG uwsgi [-] receipt.cache_time = 300 {{(pid=92056) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:50.911229 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG uwsgi [-] receipt.caching = True {{(pid=92056) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:50.911469 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG uwsgi [-] receipt.expiration = 300 {{(pid=92056) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:50.911719 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG uwsgi [-] receipt.provider = fernet {{(pid=92056) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:50.911964 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG uwsgi [-] resource.admin_project_domain_name = None {{(pid=92056) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:50.912199 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG uwsgi [-] resource.admin_project_name = None {{(pid=92056) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:50.912439 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG uwsgi [-] resource.cache_time = None {{(pid=92056) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:50.912682 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG uwsgi [-] resource.caching = True {{(pid=92056) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:50.913050 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG uwsgi [-] resource.domain_name_url_safe = off {{(pid=92056) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:50.913307 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG uwsgi [-] resource.driver = sql {{(pid=92056) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:50.913547 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG uwsgi [-] resource.list_limit = None {{(pid=92056) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:50.913800 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG uwsgi [-] resource.project_name_url_safe = off {{(pid=92056) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:50.914084 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG uwsgi [-] revoke.cache_time = 3600 {{(pid=92056) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:50.914324 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG uwsgi [-] revoke.caching = True {{(pid=92056) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:50.914574 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG uwsgi [-] revoke.driver = sql {{(pid=92056) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:50.914815 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG uwsgi [-] revoke.expiration_buffer = 1800 {{(pid=92056) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:50.915058 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG uwsgi [-] role.cache_time = None {{(pid=92056) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:50.915298 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG uwsgi [-] role.caching = True {{(pid=92056) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:50.915551 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG uwsgi [-] role.driver = sql {{(pid=92056) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:50.915786 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG uwsgi [-] role.list_limit = None {{(pid=92056) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:50.916040 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG uwsgi [-] saml.assertion_expiration_time = 3600 {{(pid=92056) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:50.916293 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG uwsgi [-] saml.certfile = /etc/keystone/ssl/certs/signing_cert.pem {{(pid=92056) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:50.916546 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG uwsgi [-] saml.idp_contact_company = Example, Inc. {{(pid=92056) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:50.916799 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG uwsgi [-] saml.idp_contact_email = support@example.com {{(pid=92056) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:50.917052 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG uwsgi [-] saml.idp_contact_name = SAML Identity Provider Support {{(pid=92056) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:50.917305 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG uwsgi [-] saml.idp_contact_surname = Support {{(pid=92056) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:50.917557 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG uwsgi [-] saml.idp_contact_telephone = +1 800 555 0100 {{(pid=92056) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:50.917809 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG uwsgi [-] saml.idp_contact_type = other {{(pid=92056) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:50.918072 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG uwsgi [-] saml.idp_entity_id = None {{(pid=92056) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:50.918324 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG uwsgi [-] saml.idp_lang = en {{(pid=92056) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:50.918576 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG uwsgi [-] saml.idp_metadata_path = /etc/keystone/saml2_idp_metadata.xml {{(pid=92056) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:50.918826 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG uwsgi [-] saml.idp_organization_display_name = OpenStack SAML Identity Provider {{(pid=92056) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:50.919083 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG uwsgi [-] saml.idp_organization_name = SAML Identity Provider {{(pid=92056) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:50.919511 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG uwsgi [-] saml.idp_organization_url = https://example.com/ {{(pid=92056) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:50.919751 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG uwsgi [-] saml.idp_sso_endpoint = None {{(pid=92056) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:50.920005 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG uwsgi [-] saml.keyfile = /etc/keystone/ssl/private/signing_key.pem {{(pid=92056) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:50.920300 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG uwsgi [-] saml.relay_state_prefix = ss:mem: {{(pid=92056) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:50.920636 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG uwsgi [-] saml.xmlsec1_binary = xmlsec1 {{(pid=92056) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:50.920895 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG uwsgi [-] security_compliance.change_password_upon_first_use = False {{(pid=92056) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:50.921132 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG uwsgi [-] security_compliance.disable_user_account_days_inactive = None {{(pid=92056) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:50.921390 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG uwsgi [-] security_compliance.lockout_duration = 10 {{(pid=92056) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:50.921646 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG uwsgi [-] security_compliance.lockout_failure_attempts = 2 {{(pid=92056) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:50.921916 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG uwsgi [-] security_compliance.minimum_password_age = 0 {{(pid=92056) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:50.922147 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG uwsgi [-] security_compliance.password_expires_days = None {{(pid=92056) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:50.922394 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG uwsgi [-] security_compliance.password_regex = None {{(pid=92056) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:50.922629 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG uwsgi [-] security_compliance.password_regex_description = None {{(pid=92056) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:50.922882 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG uwsgi [-] security_compliance.unique_last_password_count = 2 {{(pid=92056) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:50.923139 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG uwsgi [-] shadow_users.driver = sql {{(pid=92056) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:50.923390 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG uwsgi [-] token.allow_expired_window = 172800 {{(pid=92056) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:50.923631 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG uwsgi [-] token.allow_rescope_scoped_token = True {{(pid=92056) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:50.923882 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG uwsgi [-] token.cache_on_issue = True {{(pid=92056) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:50.924118 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG uwsgi [-] token.cache_time = None {{(pid=92056) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:50.924358 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG uwsgi [-] token.caching = True {{(pid=92056) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:50.924598 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG uwsgi [-] token.expiration = 3600 {{(pid=92056) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:50.924848 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG uwsgi [-] token.provider = fernet {{(pid=92056) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:50.925088 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG uwsgi [-] token.revoke_by_id = True {{(pid=92056) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:50.925347 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG uwsgi [-] tokenless_auth.issuer_attribute = SSL_CLIENT_I_DN {{(pid=92056) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:50.925598 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG uwsgi [-] tokenless_auth.protocol = x509 {{(pid=92056) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:50.925840 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG uwsgi [-] tokenless_auth.trusted_issuer = [] {{(pid=92056) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:50.926109 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG uwsgi [-] totp.included_previous_windows = 1 {{(pid=92056) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:50.926356 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG uwsgi [-] trust.allow_redelegation = False {{(pid=92056) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:50.926605 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG uwsgi [-] trust.driver = sql {{(pid=92056) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:50.926843 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG uwsgi [-] trust.max_redelegation_count = 3 {{(pid=92056) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:50.927086 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG uwsgi [-] unified_limit.cache_time = None {{(pid=92056) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:50.927324 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG uwsgi [-] unified_limit.caching = True {{(pid=92056) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:50.927570 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG uwsgi [-] unified_limit.driver = sql {{(pid=92056) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:50.927821 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG uwsgi [-] unified_limit.enforcement_model = flat {{(pid=92056) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:50.928054 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG uwsgi [-] unified_limit.list_limit = None {{(pid=92056) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:50.928300 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG uwsgi [-] wsgi.debug_middleware = False {{(pid=92056) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:50.928544 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: /usr/lib/python3/dist-packages/secretstorage/dhcrypto.py:15: CryptographyDeprecationWarning: int_from_bytes is deprecated, use int.from_bytes instead May 23 12:39:50.928544 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: from cryptography.utils import int_from_bytes May 23 12:39:50.929156 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG uwsgi [-] database.backend = sqlalchemy {{(pid=92056) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:50.929156 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG uwsgi [-] database.connection = **** {{(pid=92056) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:50.929477 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: /usr/lib/python3/dist-packages/secretstorage/util.py:19: CryptographyDeprecationWarning: int_from_bytes is deprecated, use int.from_bytes instead May 23 12:39:50.929477 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: from cryptography.utils import int_from_bytes May 23 12:39:50.929727 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG uwsgi [-] database.connection_debug = 0 {{(pid=92056) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:50.929727 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG uwsgi [-] database.connection_parameters = {{(pid=92056) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:50.929727 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG uwsgi [-] database.connection_recycle_time = 3600 {{(pid=92056) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:50.930106 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG uwsgi [-] database.connection_trace = False {{(pid=92056) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:50.930106 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG uwsgi [-] database.db_inc_retry_interval = True {{(pid=92056) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:50.930352 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG uwsgi [-] database.db_max_retries = 20 {{(pid=92056) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:50.930476 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG uwsgi [-] database.db_max_retry_interval = 10 {{(pid=92056) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:50.930710 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG uwsgi [-] database.db_retry_interval = 1 {{(pid=92056) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:50.930958 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG uwsgi [-] database.max_overflow = 50 {{(pid=92056) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:50.931197 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG uwsgi [-] database.max_pool_size = 5 {{(pid=92056) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:50.931443 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG uwsgi [-] database.max_retries = 10 {{(pid=92056) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:50.931680 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG uwsgi [-] database.mysql_enable_ndb = False {{(pid=92056) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:50.931929 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG uwsgi [-] database.mysql_sql_mode = TRADITIONAL {{(pid=92056) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:50.932168 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG uwsgi [-] database.pool_timeout = None {{(pid=92056) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:50.932413 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG uwsgi [-] database.retry_interval = 10 {{(pid=92056) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:50.932645 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG uwsgi [-] database.slave_connection = **** {{(pid=92056) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:50.932887 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG uwsgi [-] database.sqlite_synchronous = True {{(pid=92056) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:50.933123 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG uwsgi [-] database.use_db_reconnect = False {{(pid=92056) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:50.933301 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG uwsgi [-] ******************************************************************************** {{(pid=92056) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2613}} May 23 12:39:50.964762 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [-] Callback: `keystone.application_credential.core.Manager._delete_app_creds_on_user_delete_callback` subscribed to event `identity.user.deleted`. {{(pid=92056) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} May 23 12:39:50.964914 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [-] Callback: `keystone.application_credential.core.Manager._delete_app_creds_on_user_delete_callback` subscribed to event `identity.user.disabled`. {{(pid=92056) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} May 23 12:39:50.965145 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [-] Callback: `keystone.application_credential.core.Manager._delete_app_creds_on_assignment_removal` subscribed to event `identity.remove_application_credentials_for_user.internal`. {{(pid=92056) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} May 23 12:39:50.974446 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [-] Callback: `keystone.assignment.core.Manager._delete_domain_assignments` subscribed to event `identity.domain.deleted`. {{(pid=92056) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} May 23 12:39:50.978650 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG uwsgi [-] ******************************************************************************** {{(pid=92057) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2589}} May 23 12:39:50.978899 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG uwsgi [-] Configuration options gathered from: {{(pid=92057) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2590}} May 23 12:39:50.979121 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG uwsgi [-] command line args: [] {{(pid=92057) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2591}} May 23 12:39:50.979408 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG uwsgi [-] config files: ['/etc/keystone/keystone.conf'] {{(pid=92057) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2592}} May 23 12:39:50.979588 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG uwsgi [-] ================================================================================ {{(pid=92057) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2594}} May 23 12:39:50.979843 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG uwsgi [-] admin_token = **** {{(pid=92057) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 23 12:39:50.980090 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG uwsgi [-] config_dir = [] {{(pid=92057) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 23 12:39:50.980277 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG uwsgi [-] config_file = ['/etc/keystone/keystone.conf'] {{(pid=92057) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 23 12:39:50.980458 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG uwsgi [-] config_source = [] {{(pid=92057) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 23 12:39:50.980639 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG uwsgi [-] debug = True {{(pid=92057) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 23 12:39:50.980855 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG uwsgi [-] default_log_levels = ['amqp=WARN', 'amqplib=WARN', 'boto=WARN', 'qpid=WARN', 'sqlalchemy=WARN', 'suds=INFO', 'oslo.messaging=INFO', 'oslo_messaging=INFO', 'iso8601=WARN', 'requests.packages.urllib3.connectionpool=WARN', 'urllib3.connectionpool=WARN', 'websocket=WARN', 'requests.packages.urllib3.util.retry=WARN', 'urllib3.util.retry=WARN', 'keystonemiddleware=WARN', 'routes.middleware=WARN', 'stevedore=WARN', 'taskflow=WARN', 'keystoneauth=WARN', 'oslo.cache=INFO', 'oslo_policy=INFO', 'dogpile.core.dogpile=INFO', 'dogpile=INFO', 'routes=INFO'] {{(pid=92057) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 23 12:39:50.981145 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG uwsgi [-] default_publisher_id = None {{(pid=92057) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 23 12:39:50.981323 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG uwsgi [-] insecure_debug = False {{(pid=92057) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 23 12:39:50.981626 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG uwsgi [-] instance_format = [instance: %(uuid)s] {{(pid=92057) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 23 12:39:50.981941 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG uwsgi [-] instance_uuid_format = [instance: %(uuid)s] {{(pid=92057) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 23 12:39:50.982176 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG uwsgi [-] list_limit = None {{(pid=92057) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 23 12:39:50.982355 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG uwsgi [-] log_config_append = None {{(pid=92057) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 23 12:39:50.982531 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG uwsgi [-] log_date_format = %Y-%m-%d %H:%M:%S {{(pid=92057) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 23 12:39:50.982731 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG uwsgi [-] log_dir = None {{(pid=92057) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 23 12:39:50.982982 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG uwsgi [-] log_file = None {{(pid=92057) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 23 12:39:50.983186 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG uwsgi [-] log_rotate_interval = 1 {{(pid=92057) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 23 12:39:50.983497 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG uwsgi [-] log_rotate_interval_type = days {{(pid=92057) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 23 12:39:50.983772 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG uwsgi [-] log_rotation_type = none {{(pid=92057) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 23 12:39:50.984060 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG uwsgi [-] logging_context_format_string = %(color)s%(levelname)s %(name)s [%(global_request_id)s %(request_id)s %(project_name)s %(user_name)s%(color)s] %(instance)s%(color)s%(message)s {{(pid=92057) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 23 12:39:50.984264 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG uwsgi [-] logging_debug_format_suffix = {{(pid=%(process)d) %(funcName)s %(pathname)s:%(lineno)d}} {{(pid=92057) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 23 12:39:50.984467 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG uwsgi [-] logging_default_format_string = %(color)s%(levelname)s %(name)s [-%(color)s] %(instance)s%(color)s%(message)s {{(pid=92057) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 23 12:39:50.984714 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG uwsgi [-] logging_exception_prefix = ERROR %(name)s %(instance)s {{(pid=92057) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 23 12:39:50.984962 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG uwsgi [-] logging_user_identity_format = %(user)s %(project)s %(domain)s %(system_scope)s %(user_domain)s %(project_domain)s {{(pid=92057) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 23 12:39:50.985251 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG uwsgi [-] max_logfile_count = 30 {{(pid=92057) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 23 12:39:50.985518 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG uwsgi [-] max_logfile_size_mb = 200 {{(pid=92057) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 23 12:39:50.985760 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG uwsgi [-] max_param_size = 64 {{(pid=92057) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 23 12:39:50.986074 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG uwsgi [-] max_project_tree_depth = 5 {{(pid=92057) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 23 12:39:50.986340 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG uwsgi [-] max_token_size = 16384 {{(pid=92057) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 23 12:39:50.986592 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG uwsgi [-] notification_format = cadf {{(pid=92057) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 23 12:39:50.986874 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG uwsgi [-] notification_opt_out = ['identity.authenticate.success', 'identity.authenticate.pending', 'identity.authenticate.failed'] {{(pid=92057) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 23 12:39:50.987358 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG uwsgi [-] public_endpoint = http://10.222.0.30/identity {{(pid=92057) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 23 12:39:50.987590 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG uwsgi [-] publish_errors = False {{(pid=92057) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 23 12:39:50.987894 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG uwsgi [-] pydev_debug_host = None {{(pid=92057) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 23 12:39:50.988160 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG uwsgi [-] pydev_debug_port = None {{(pid=92057) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 23 12:39:50.988339 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG uwsgi [-] rate_limit_burst = 0 {{(pid=92057) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 23 12:39:50.988648 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG uwsgi [-] rate_limit_except_level = CRITICAL {{(pid=92057) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 23 12:39:50.988945 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG uwsgi [-] rate_limit_interval = 0 {{(pid=92057) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 23 12:39:50.989232 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG uwsgi [-] standard_threads = False {{(pid=92057) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 23 12:39:50.989529 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG uwsgi [-] strict_password_check = False {{(pid=92057) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 23 12:39:50.989812 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG uwsgi [-] syslog_log_facility = LOG_USER {{(pid=92057) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 23 12:39:50.990013 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG uwsgi [-] use_eventlog = False {{(pid=92057) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 23 12:39:50.990250 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG uwsgi [-] use_journal = False {{(pid=92057) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 23 12:39:50.990459 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG uwsgi [-] use_json = False {{(pid=92057) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 23 12:39:50.990635 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG uwsgi [-] use_stderr = False {{(pid=92057) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 23 12:39:50.990858 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG uwsgi [-] use_syslog = False {{(pid=92057) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 23 12:39:50.991152 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG uwsgi [-] watch_log_file = False {{(pid=92057) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 23 12:39:50.991430 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG uwsgi [-] oslo_concurrency.disable_process_locking = False {{(pid=92057) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:50.991669 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG uwsgi [-] oslo_concurrency.lock_path = None {{(pid=92057) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:50.991927 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG uwsgi [-] profiler.connection_string = messaging:// {{(pid=92057) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:50.992163 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG uwsgi [-] profiler.enabled = False {{(pid=92057) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:50.992410 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG uwsgi [-] profiler.es_doc_type = notification {{(pid=92057) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:50.992648 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG uwsgi [-] profiler.es_scroll_size = 10000 {{(pid=92057) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:50.992948 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG uwsgi [-] profiler.es_scroll_time = 2m {{(pid=92057) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:50.993245 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG uwsgi [-] profiler.filter_error_trace = False {{(pid=92057) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:50.993557 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG uwsgi [-] profiler.hmac_keys = SECRET_KEY {{(pid=92057) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:50.993895 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG uwsgi [-] profiler.sentinel_service_name = mymaster {{(pid=92057) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:50.994206 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG uwsgi [-] profiler.socket_timeout = 0.1 {{(pid=92057) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:50.994490 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG uwsgi [-] profiler.trace_sqlalchemy = False {{(pid=92057) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:50.994746 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_buffer_size = 1000 {{(pid=92057) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:50.995040 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_enabled = False {{(pid=92057) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:50.995323 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_process_name = {{(pid=92057) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:50.995641 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_socket_file = /var/tmp/metrics_collector.sock {{(pid=92057) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:50.995907 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_thread_stop_timeout = 10 {{(pid=92057) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:50.996166 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG uwsgi [-] oslo_middleware.http_basic_auth_user_file = /etc/htpasswd {{(pid=92057) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:50.996447 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG uwsgi [-] audit.namespace = openstack {{(pid=92057) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:50.996733 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG uwsgi [-] oslo_policy.enforce_new_defaults = False {{(pid=92057) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:50.996985 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG uwsgi [-] oslo_policy.enforce_scope = False {{(pid=92057) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:50.997302 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG uwsgi [-] oslo_policy.policy_default_rule = default {{(pid=92057) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:50.997566 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG uwsgi [-] oslo_policy.policy_dirs = ['policy.d'] {{(pid=92057) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:50.997818 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG uwsgi [-] oslo_policy.policy_file = policy.yaml {{(pid=92057) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:50.998092 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG uwsgi [-] oslo_policy.remote_content_type = application/x-www-form-urlencoded {{(pid=92057) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:50.998413 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG uwsgi [-] oslo_policy.remote_ssl_ca_crt_file = None {{(pid=92057) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:50.998674 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG uwsgi [-] oslo_policy.remote_ssl_client_crt_file = None {{(pid=92057) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:50.998959 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG uwsgi [-] oslo_policy.remote_ssl_client_key_file = None {{(pid=92057) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:50.999241 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG uwsgi [-] oslo_policy.remote_ssl_verify_server_crt = False {{(pid=92057) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:50.999491 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG uwsgi [-] keystone_authtoken.admin_password = **** {{(pid=92057) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:50.999739 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG uwsgi [-] keystone_authtoken.admin_tenant_name = admin {{(pid=92057) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:50.999970 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG uwsgi [-] keystone_authtoken.admin_token = **** {{(pid=92057) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:51.000200 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG uwsgi [-] keystone_authtoken.admin_user = None {{(pid=92057) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:51.000446 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG uwsgi [-] keystone_authtoken.auth_admin_prefix = {{(pid=92057) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:51.000694 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG uwsgi [-] keystone_authtoken.auth_host = 127.0.0.1 {{(pid=92057) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:51.000930 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG uwsgi [-] keystone_authtoken.auth_port = 35357 {{(pid=92057) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:51.001177 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG uwsgi [-] keystone_authtoken.auth_protocol = https {{(pid=92057) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:51.001408 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG uwsgi [-] keystone_authtoken.auth_section = None {{(pid=92057) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:51.001643 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG uwsgi [-] keystone_authtoken.auth_type = None {{(pid=92057) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:51.001890 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG uwsgi [-] keystone_authtoken.auth_uri = None {{(pid=92057) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:51.002123 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG uwsgi [-] keystone_authtoken.auth_version = None {{(pid=92057) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:51.002416 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG uwsgi [-] keystone_authtoken.cache = None {{(pid=92057) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:51.002598 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [-] Callback: `keystone.catalog.core.Manager._on_project_or_endpoint_delete` subscribed to event `identity.project.deleted`. {{(pid=92056) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} May 23 12:39:51.002682 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG uwsgi [-] keystone_authtoken.cafile = None {{(pid=92057) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:51.002834 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [-] Callback: `keystone.catalog.core.Manager._on_project_or_endpoint_delete` subscribed to event `identity.endpoint.deleted`. {{(pid=92056) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} May 23 12:39:51.002938 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG uwsgi [-] keystone_authtoken.certfile = None {{(pid=92057) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:51.003194 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG uwsgi [-] keystone_authtoken.delay_auth_decision = False {{(pid=92057) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:51.003443 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG uwsgi [-] keystone_authtoken.enforce_token_bind = permissive {{(pid=92057) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:51.003672 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG uwsgi [-] keystone_authtoken.http_connect_timeout = None {{(pid=92057) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:51.003904 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG uwsgi [-] keystone_authtoken.http_request_max_retries = 3 {{(pid=92057) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:51.004148 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG uwsgi [-] keystone_authtoken.identity_uri = None {{(pid=92057) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:51.004366 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG uwsgi [-] keystone_authtoken.include_service_catalog = True {{(pid=92057) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:51.004653 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG uwsgi [-] keystone_authtoken.insecure = False {{(pid=92057) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:51.004959 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG uwsgi [-] keystone_authtoken.interface = internal {{(pid=92057) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:51.005256 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG uwsgi [-] keystone_authtoken.keyfile = None {{(pid=92057) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:51.005539 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_conn_get_timeout = 10 {{(pid=92057) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:51.005777 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_dead_retry = 300 {{(pid=92057) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:51.006099 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_maxsize = 10 {{(pid=92057) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:51.006361 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_socket_timeout = 3 {{(pid=92057) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:51.006652 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_unused_timeout = 60 {{(pid=92057) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:51.006959 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG uwsgi [-] keystone_authtoken.memcache_secret_key = **** {{(pid=92057) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:51.007253 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG uwsgi [-] keystone_authtoken.memcache_security_strategy = None {{(pid=92057) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:51.007513 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG uwsgi [-] keystone_authtoken.memcache_use_advanced_pool = True {{(pid=92057) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:51.007790 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG uwsgi [-] keystone_authtoken.memcached_servers = None {{(pid=92057) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:51.008043 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG uwsgi [-] keystone_authtoken.region_name = None {{(pid=92057) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:51.008301 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG uwsgi [-] keystone_authtoken.service_token_roles = ['service'] {{(pid=92057) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:51.008536 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG uwsgi [-] keystone_authtoken.service_token_roles_required = False {{(pid=92057) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:51.008765 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG uwsgi [-] keystone_authtoken.service_type = None {{(pid=92057) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:51.009009 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG uwsgi [-] keystone_authtoken.token_cache_time = 300 {{(pid=92057) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:51.009243 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG uwsgi [-] keystone_authtoken.www_authenticate_uri = None {{(pid=92057) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:51.009504 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG uwsgi [-] cache.backend = dogpile.cache.memcached {{(pid=92057) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:51.009798 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG uwsgi [-] cache.backend_argument = **** {{(pid=92057) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:51.010110 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG uwsgi [-] cache.config_prefix = cache.oslo {{(pid=92057) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:51.010403 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG uwsgi [-] cache.dead_timeout = 60.0 {{(pid=92057) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:51.010674 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG uwsgi [-] cache.debug_cache_backend = False {{(pid=92057) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:51.010917 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG uwsgi [-] cache.enable_retry_client = False {{(pid=92057) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:51.011154 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG uwsgi [-] cache.enable_socket_keepalive = False {{(pid=92057) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:51.011400 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG uwsgi [-] cache.enabled = True {{(pid=92057) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:51.011676 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG uwsgi [-] cache.expiration_time = 600 {{(pid=92057) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:51.011937 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG uwsgi [-] cache.hashclient_retry_attempts = 2 {{(pid=92057) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:51.012179 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG uwsgi [-] cache.hashclient_retry_delay = 1.0 {{(pid=92057) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:51.012412 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG uwsgi [-] cache.memcache_dead_retry = 300 {{(pid=92057) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:51.012693 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG uwsgi [-] cache.memcache_pool_connection_get_timeout = 10 {{(pid=92057) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:51.012969 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG uwsgi [-] cache.memcache_pool_flush_on_reconnect = False {{(pid=92057) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:51.013206 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG uwsgi [-] cache.memcache_pool_maxsize = 10 {{(pid=92057) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:51.013440 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG uwsgi [-] cache.memcache_pool_unused_timeout = 60 {{(pid=92057) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:51.013697 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG uwsgi [-] cache.memcache_servers = ['localhost:11211'] {{(pid=92057) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:51.014533 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG uwsgi [-] cache.memcache_socket_timeout = 1.0 {{(pid=92057) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:51.014805 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG uwsgi [-] cache.proxies = [] {{(pid=92057) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:51.015042 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG uwsgi [-] cache.retry_attempts = 2 {{(pid=92057) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:51.015281 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG uwsgi [-] cache.retry_delay = 0.0 {{(pid=92057) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:51.015569 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG uwsgi [-] cache.socket_keepalive_count = 1 {{(pid=92057) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:51.015806 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG uwsgi [-] cache.socket_keepalive_idle = 1 {{(pid=92057) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:51.016074 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG uwsgi [-] cache.socket_keepalive_interval = 1 {{(pid=92057) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:51.016367 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG uwsgi [-] cache.tls_allowed_ciphers = None {{(pid=92057) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:51.016663 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG uwsgi [-] cache.tls_cafile = None {{(pid=92057) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:51.016967 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG uwsgi [-] cache.tls_certfile = None {{(pid=92057) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:51.017248 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG uwsgi [-] cache.tls_enabled = False {{(pid=92057) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:51.017479 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG uwsgi [-] cache.tls_keyfile = None {{(pid=92057) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:51.017725 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG uwsgi [-] application_credential.cache_time = None {{(pid=92057) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:51.018145 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG uwsgi [-] application_credential.caching = True {{(pid=92057) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:51.018460 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG uwsgi [-] application_credential.driver = sql {{(pid=92057) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:51.018698 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG uwsgi [-] application_credential.user_limit = -1 {{(pid=92057) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:51.018953 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG uwsgi [-] assignment.driver = sql {{(pid=92057) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:51.019266 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG uwsgi [-] assignment.prohibited_implied_role = ['admin'] {{(pid=92057) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:51.019539 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG uwsgi [-] auth.application_credential = None {{(pid=92057) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:51.019770 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG uwsgi [-] auth.external = None {{(pid=92057) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:51.019999 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG uwsgi [-] auth.mapped = None {{(pid=92057) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:51.020346 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG uwsgi [-] auth.methods = ['external', 'password', 'token', 'oauth1', 'mapped', 'application_credential'] {{(pid=92057) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:51.020622 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG uwsgi [-] auth.oauth1 = None {{(pid=92057) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:51.020861 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG uwsgi [-] auth.password = None {{(pid=92057) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:51.021149 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG uwsgi [-] auth.token = None {{(pid=92057) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:51.021411 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG uwsgi [-] catalog.cache_time = None {{(pid=92057) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:51.021699 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG uwsgi [-] catalog.caching = True {{(pid=92057) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:51.022069 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG uwsgi [-] catalog.driver = sql {{(pid=92057) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:51.022361 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG uwsgi [-] catalog.list_limit = None {{(pid=92057) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:51.022676 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG uwsgi [-] catalog.template_file = default_catalog.templates {{(pid=92057) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:51.022962 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG uwsgi [-] credential.auth_ttl = 15 {{(pid=92057) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:51.023218 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG uwsgi [-] credential.cache_time = None {{(pid=92057) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:51.023486 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG uwsgi [-] credential.caching = True {{(pid=92057) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:51.023764 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG uwsgi [-] credential.driver = sql {{(pid=92057) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:51.024065 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG uwsgi [-] credential.key_repository = /etc/keystone/credential-keys/ {{(pid=92057) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:51.024354 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG uwsgi [-] credential.provider = fernet {{(pid=92057) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:51.024653 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG uwsgi [-] credential.user_limit = -1 {{(pid=92057) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:51.024939 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG uwsgi [-] domain_config.cache_time = 300 {{(pid=92057) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:51.025194 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG uwsgi [-] domain_config.caching = True {{(pid=92057) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:51.025509 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG uwsgi [-] domain_config.driver = sql {{(pid=92057) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:51.025779 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG uwsgi [-] endpoint_filter.driver = sql {{(pid=92057) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:51.026106 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG uwsgi [-] endpoint_filter.return_all_endpoints_if_no_filter = True {{(pid=92057) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:51.026426 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG uwsgi [-] endpoint_policy.driver = sql {{(pid=92057) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:51.026748 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG uwsgi [-] eventlet_server.admin_bind_host = 0.0.0.0 {{(pid=92057) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:51.027061 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG uwsgi [-] eventlet_server.admin_port = 35357 {{(pid=92057) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:51.027354 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG uwsgi [-] eventlet_server.public_bind_host = 0.0.0.0 {{(pid=92057) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:51.027643 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG uwsgi [-] eventlet_server.public_port = 5000 {{(pid=92057) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:51.027945 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG uwsgi [-] federation.assertion_prefix = {{(pid=92057) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:51.028217 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG uwsgi [-] federation.caching = True {{(pid=92057) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:51.028485 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG uwsgi [-] federation.default_authorization_ttl = 0 {{(pid=92057) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:51.028734 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG uwsgi [-] federation.driver = sql {{(pid=92057) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:51.028984 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG uwsgi [-] federation.federated_domain_name = Federated {{(pid=92057) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:51.029215 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG uwsgi [-] federation.remote_id_attribute = None {{(pid=92057) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:51.029462 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG uwsgi [-] federation.sso_callback_template = /etc/keystone/sso_callback_template.html {{(pid=92057) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:51.030273 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG uwsgi [-] federation.trusted_dashboard = [] {{(pid=92057) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:51.030533 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG uwsgi [-] fernet_receipts.key_repository = /etc/keystone/fernet-keys/ {{(pid=92057) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:51.030765 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG uwsgi [-] fernet_receipts.max_active_keys = 3 {{(pid=92057) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:51.031018 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG uwsgi [-] fernet_tokens.key_repository = /etc/keystone/fernet-keys/ {{(pid=92057) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:51.031505 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG uwsgi [-] fernet_tokens.max_active_keys = 3 {{(pid=92057) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:51.031750 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG uwsgi [-] identity.cache_time = 600 {{(pid=92057) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:51.031983 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG uwsgi [-] identity.caching = True {{(pid=92057) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:51.032231 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG uwsgi [-] identity.default_domain_id = default {{(pid=92057) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:51.032477 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG uwsgi [-] identity.domain_config_dir = /etc/keystone/domains {{(pid=92057) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:51.032710 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG uwsgi [-] identity.domain_configurations_from_database = False {{(pid=92057) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:51.032943 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG uwsgi [-] identity.domain_specific_drivers_enabled = False {{(pid=92057) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:51.033288 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG uwsgi [-] identity.driver = sql {{(pid=92057) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:51.033519 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG uwsgi [-] identity.list_limit = None {{(pid=92057) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:51.033755 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG uwsgi [-] identity.max_password_length = 4096 {{(pid=92057) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:51.034028 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG uwsgi [-] identity.password_hash_algorithm = bcrypt {{(pid=92057) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:51.034278 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG uwsgi [-] identity.password_hash_rounds = 4 {{(pid=92057) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:51.034507 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG uwsgi [-] identity.salt_bytesize = None {{(pid=92057) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:51.034737 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG uwsgi [-] identity.scrypt_block_size = None {{(pid=92057) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:51.034967 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG uwsgi [-] identity.scrypt_parallelism = None {{(pid=92057) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:51.035224 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG uwsgi [-] identity_mapping.backward_compatible_ids = True {{(pid=92057) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:51.035497 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG uwsgi [-] identity_mapping.driver = sql {{(pid=92057) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:51.035743 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG uwsgi [-] identity_mapping.generator = sha256 {{(pid=92057) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:51.036050 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG uwsgi [-] jwt_tokens.jws_private_key_repository = /etc/keystone/jws-keys/private {{(pid=92057) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:51.036354 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG uwsgi [-] jwt_tokens.jws_public_key_repository = /etc/keystone/jws-keys/public {{(pid=92057) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:51.036629 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG uwsgi [-] ldap.alias_dereferencing = default {{(pid=92057) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:51.036865 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG uwsgi [-] ldap.auth_pool_connection_lifetime = 60 {{(pid=92057) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:51.037102 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG uwsgi [-] ldap.auth_pool_size = 100 {{(pid=92057) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:51.037332 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG uwsgi [-] ldap.chase_referrals = None {{(pid=92057) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:51.037566 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG uwsgi [-] ldap.connection_timeout = -1 {{(pid=92057) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:51.037793 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG uwsgi [-] ldap.debug_level = None {{(pid=92057) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:51.038054 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG uwsgi [-] ldap.group_ad_nesting = False {{(pid=92057) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:51.038294 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG uwsgi [-] ldap.group_additional_attribute_mapping = [] {{(pid=92057) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:51.038532 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG uwsgi [-] ldap.group_attribute_ignore = [] {{(pid=92057) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:51.038777 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG uwsgi [-] ldap.group_desc_attribute = description {{(pid=92057) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:51.039006 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG uwsgi [-] ldap.group_filter = None {{(pid=92057) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:51.039252 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG uwsgi [-] ldap.group_id_attribute = cn {{(pid=92057) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:51.039497 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG uwsgi [-] ldap.group_member_attribute = member {{(pid=92057) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:51.039730 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG uwsgi [-] ldap.group_members_are_ids = False {{(pid=92057) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:51.039982 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG uwsgi [-] ldap.group_name_attribute = ou {{(pid=92057) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:51.040226 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG uwsgi [-] ldap.group_objectclass = groupOfNames {{(pid=92057) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:51.040454 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG uwsgi [-] ldap.group_tree_dn = None {{(pid=92057) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:51.040688 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG uwsgi [-] ldap.page_size = 0 {{(pid=92057) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:51.040915 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG uwsgi [-] ldap.password = **** {{(pid=92057) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:51.041150 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG uwsgi [-] ldap.pool_connection_lifetime = 600 {{(pid=92057) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:51.041382 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG uwsgi [-] ldap.pool_connection_timeout = -1 {{(pid=92057) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:51.041620 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG uwsgi [-] ldap.pool_retry_delay = 0.1 {{(pid=92057) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:51.041854 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG uwsgi [-] ldap.pool_retry_max = 3 {{(pid=92057) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:51.042126 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG uwsgi [-] ldap.pool_size = 10 {{(pid=92057) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:51.042372 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG uwsgi [-] ldap.query_scope = one {{(pid=92057) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:51.042616 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG uwsgi [-] ldap.suffix = cn=example,cn=com {{(pid=92057) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:51.042845 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG uwsgi [-] ldap.tls_cacertdir = None {{(pid=92057) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:51.043073 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG uwsgi [-] ldap.tls_cacertfile = None {{(pid=92057) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:51.043317 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG uwsgi [-] ldap.tls_req_cert = demand {{(pid=92057) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:51.043560 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG uwsgi [-] ldap.url = ldap://localhost {{(pid=92057) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:51.043791 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG uwsgi [-] ldap.use_auth_pool = True {{(pid=92057) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:51.044030 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG uwsgi [-] ldap.use_pool = True {{(pid=92057) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:51.044261 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG uwsgi [-] ldap.use_tls = False {{(pid=92057) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:51.044488 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG uwsgi [-] ldap.user = None {{(pid=92057) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:51.044725 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG uwsgi [-] ldap.user_additional_attribute_mapping = [] {{(pid=92057) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:51.044980 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG uwsgi [-] ldap.user_attribute_ignore = ['default_project_id'] {{(pid=92057) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:51.045243 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG uwsgi [-] ldap.user_default_project_id_attribute = None {{(pid=92057) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:51.045487 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG uwsgi [-] ldap.user_description_attribute = description {{(pid=92057) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:51.045730 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG uwsgi [-] ldap.user_enabled_attribute = enabled {{(pid=92057) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:51.046000 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG uwsgi [-] ldap.user_enabled_default = True {{(pid=92057) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:51.046235 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG uwsgi [-] ldap.user_enabled_emulation = False {{(pid=92057) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:51.046462 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG uwsgi [-] ldap.user_enabled_emulation_dn = None {{(pid=92057) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:51.046700 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG uwsgi [-] ldap.user_enabled_emulation_use_group_config = False {{(pid=92057) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:51.046931 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG uwsgi [-] ldap.user_enabled_invert = False {{(pid=92057) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:51.047161 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG uwsgi [-] ldap.user_enabled_mask = 0 {{(pid=92057) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:51.047388 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG uwsgi [-] ldap.user_filter = None {{(pid=92057) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:51.047628 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG uwsgi [-] ldap.user_id_attribute = cn {{(pid=92057) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:51.047871 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG uwsgi [-] ldap.user_mail_attribute = mail {{(pid=92057) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:51.048114 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG uwsgi [-] ldap.user_name_attribute = sn {{(pid=92057) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:51.048355 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG uwsgi [-] ldap.user_objectclass = inetOrgPerson {{(pid=92057) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:51.048597 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG uwsgi [-] ldap.user_pass_attribute = userPassword {{(pid=92057) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:51.048822 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG uwsgi [-] ldap.user_tree_dn = None {{(pid=92057) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:51.049064 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG uwsgi [-] memcache.dead_retry = 300 {{(pid=92057) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:51.049295 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG uwsgi [-] memcache.pool_connection_get_timeout = 10 {{(pid=92057) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:51.049526 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG uwsgi [-] memcache.pool_maxsize = 10 {{(pid=92057) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:51.049756 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG uwsgi [-] memcache.pool_unused_timeout = 60 {{(pid=92057) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:51.050009 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG uwsgi [-] memcache.socket_timeout = 3 {{(pid=92057) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:51.050252 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG uwsgi [-] oauth1.access_token_duration = 86400 {{(pid=92057) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:51.050496 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG uwsgi [-] oauth1.driver = sql {{(pid=92057) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:51.050728 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG uwsgi [-] oauth1.request_token_duration = 28800 {{(pid=92057) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:51.050977 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG uwsgi [-] policy.driver = sql {{(pid=92057) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:51.051204 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG uwsgi [-] policy.list_limit = None {{(pid=92057) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:51.051444 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG uwsgi [-] receipt.cache_on_issue = True {{(pid=92057) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:51.051675 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG uwsgi [-] receipt.cache_time = 300 {{(pid=92057) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:51.051906 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG uwsgi [-] receipt.caching = True {{(pid=92057) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:51.052137 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG uwsgi [-] receipt.expiration = 300 {{(pid=92057) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:51.052378 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG uwsgi [-] receipt.provider = fernet {{(pid=92057) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:51.052614 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG uwsgi [-] resource.admin_project_domain_name = None {{(pid=92057) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:51.052842 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG uwsgi [-] resource.admin_project_name = None {{(pid=92057) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:51.053075 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG uwsgi [-] resource.cache_time = None {{(pid=92057) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:51.053311 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG uwsgi [-] resource.caching = True {{(pid=92057) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:51.053554 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG uwsgi [-] resource.domain_name_url_safe = off {{(pid=92057) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:51.053797 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG uwsgi [-] resource.driver = sql {{(pid=92057) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:51.054050 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG uwsgi [-] resource.list_limit = None {{(pid=92057) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:51.054295 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG uwsgi [-] resource.project_name_url_safe = off {{(pid=92057) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:51.054538 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG uwsgi [-] revoke.cache_time = 3600 {{(pid=92057) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:51.054769 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG uwsgi [-] revoke.caching = True {{(pid=92057) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:51.055009 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG uwsgi [-] revoke.driver = sql {{(pid=92057) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:51.055240 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG uwsgi [-] revoke.expiration_buffer = 1800 {{(pid=92057) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:51.055473 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG uwsgi [-] role.cache_time = None {{(pid=92057) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:51.055702 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG uwsgi [-] role.caching = True {{(pid=92057) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:51.055944 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG uwsgi [-] role.driver = sql {{(pid=92057) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:51.056183 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG uwsgi [-] role.list_limit = None {{(pid=92057) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:51.056428 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG uwsgi [-] saml.assertion_expiration_time = 3600 {{(pid=92057) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:51.056677 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG uwsgi [-] saml.certfile = /etc/keystone/ssl/certs/signing_cert.pem {{(pid=92057) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:51.056755 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [-] Callback: `keystone.federation.core.Manager._cleanup_identity_provider` subscribed to event `identity.domain_deleted.internal`. {{(pid=92056) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} May 23 12:39:51.056922 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG uwsgi [-] saml.idp_contact_company = Example, Inc. {{(pid=92057) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:51.057165 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG uwsgi [-] saml.idp_contact_email = support@example.com {{(pid=92057) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:51.057406 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG uwsgi [-] saml.idp_contact_name = SAML Identity Provider Support {{(pid=92057) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:51.057649 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG uwsgi [-] saml.idp_contact_surname = Support {{(pid=92057) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:51.057908 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG uwsgi [-] saml.idp_contact_telephone = +1 800 555 0100 {{(pid=92057) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:51.058166 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG uwsgi [-] saml.idp_contact_type = other {{(pid=92057) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:51.058396 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG uwsgi [-] saml.idp_entity_id = None {{(pid=92057) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:51.058638 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG uwsgi [-] saml.idp_lang = en {{(pid=92057) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:51.058879 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG uwsgi [-] saml.idp_metadata_path = /etc/keystone/saml2_idp_metadata.xml {{(pid=92057) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:51.059122 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG uwsgi [-] saml.idp_organization_display_name = OpenStack SAML Identity Provider {{(pid=92057) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:51.059365 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG uwsgi [-] saml.idp_organization_name = SAML Identity Provider {{(pid=92057) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:51.059757 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG uwsgi [-] saml.idp_organization_url = https://example.com/ {{(pid=92057) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:51.059989 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG uwsgi [-] saml.idp_sso_endpoint = None {{(pid=92057) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:51.060234 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG uwsgi [-] saml.keyfile = /etc/keystone/ssl/private/signing_key.pem {{(pid=92057) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:51.060479 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG uwsgi [-] saml.relay_state_prefix = ss:mem: {{(pid=92057) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:51.060743 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG uwsgi [-] saml.xmlsec1_binary = xmlsec1 {{(pid=92057) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:51.061005 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG uwsgi [-] security_compliance.change_password_upon_first_use = False {{(pid=92057) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:51.061237 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG uwsgi [-] security_compliance.disable_user_account_days_inactive = None {{(pid=92057) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:51.061488 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG uwsgi [-] security_compliance.lockout_duration = 10 {{(pid=92057) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:51.061735 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG uwsgi [-] security_compliance.lockout_failure_attempts = 2 {{(pid=92057) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:51.061996 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG uwsgi [-] security_compliance.minimum_password_age = 0 {{(pid=92057) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:51.062226 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG uwsgi [-] security_compliance.password_expires_days = None {{(pid=92057) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:51.062454 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG uwsgi [-] security_compliance.password_regex = None {{(pid=92057) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:51.062678 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG uwsgi [-] security_compliance.password_regex_description = None {{(pid=92057) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:51.062924 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG uwsgi [-] security_compliance.unique_last_password_count = 2 {{(pid=92057) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:51.063171 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG uwsgi [-] shadow_users.driver = sql {{(pid=92057) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:51.063411 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG uwsgi [-] token.allow_expired_window = 172800 {{(pid=92057) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:51.063643 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG uwsgi [-] token.allow_rescope_scoped_token = True {{(pid=92057) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:51.063874 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG uwsgi [-] token.cache_on_issue = True {{(pid=92057) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:51.064098 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG uwsgi [-] token.cache_time = None {{(pid=92057) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:51.064328 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG uwsgi [-] token.caching = True {{(pid=92057) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:51.064558 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG uwsgi [-] token.expiration = 3600 {{(pid=92057) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:51.064798 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG uwsgi [-] token.provider = fernet {{(pid=92057) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:51.065027 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG uwsgi [-] token.revoke_by_id = True {{(pid=92057) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:51.065276 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG uwsgi [-] tokenless_auth.issuer_attribute = SSL_CLIENT_I_DN {{(pid=92057) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:51.065516 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG uwsgi [-] tokenless_auth.protocol = x509 {{(pid=92057) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:51.065749 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG uwsgi [-] tokenless_auth.trusted_issuer = [] {{(pid=92057) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:51.066061 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG uwsgi [-] totp.included_previous_windows = 1 {{(pid=92057) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:51.066379 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG uwsgi [-] trust.allow_redelegation = False {{(pid=92057) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:51.066677 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG uwsgi [-] trust.driver = sql {{(pid=92057) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:51.066914 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG uwsgi [-] trust.max_redelegation_count = 3 {{(pid=92057) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:51.067149 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG uwsgi [-] unified_limit.cache_time = None {{(pid=92057) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:51.067428 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG uwsgi [-] unified_limit.caching = True {{(pid=92057) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:51.067708 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG uwsgi [-] unified_limit.driver = sql {{(pid=92057) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:51.067999 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG uwsgi [-] unified_limit.enforcement_model = flat {{(pid=92057) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:51.068252 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG uwsgi [-] unified_limit.list_limit = None {{(pid=92057) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:51.068537 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG uwsgi [-] wsgi.debug_middleware = False {{(pid=92057) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:51.068889 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG uwsgi [-] database.backend = sqlalchemy {{(pid=92057) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:51.069196 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG uwsgi [-] database.connection = **** {{(pid=92057) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:51.069474 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG uwsgi [-] database.connection_debug = 0 {{(pid=92057) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:51.069726 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG uwsgi [-] database.connection_parameters = {{(pid=92057) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:51.069991 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG uwsgi [-] database.connection_recycle_time = 3600 {{(pid=92057) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:51.070232 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG uwsgi [-] database.connection_trace = False {{(pid=92057) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:51.070465 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG uwsgi [-] database.db_inc_retry_interval = True {{(pid=92057) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:51.070699 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG uwsgi [-] database.db_max_retries = 20 {{(pid=92057) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:51.070932 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG uwsgi [-] database.db_max_retry_interval = 10 {{(pid=92057) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:51.071165 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG uwsgi [-] database.db_retry_interval = 1 {{(pid=92057) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:51.071407 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG uwsgi [-] database.max_overflow = 50 {{(pid=92057) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:51.071639 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG uwsgi [-] database.max_pool_size = 5 {{(pid=92057) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:51.071877 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG uwsgi [-] database.max_retries = 10 {{(pid=92057) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:51.072108 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG uwsgi [-] database.mysql_enable_ndb = False {{(pid=92057) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:51.072349 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG uwsgi [-] database.mysql_sql_mode = TRADITIONAL {{(pid=92057) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:51.072581 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG uwsgi [-] database.pool_timeout = None {{(pid=92057) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:51.072818 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG uwsgi [-] database.retry_interval = 10 {{(pid=92057) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:51.073045 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG uwsgi [-] database.slave_connection = **** {{(pid=92057) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:51.073279 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG uwsgi [-] database.sqlite_synchronous = True {{(pid=92057) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:51.073508 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG uwsgi [-] database.use_db_reconnect = False {{(pid=92057) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 23 12:39:51.073675 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG uwsgi [-] ******************************************************************************** {{(pid=92057) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2613}} May 23 12:39:51.102229 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [-] Callback: `keystone.application_credential.core.Manager._delete_app_creds_on_user_delete_callback` subscribed to event `identity.user.deleted`. {{(pid=92057) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} May 23 12:39:51.102478 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [-] Callback: `keystone.application_credential.core.Manager._delete_app_creds_on_user_delete_callback` subscribed to event `identity.user.disabled`. {{(pid=92057) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} May 23 12:39:51.102759 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [-] Callback: `keystone.application_credential.core.Manager._delete_app_creds_on_assignment_removal` subscribed to event `identity.remove_application_credentials_for_user.internal`. {{(pid=92057) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} May 23 12:39:51.108617 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [-] Callback: `keystone.identity.core.Manager._domain_deleted` subscribed to event `identity.domain_deleted.internal`. {{(pid=92056) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} May 23 12:39:51.108851 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [-] Callback: `keystone.identity.core.Manager._unset_default_project` subscribed to event `identity.project.deleted`. {{(pid=92056) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} May 23 12:39:51.112060 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [-] Callback: `keystone.assignment.core.Manager._delete_domain_assignments` subscribed to event `identity.domain.deleted`. {{(pid=92057) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} May 23 12:39:51.131834 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [-] Callback: `keystone.catalog.core.Manager._on_project_or_endpoint_delete` subscribed to event `identity.project.deleted`. {{(pid=92057) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} May 23 12:39:51.132066 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [-] Callback: `keystone.catalog.core.Manager._on_project_or_endpoint_delete` subscribed to event `identity.endpoint.deleted`. {{(pid=92057) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} May 23 12:39:51.171963 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [-] Callback: `keystone.federation.core.Manager._cleanup_identity_provider` subscribed to event `identity.domain_deleted.internal`. {{(pid=92057) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} May 23 12:39:51.175586 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._trust_callback` subscribed to event `identity.OS-TRUST:trust.deleted`. {{(pid=92056) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} May 23 12:39:51.175826 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._consumer_callback` subscribed to event `identity.OS-OAUTH1:consumer.deleted`. {{(pid=92056) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} May 23 12:39:51.176057 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._user_callback` subscribed to event `identity.user.deleted`. {{(pid=92056) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} May 23 12:39:51.176282 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._project_callback` subscribed to event `identity.project.deleted`. {{(pid=92056) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} May 23 12:39:51.176502 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._user_callback` subscribed to event `identity.user.disabled`. {{(pid=92056) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} May 23 12:39:51.176719 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._user_callback` subscribed to event `identity.persist_revocation_event_for_user.internal`. {{(pid=92056) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} May 23 12:39:51.200490 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.OS-TRUST:trust.deleted`. {{(pid=92056) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} May 23 12:39:51.200722 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.user.deleted`. {{(pid=92056) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} May 23 12:39:51.200948 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.domain.deleted`. {{(pid=92056) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} May 23 12:39:51.201170 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.user.disabled`. {{(pid=92056) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} May 23 12:39:51.201999 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.domain.disabled`. {{(pid=92056) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} May 23 12:39:51.202257 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.project.disabled`. {{(pid=92056) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} May 23 12:39:51.202477 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.invalidate_token_cache.internal`. {{(pid=92056) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} May 23 12:39:51.211927 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [-] Callback: `keystone.identity.core.Manager._domain_deleted` subscribed to event `identity.domain_deleted.internal`. {{(pid=92057) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} May 23 12:39:51.212159 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [-] Callback: `keystone.identity.core.Manager._unset_default_project` subscribed to event `identity.project.deleted`. {{(pid=92057) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} May 23 12:39:51.213848 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [-] Callback: `keystone.trust.core.Manager._on_user_delete` subscribed to event `identity.user.deleted`. {{(pid=92056) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} May 23 12:39:51.226986 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.OS-TRUST:trust.deleted`. {{(pid=92056) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} May 23 12:39:51.227163 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.user.deleted`. {{(pid=92056) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} May 23 12:39:51.227387 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.domain.deleted`. {{(pid=92056) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} May 23 12:39:51.227610 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.user.disabled`. {{(pid=92056) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} May 23 12:39:51.227829 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.domain.disabled`. {{(pid=92056) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} May 23 12:39:51.228046 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.project.disabled`. {{(pid=92056) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} May 23 12:39:51.228262 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.invalidate_token_cache.internal`. {{(pid=92056) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} May 23 12:39:51.250354 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/projects' {}] {{(pid=92056) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} May 23 12:39:51.250562 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.common [-] Adding additional resource route (alternate) to API auth: ['/OS-FEDERATION/projects' {}] {{(pid=92056) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:433}} May 23 12:39:51.250904 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/domains' {}] {{(pid=92056) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} May 23 12:39:51.251108 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.common [-] Adding additional resource route (alternate) to API auth: ['/OS-FEDERATION/domains' {}] {{(pid=92056) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:433}} May 23 12:39:51.251381 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/system' {}] {{(pid=92056) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} May 23 12:39:51.251643 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/catalog' {}] {{(pid=92056) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} May 23 12:39:51.251899 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/tokens/OS-PKI/revoked' {}] {{(pid=92056) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} May 23 12:39:51.252154 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/tokens' {}] {{(pid=92056) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} May 23 12:39:51.258129 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._trust_callback` subscribed to event `identity.OS-TRUST:trust.deleted`. {{(pid=92057) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} May 23 12:39:51.258365 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._consumer_callback` subscribed to event `identity.OS-OAUTH1:consumer.deleted`. {{(pid=92057) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} May 23 12:39:51.258583 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._user_callback` subscribed to event `identity.user.deleted`. {{(pid=92057) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} May 23 12:39:51.258798 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._project_callback` subscribed to event `identity.project.deleted`. {{(pid=92057) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} May 23 12:39:51.259012 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._user_callback` subscribed to event `identity.user.disabled`. {{(pid=92057) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} May 23 12:39:51.259226 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._user_callback` subscribed to event `identity.persist_revocation_event_for_user.internal`. {{(pid=92057) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} May 23 12:39:51.263124 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth/OS-FEDERATION: ['/auth/OS-FEDERATION/saml2' {}] {{(pid=92056) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} May 23 12:39:51.263404 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth/OS-FEDERATION: ['/auth/OS-FEDERATION/saml2/ecp' {}] {{(pid=92056) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} May 23 12:39:51.263670 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth/OS-FEDERATION: ['/auth/OS-FEDERATION/websso/' {}] {{(pid=92056) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} May 23 12:39:51.264514 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth/OS-FEDERATION: ['/auth/OS-FEDERATION/identity_providers//protocols//websso' {}] {{(pid=92056) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} May 23 12:39:51.272629 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.common [-] Adding standard routes to API credentials for `MethodViewType` (API Prefix: ) [/credentials, /credentials/] {{(pid=92056) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} May 23 12:39:51.274813 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.OS-TRUST:trust.deleted`. {{(pid=92057) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} May 23 12:39:51.275044 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.user.deleted`. {{(pid=92057) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} May 23 12:39:51.275267 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.domain.deleted`. {{(pid=92057) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} May 23 12:39:51.275483 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.user.disabled`. {{(pid=92057) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} May 23 12:39:51.275694 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.domain.disabled`. {{(pid=92057) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} May 23 12:39:51.275906 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.project.disabled`. {{(pid=92057) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} May 23 12:39:51.275988 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.common [-] Adding standard routes to API domains for `MethodViewType` (API Prefix: ) [/domains, /domains/] {{(pid=92056) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} May 23 12:39:51.276161 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.invalidate_token_cache.internal`. {{(pid=92057) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} May 23 12:39:51.276245 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//config' {}] {{(pid=92056) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} May 23 12:39:51.276467 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//config/' {}] {{(pid=92056) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} May 23 12:39:51.276747 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//config//' {}] {{(pid=92056) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} May 23 12:39:51.277027 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains/config/default' {}] {{(pid=92056) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} May 23 12:39:51.277285 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains/config//default' {}] {{(pid=92056) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} May 23 12:39:51.277665 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains/config///default' {}] {{(pid=92056) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} May 23 12:39:51.277969 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//users//roles' {}] {{(pid=92056) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} May 23 12:39:51.278246 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//users//roles/' {}] {{(pid=92056) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} May 23 12:39:51.278522 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//groups//roles' {}] {{(pid=92056) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} May 23 12:39:51.278793 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//groups//roles/' {}] {{(pid=92056) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} May 23 12:39:51.284925 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [-] Callback: `keystone.trust.core.Manager._on_user_delete` subscribed to event `identity.user.deleted`. {{(pid=92057) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} May 23 12:39:51.288195 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.OS-TRUST:trust.deleted`. {{(pid=92057) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} May 23 12:39:51.288429 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.user.deleted`. {{(pid=92057) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} May 23 12:39:51.288651 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.domain.deleted`. {{(pid=92057) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} May 23 12:39:51.288869 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.user.disabled`. {{(pid=92057) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} May 23 12:39:51.289082 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.domain.disabled`. {{(pid=92057) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} May 23 12:39:51.289290 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.project.disabled`. {{(pid=92057) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} May 23 12:39:51.289498 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.invalidate_token_cache.internal`. {{(pid=92057) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} May 23 12:39:51.300479 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/projects' {}] {{(pid=92057) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} May 23 12:39:51.300683 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.common [-] Adding additional resource route (alternate) to API auth: ['/OS-FEDERATION/projects' {}] {{(pid=92057) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:433}} May 23 12:39:51.301012 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/domains' {}] {{(pid=92057) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} May 23 12:39:51.301210 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.common [-] Adding additional resource route (alternate) to API auth: ['/OS-FEDERATION/domains' {}] {{(pid=92057) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:433}} May 23 12:39:51.301473 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/system' {}] {{(pid=92057) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} May 23 12:39:51.301724 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/catalog' {}] {{(pid=92057) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} May 23 12:39:51.302045 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/tokens/OS-PKI/revoked' {}] {{(pid=92057) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} May 23 12:39:51.302297 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/tokens' {}] {{(pid=92057) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} May 23 12:39:51.305410 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.common [-] Adding resource routes to API ec2tokens: ['/ec2tokens' {}] {{(pid=92056) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} May 23 12:39:51.307168 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.common [-] Adding standard routes to API endpoints for `MethodViewType` (API Prefix: ) [/endpoints, /endpoints/] {{(pid=92056) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} May 23 12:39:51.307453 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.common [-] Adding resource routes to API endpoints: ['/endpoints//OS-ENDPOINT-POLICY/policy' {}] {{(pid=92056) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} May 23 12:39:51.313440 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.common [-] Adding standard routes to API groups for `MethodViewType` (API Prefix: ) [/groups, /groups/] {{(pid=92056) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} May 23 12:39:51.313721 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.common [-] Adding resource routes to API groups: ['/groups//users' {}] {{(pid=92056) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} May 23 12:39:51.314028 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.common [-] Adding resource routes to API groups: ['/groups//users/' {}] {{(pid=92056) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} May 23 12:39:51.314578 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth/OS-FEDERATION: ['/auth/OS-FEDERATION/saml2' {}] {{(pid=92057) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} May 23 12:39:51.314848 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth/OS-FEDERATION: ['/auth/OS-FEDERATION/saml2/ecp' {}] {{(pid=92057) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} May 23 12:39:51.315209 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth/OS-FEDERATION: ['/auth/OS-FEDERATION/websso/' {}] {{(pid=92057) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} May 23 12:39:51.315547 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth/OS-FEDERATION: ['/auth/OS-FEDERATION/identity_providers//protocols//websso' {}] {{(pid=92057) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} May 23 12:39:51.321298 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.common [-] Adding standard routes to API limits for `MethodViewType` (API Prefix: ) [/limits, /limits/] {{(pid=92056) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} May 23 12:39:51.321586 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.common [-] Adding resource routes to API limits: ['/limits/model' {}] {{(pid=92056) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} May 23 12:39:51.324599 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.common [-] Adding standard routes to API credentials for `MethodViewType` (API Prefix: ) [/credentials, /credentials/] {{(pid=92057) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} May 23 12:39:51.326098 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.common [-] Adding standard routes to API OS-EP-FILTER for `MethodViewType` (API Prefix: /OS-EP-FILTER) [/endpoint_groups, /endpoint_groups/] {{(pid=92056) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} May 23 12:39:51.327108 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/endpoints//projects' {}] {{(pid=92056) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} May 23 12:39:51.327450 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/projects//endpoints/' {}] {{(pid=92056) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} May 23 12:39:51.327754 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/projects//endpoints' {}] {{(pid=92056) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} May 23 12:39:51.328031 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/projects//endpoint_groups' {}] {{(pid=92056) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} May 23 12:39:51.328305 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/endpoint_groups//endpoints' {}] {{(pid=92056) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} May 23 12:39:51.328574 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/endpoint_groups//projects' {}] {{(pid=92056) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} May 23 12:39:51.328718 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.common [-] Adding standard routes to API domains for `MethodViewType` (API Prefix: ) [/domains, /domains/] {{(pid=92057) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} May 23 12:39:51.328842 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/endpoint_groups//projects/' {}] {{(pid=92056) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} May 23 12:39:51.329003 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//config' {}] {{(pid=92057) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} May 23 12:39:51.329282 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//config/' {}] {{(pid=92057) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} May 23 12:39:51.329553 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//config//' {}] {{(pid=92057) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} May 23 12:39:51.329825 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains/config/default' {}] {{(pid=92057) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} May 23 12:39:51.330099 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains/config//default' {}] {{(pid=92057) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} May 23 12:39:51.330414 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains/config///default' {}] {{(pid=92057) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} May 23 12:39:51.330740 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//users//roles' {}] {{(pid=92057) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} May 23 12:39:51.331014 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//users//roles/' {}] {{(pid=92057) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} May 23 12:39:51.331387 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//groups//roles' {}] {{(pid=92057) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} May 23 12:39:51.331651 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//groups//roles/' {}] {{(pid=92057) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} May 23 12:39:51.347696 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-FEDERATION: ['/saml2/metadata' {}] {{(pid=92056) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} May 23 12:39:51.347975 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-FEDERATION: ['/identity_providers//protocols//auth' {}] {{(pid=92056) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} May 23 12:39:51.352658 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.common [-] Adding standard routes to API identity_providers for `MethodViewType` (API Prefix: /OS-FEDERATION) [/identity_providers, /identity_providers/] {{(pid=92056) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} May 23 12:39:51.356353 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.common [-] Adding resource routes to API protocols: ['/OS-FEDERATION/identity_providers//protocols/' {}] {{(pid=92056) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} May 23 12:39:51.356648 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.common [-] Adding resource routes to API protocols: ['/OS-FEDERATION/identity_providers//protocols' {}] {{(pid=92056) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} May 23 12:39:51.358638 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.common [-] Adding resource routes to API ec2tokens: ['/ec2tokens' {}] {{(pid=92057) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} May 23 12:39:51.360281 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.common [-] Adding standard routes to API endpoints for `MethodViewType` (API Prefix: ) [/endpoints, /endpoints/] {{(pid=92057) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} May 23 12:39:51.360561 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.common [-] Adding resource routes to API endpoints: ['/endpoints//OS-ENDPOINT-POLICY/policy' {}] {{(pid=92057) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} May 23 12:39:51.361934 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.common [-] Adding standard routes to API mappings for `MethodViewType` (API Prefix: /OS-FEDERATION) [/mappings, /mappings/] {{(pid=92056) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} May 23 12:39:51.365521 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.common [-] Adding standard routes to API service_providers for `MethodViewType` (API Prefix: /OS-FEDERATION) [/service_providers, /service_providers/] {{(pid=92056) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} May 23 12:39:51.366148 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.common [-] Adding standard routes to API groups for `MethodViewType` (API Prefix: ) [/groups, /groups/] {{(pid=92057) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} May 23 12:39:51.366463 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.common [-] Adding resource routes to API groups: ['/groups//users' {}] {{(pid=92057) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} May 23 12:39:51.366770 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.common [-] Adding resource routes to API groups: ['/groups//users/' {}] {{(pid=92057) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} May 23 12:39:51.369315 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/domains//groups//roles//inherited_to_projects' {}] {{(pid=92056) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} May 23 12:39:51.369616 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/domains//groups//roles/inherited_to_projects' {}] {{(pid=92056) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} May 23 12:39:51.369929 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/domains//users//roles//inherited_to_projects' {}] {{(pid=92056) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} May 23 12:39:51.370220 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/domains//users//roles/inherited_to_projects' {}] {{(pid=92056) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} May 23 12:39:51.370509 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/projects//users//roles//inherited_to_projects' {}] {{(pid=92056) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} May 23 12:39:51.370795 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/projects//groups//roles//inherited_to_projects' {}] {{(pid=92056) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} May 23 12:39:51.374943 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.common [-] Adding standard routes to API limits for `MethodViewType` (API Prefix: ) [/limits, /limits/] {{(pid=92057) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} May 23 12:39:51.375257 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.common [-] Adding resource routes to API limits: ['/limits/model' {}] {{(pid=92057) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} May 23 12:39:51.380286 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.common [-] Adding standard routes to API OS-EP-FILTER for `MethodViewType` (API Prefix: /OS-EP-FILTER) [/endpoint_groups, /endpoint_groups/] {{(pid=92057) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} May 23 12:39:51.380571 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/endpoints//projects' {}] {{(pid=92057) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} May 23 12:39:51.380845 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/projects//endpoints/' {}] {{(pid=92057) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} May 23 12:39:51.381117 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/projects//endpoints' {}] {{(pid=92057) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} May 23 12:39:51.381389 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/projects//endpoint_groups' {}] {{(pid=92057) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} May 23 12:39:51.381761 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/endpoint_groups//endpoints' {}] {{(pid=92057) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} May 23 12:39:51.382097 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/endpoint_groups//projects' {}] {{(pid=92057) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} May 23 12:39:51.382454 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/endpoint_groups//projects/' {}] {{(pid=92057) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} May 23 12:39:51.391072 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.common [-] Adding standard routes to API OS-OAUTH1 for `MethodViewType` (API Prefix: /OS-OAUTH1) [/consumers, /consumers/] {{(pid=92056) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} May 23 12:39:51.391358 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-OAUTH1: ['/request_token' {}] {{(pid=92056) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} May 23 12:39:51.391631 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-OAUTH1: ['/access_token' {}] {{(pid=92056) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} May 23 12:39:51.391894 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-OAUTH1: ['/authorize/' {}] {{(pid=92056) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} May 23 12:39:51.399941 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.common [-] Adding resource routes to API events: ['/events' {}] {{(pid=92056) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} May 23 12:39:51.401649 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-SIMPLE-CERT: ['/OS-SIMPLE-CERT/ca' {}] {{(pid=92056) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} May 23 12:39:51.401956 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-SIMPLE-CERT: ['/OS-SIMPLE-CERT/certificates' {}] {{(pid=92056) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} May 23 12:39:51.403966 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-FEDERATION: ['/saml2/metadata' {}] {{(pid=92057) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} May 23 12:39:51.404272 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-FEDERATION: ['/identity_providers//protocols//auth' {}] {{(pid=92057) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} May 23 12:39:51.405153 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.common [-] Adding standard routes to API policy for `MethodViewType` (API Prefix: ) [/policies, /policies/] {{(pid=92056) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} May 23 12:39:51.405436 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.common [-] Adding resource routes to API policy: ['/policies//OS-ENDPOINT-POLICY/endpoints' {}] {{(pid=92056) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} May 23 12:39:51.405714 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.common [-] Adding resource routes to API policy: ['/policies//OS-ENDPOINT-POLICY/endpoints/' {}] {{(pid=92056) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} May 23 12:39:51.406016 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.common [-] Adding resource routes to API policy: ['/policies//OS-ENDPOINT-POLICY/services/' {}] {{(pid=92056) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} May 23 12:39:51.406546 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.common [-] Adding resource routes to API policy: ['/policies//OS-ENDPOINT-POLICY/services//regions/' {}] {{(pid=92056) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} May 23 12:39:51.409612 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.common [-] Adding standard routes to API identity_providers for `MethodViewType` (API Prefix: /OS-FEDERATION) [/identity_providers, /identity_providers/] {{(pid=92057) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} May 23 12:39:51.413598 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.common [-] Adding resource routes to API protocols: ['/OS-FEDERATION/identity_providers//protocols/' {}] {{(pid=92057) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} May 23 12:39:51.413957 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.common [-] Adding resource routes to API protocols: ['/OS-FEDERATION/identity_providers//protocols' {}] {{(pid=92057) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} May 23 12:39:51.419331 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.common [-] Adding standard routes to API mappings for `MethodViewType` (API Prefix: /OS-FEDERATION) [/mappings, /mappings/] {{(pid=92057) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} May 23 12:39:51.419925 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.common [-] Adding standard routes to API projects for `MethodViewType` (API Prefix: ) [/projects, /projects/] {{(pid=92056) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} May 23 12:39:51.420264 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//tags' {}] {{(pid=92056) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} May 23 12:39:51.420740 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//tags/' {}] {{(pid=92056) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} May 23 12:39:51.420826 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//users//roles/' {}] {{(pid=92056) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} May 23 12:39:51.421084 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//users//roles' {}] {{(pid=92056) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} May 23 12:39:51.421361 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//groups//roles/' {}] {{(pid=92056) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} May 23 12:39:51.421643 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//groups//roles' {}] {{(pid=92056) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} May 23 12:39:51.423511 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.common [-] Adding standard routes to API service_providers for `MethodViewType` (API Prefix: /OS-FEDERATION) [/service_providers, /service_providers/] {{(pid=92057) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} May 23 12:39:51.427136 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/domains//groups//roles//inherited_to_projects' {}] {{(pid=92057) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} May 23 12:39:51.427472 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/domains//groups//roles/inherited_to_projects' {}] {{(pid=92057) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} May 23 12:39:51.427793 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/domains//users//roles//inherited_to_projects' {}] {{(pid=92057) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} May 23 12:39:51.428236 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/domains//users//roles/inherited_to_projects' {}] {{(pid=92057) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} May 23 12:39:51.428563 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/projects//users//roles//inherited_to_projects' {}] {{(pid=92057) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} May 23 12:39:51.428911 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/projects//groups//roles//inherited_to_projects' {}] {{(pid=92057) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} May 23 12:39:51.440475 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.common [-] Adding standard routes to API regions for `MethodViewType` (API Prefix: ) [/regions, /regions/] {{(pid=92056) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} May 23 12:39:51.443761 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.common [-] Adding standard routes to API registered_limit for `MethodViewType` (API Prefix: ) [/registered_limits, /registered_limits/] {{(pid=92056) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} May 23 12:39:51.447093 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.common [-] Adding resource routes to API role_assignments: ['/role_assignments' {}] {{(pid=92056) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} May 23 12:39:51.448710 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.common [-] Adding resource routes to API role_inferences: ['/role_inferences' {}] {{(pid=92056) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} May 23 12:39:51.449048 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.common [-] Adding standard routes to API OS-OAUTH1 for `MethodViewType` (API Prefix: /OS-OAUTH1) [/consumers, /consumers/] {{(pid=92057) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} May 23 12:39:51.449325 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-OAUTH1: ['/request_token' {}] {{(pid=92057) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} May 23 12:39:51.449581 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-OAUTH1: ['/access_token' {}] {{(pid=92057) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} May 23 12:39:51.449837 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-OAUTH1: ['/authorize/' {}] {{(pid=92057) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} May 23 12:39:51.450365 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.common [-] Adding standard routes to API roles for `MethodViewType` (API Prefix: ) [/roles, /roles/] {{(pid=92056) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} May 23 12:39:51.450659 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.common [-] Adding resource routes to API roles: ['/roles//implies' {}] {{(pid=92056) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} May 23 12:39:51.450939 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.common [-] Adding resource routes to API roles: ['/roles//implies/' {}] {{(pid=92056) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} May 23 12:39:51.458370 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.common [-] Adding resource routes to API events: ['/events' {}] {{(pid=92057) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} May 23 12:39:51.459308 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.common [-] Adding resource routes to API s3tokens: ['/s3tokens' {}] {{(pid=92056) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} May 23 12:39:51.460060 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-SIMPLE-CERT: ['/OS-SIMPLE-CERT/ca' {}] {{(pid=92057) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} May 23 12:39:51.460328 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-SIMPLE-CERT: ['/OS-SIMPLE-CERT/certificates' {}] {{(pid=92057) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} May 23 12:39:51.460917 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.common [-] Adding standard routes to API services for `MethodViewType` (API Prefix: ) [/services, /services/] {{(pid=92056) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} May 23 12:39:51.463439 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.common [-] Adding standard routes to API policy for `MethodViewType` (API Prefix: ) [/policies, /policies/] {{(pid=92057) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} May 23 12:39:51.463722 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.common [-] Adding resource routes to API policy: ['/policies//OS-ENDPOINT-POLICY/endpoints' {}] {{(pid=92057) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} May 23 12:39:51.464005 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.common [-] Adding resource routes to API policy: ['/policies//OS-ENDPOINT-POLICY/endpoints/' {}] {{(pid=92057) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} May 23 12:39:51.464251 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.common [-] Adding resource routes to API system: ['/system/users//roles' {}] {{(pid=92056) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} May 23 12:39:51.464335 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.common [-] Adding resource routes to API policy: ['/policies//OS-ENDPOINT-POLICY/services/' {}] {{(pid=92057) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} May 23 12:39:51.464541 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.common [-] Adding resource routes to API system: ['/system/users//roles/' {}] {{(pid=92056) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} May 23 12:39:51.464636 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.common [-] Adding resource routes to API policy: ['/policies//OS-ENDPOINT-POLICY/services//regions/' {}] {{(pid=92057) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} May 23 12:39:51.464825 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.common [-] Adding resource routes to API system: ['/system/groups//roles' {}] {{(pid=92056) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} May 23 12:39:51.465099 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.common [-] Adding resource routes to API system: ['/system/groups//roles/' {}] {{(pid=92056) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} May 23 12:39:51.474488 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.common [-] Adding standard routes to API trusts for `MethodViewType` (API Prefix: /OS-TRUST) [/trusts, /trusts/] {{(pid=92056) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} May 23 12:39:51.474777 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.common [-] Adding resource routes to API trusts: ['/trusts//roles' {}] {{(pid=92056) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} May 23 12:39:51.475162 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.common [-] Adding resource routes to API trusts: ['/trusts//roles/' {}] {{(pid=92056) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} May 23 12:39:51.478698 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.common [-] Adding standard routes to API projects for `MethodViewType` (API Prefix: ) [/projects, /projects/] {{(pid=92057) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} May 23 12:39:51.478948 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//tags' {}] {{(pid=92057) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} May 23 12:39:51.479285 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//tags/' {}] {{(pid=92057) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} May 23 12:39:51.479563 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//users//roles/' {}] {{(pid=92057) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} May 23 12:39:51.479845 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//users//roles' {}] {{(pid=92057) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} May 23 12:39:51.480117 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//groups//roles/' {}] {{(pid=92057) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} May 23 12:39:51.480386 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//groups//roles' {}] {{(pid=92057) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} May 23 12:39:51.483093 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.common [-] Adding standard routes to API users for `MethodViewType` (API Prefix: ) [/users, /users/] {{(pid=92056) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} May 23 12:39:51.483369 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//password' {}] {{(pid=92056) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} May 23 12:39:51.483649 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//groups' {}] {{(pid=92056) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} May 23 12:39:51.483923 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//projects' {}] {{(pid=92056) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} May 23 12:39:51.484194 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//credentials/OS-EC2' {}] {{(pid=92056) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} May 23 12:39:51.484464 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//credentials/OS-EC2/' {}] {{(pid=92056) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} May 23 12:39:51.484753 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//OS-OAUTH1/access_tokens' {}] {{(pid=92056) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} May 23 12:39:51.485020 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//OS-OAUTH1/access_tokens/' {}] {{(pid=92056) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} May 23 12:39:51.485305 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//OS-OAUTH1/access_tokens//roles' {}] {{(pid=92056) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} May 23 12:39:51.485579 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//OS-OAUTH1/access_tokens//roles/' {}] {{(pid=92056) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} May 23 12:39:51.485848 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//application_credentials' {}] {{(pid=92056) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} May 23 12:39:51.486137 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//application_credentials/' {}] {{(pid=92056) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} May 23 12:39:51.486408 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//access_rules' {}] {{(pid=92056) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} May 23 12:39:51.486669 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//access_rules/' {}] {{(pid=92056) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} May 23 12:39:51.499414 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.common [-] Adding standard routes to API regions for `MethodViewType` (API Prefix: ) [/regions, /regions/] {{(pid=92057) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} May 23 12:39:51.504095 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.common [-] Adding standard routes to API registered_limit for `MethodViewType` (API Prefix: ) [/registered_limits, /registered_limits/] {{(pid=92057) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} May 23 12:39:51.507420 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.common [-] Adding resource routes to API role_assignments: ['/role_assignments' {}] {{(pid=92057) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} May 23 12:39:51.509019 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.common [-] Adding resource routes to API role_inferences: ['/role_inferences' {}] {{(pid=92057) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} May 23 12:39:51.510892 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.common [-] Adding standard routes to API roles for `MethodViewType` (API Prefix: ) [/roles, /roles/] {{(pid=92057) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} May 23 12:39:51.511196 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.common [-] Adding resource routes to API roles: ['/roles//implies' {}] {{(pid=92057) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} May 23 12:39:51.511472 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.common [-] Adding resource routes to API roles: ['/roles//implies/' {}] {{(pid=92057) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} May 23 12:39:51.520042 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.common [-] Adding resource routes to API s3tokens: ['/s3tokens' {}] {{(pid=92057) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} May 23 12:39:51.521635 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.common [-] Adding standard routes to API services for `MethodViewType` (API Prefix: ) [/services, /services/] {{(pid=92057) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} May 23 12:39:51.525100 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.common [-] Adding resource routes to API system: ['/system/users//roles' {}] {{(pid=92057) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} May 23 12:39:51.525382 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.common [-] Adding resource routes to API system: ['/system/users//roles/' {}] {{(pid=92057) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} May 23 12:39:51.525713 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.common [-] Adding resource routes to API system: ['/system/groups//roles' {}] {{(pid=92057) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} May 23 12:39:51.526053 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.common [-] Adding resource routes to API system: ['/system/groups//roles/' {}] {{(pid=92057) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} May 23 12:39:51.526597 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: WSGI app 0 (mountpoint='') ready in 6 seconds on interpreter 0x558e29d869c0 pid: 92056 (default app) May 23 12:39:51.529870 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94e6953b-4528-4277-b264-8a450665fcfb None None] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:39:51.530261 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94e6953b-4528-4277-b264-8a450665fcfb None None] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:39:51.530547 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94e6953b-4528-4277-b264-8a450665fcfb None None] PATH_INFO: `/v3` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:39:51.535121 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 1/1] 10.222.0.30 () {50 vars in 791 bytes} [Mon May 23 12:39:51 2022] GET /identity/v3/ => generated 255 bytes in 8 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) May 23 12:39:51.536136 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.common [-] Adding standard routes to API trusts for `MethodViewType` (API Prefix: /OS-TRUST) [/trusts, /trusts/] {{(pid=92057) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} May 23 12:39:51.536425 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.common [-] Adding resource routes to API trusts: ['/trusts//roles' {}] {{(pid=92057) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} May 23 12:39:51.537006 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.common [-] Adding resource routes to API trusts: ['/trusts//roles/' {}] {{(pid=92057) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} May 23 12:39:51.545337 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.common [-] Adding standard routes to API users for `MethodViewType` (API Prefix: ) [/users, /users/] {{(pid=92057) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} May 23 12:39:51.545648 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//password' {}] {{(pid=92057) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} May 23 12:39:51.546069 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//groups' {}] {{(pid=92057) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} May 23 12:39:51.546398 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//projects' {}] {{(pid=92057) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} May 23 12:39:51.546665 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//credentials/OS-EC2' {}] {{(pid=92057) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} May 23 12:39:51.546926 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//credentials/OS-EC2/' {}] {{(pid=92057) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} May 23 12:39:51.547197 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//OS-OAUTH1/access_tokens' {}] {{(pid=92057) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} May 23 12:39:51.547464 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//OS-OAUTH1/access_tokens/' {}] {{(pid=92057) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} May 23 12:39:51.547732 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//OS-OAUTH1/access_tokens//roles' {}] {{(pid=92057) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} May 23 12:39:51.547992 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//OS-OAUTH1/access_tokens//roles/' {}] {{(pid=92057) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} May 23 12:39:51.548257 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//application_credentials' {}] {{(pid=92057) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} May 23 12:39:51.548515 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//application_credentials/' {}] {{(pid=92057) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} May 23 12:39:51.548778 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//access_rules' {}] {{(pid=92057) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} May 23 12:39:51.549035 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//access_rules/' {}] {{(pid=92057) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} May 23 12:39:51.589557 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: WSGI app 0 (mountpoint='') ready in 6 seconds on interpreter 0x558e29d869c0 pid: 92057 (default app) May 23 12:40:00.425823 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1be517b0-b76b-4e58-b791-65f161ac3289 None None] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:40:00.426423 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1be517b0-b76b-4e58-b791-65f161ac3289 None None] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:40:00.426423 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1be517b0-b76b-4e58-b791-65f161ac3289 None None] PATH_INFO: `/` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:40:00.427744 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 2/2] 10.222.0.30 () {54 vars in 926 bytes} [Mon May 23 12:40:00 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) May 23 12:40:00.434966 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6022e8e4-3179-42fc-a4e1-56df7490c785 None None] REQUEST_METHOD: `POST` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:40:00.435349 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6022e8e4-3179-42fc-a4e1-56df7490c785 None None] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:40:00.435676 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6022e8e4-3179-42fc-a4e1-56df7490c785 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:40:00.503585 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: INFO dbcounter [None req-6022e8e4-3179-42fc-a4e1-56df7490c785 None None] Registered counter for database keystone May 23 12:40:00.524373 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG oslo_db.sqlalchemy.engines [None req-6022e8e4-3179-42fc-a4e1-56df7490c785 None None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=92057) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} May 23 12:40:00.531303 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writer thread running {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:99}} May 23 12:40:00.706381 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG passlib.handlers.bcrypt [None req-6022e8e4-3179-42fc-a4e1-56df7490c785 None None] detected 'bcrypt' backend, version '3.2.2' {{(pid=92057) _load_backend_mixin /usr/local/lib/python3.8/dist-packages/passlib/handlers/bcrypt.py:625}} May 23 12:40:00.706793 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG passlib.handlers.bcrypt [None req-6022e8e4-3179-42fc-a4e1-56df7490c785 None None] 'bcrypt' backend lacks $2$ support, enabling workaround {{(pid=92057) _finalize_backend_mixin /usr/local/lib/python3.8/dist-packages/passlib/handlers/bcrypt.py:406}} May 23 12:40:00.761845 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.auth.core [None req-6022e8e4-3179-42fc-a4e1-56df7490c785 None None] MFA Rules not processed for user `0f7119f8fa5c4445977671140ded91a4`. Rule list: `[]` (Enabled: `True`). {{(pid=92057) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 12:40:00.763617 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.fernet_utils [None req-6022e8e4-3179-42fc-a4e1-56df7490c785 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92057) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 12:40:00.883747 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 1/3] 10.222.0.30 () {58 vars in 1023 bytes} [Mon May 23 12:40:00 2022] POST /identity/v3/auth/tokens => generated 914 bytes in 452 msecs (HTTP/1.1 201) 6 headers in 384 bytes (1 switches on core 0) May 23 12:40:01.071633 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1be517b0-b76b-4e58-b791-65f161ac3289 None None] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:40:01.141138 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: INFO dbcounter [None req-1be517b0-b76b-4e58-b791-65f161ac3289 None None] Registered counter for database keystone May 23 12:40:01.160704 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG oslo_db.sqlalchemy.engines [None req-1be517b0-b76b-4e58-b791-65f161ac3289 None None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=92056) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} May 23 12:40:01.166929 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writer thread running {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:99}} May 23 12:40:01.299436 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1be517b0-b76b-4e58-b791-65f161ac3289 None None] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:40:01.303716 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61fe312a-a454-4fb9-88b1-ce2cfb9a9a8d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:40:01.304382 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61fe312a-a454-4fb9-88b1-ce2cfb9a9a8d admin admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:40:01.304666 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61fe312a-a454-4fb9-88b1-ce2cfb9a9a8d admin admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:40:01.304943 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61fe312a-a454-4fb9-88b1-ce2cfb9a9a8d admin admin] PATH_INFO: `/v3/auth/tokens` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:40:01.313026 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61fe312a-a454-4fb9-88b1-ce2cfb9a9a8d admin admin] RBAC: Authorizing `identity:validate_token()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:40:01.364676 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: WARNING keystone.common.rbac_enforcer.enforcer [None req-61fe312a-a454-4fb9-88b1-ce2cfb9a9a8d admin admin] Deprecated policy rules found. Use oslopolicy-policy-generator and oslopolicy-policy-upgrade to detect and resolve deprecated policies in your configuration. May 23 12:40:01.365022 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61fe312a-a454-4fb9-88b1-ce2cfb9a9a8d admin admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:40:01.374480 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 3/4] 10.222.0.30 () {58 vars in 1324 bytes} [Mon May 23 12:40:01 2022] GET /identity/v3/auth/tokens => generated 914 bytes in 303 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) May 23 12:40:01.380542 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6022e8e4-3179-42fc-a4e1-56df7490c785 None None] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:40:01.395373 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6022e8e4-3179-42fc-a4e1-56df7490c785 None None] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:40:01.399752 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c67a6dbd-bb29-4dab-9a04-fd952fc6df50 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:40:01.400886 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c67a6dbd-bb29-4dab-9a04-fd952fc6df50 admin admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:40:01.401173 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c67a6dbd-bb29-4dab-9a04-fd952fc6df50 admin admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:40:01.401451 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c67a6dbd-bb29-4dab-9a04-fd952fc6df50 admin admin] PATH_INFO: `/v3/projects/af8b84d1e3e44efba851eab6bed63282` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:40:01.406420 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c67a6dbd-bb29-4dab-9a04-fd952fc6df50 admin admin] RBAC: Authorizing `identity:get_project(project_id=af8b84d1e3e44efba851eab6bed63282)` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:40:01.456755 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: WARNING keystone.common.rbac_enforcer.enforcer [None req-c67a6dbd-bb29-4dab-9a04-fd952fc6df50 admin admin] Deprecated policy rules found. Use oslopolicy-policy-generator and oslopolicy-policy-upgrade to detect and resolve deprecated policies in your configuration. May 23 12:40:01.457282 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c67a6dbd-bb29-4dab-9a04-fd952fc6df50 admin admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:40:01.461589 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 2/5] 10.222.0.30 () {56 vars in 1207 bytes} [Mon May 23 12:40:01 2022] GET /identity/v3/projects/af8b84d1e3e44efba851eab6bed63282 => generated 344 bytes in 82 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) May 23 12:40:03.639307 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49c506fc-1f94-4ebf-b4b4-1178a85cb819 None None] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:40:03.639866 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49c506fc-1f94-4ebf-b4b4-1178a85cb819 None None] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:40:03.639866 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49c506fc-1f94-4ebf-b4b4-1178a85cb819 None None] PATH_INFO: `/` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:40:03.643938 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 4/6] 10.222.0.30 () {54 vars in 926 bytes} [Mon May 23 12:40:03 2022] GET /identity => generated 270 bytes in 7 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) May 23 12:40:03.649466 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ca4af67-7374-4409-91d9-1276f6c61861 None None] REQUEST_METHOD: `POST` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:40:03.649838 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ca4af67-7374-4409-91d9-1276f6c61861 None None] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:40:03.650081 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ca4af67-7374-4409-91d9-1276f6c61861 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:40:03.673464 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.auth.core [None req-0ca4af67-7374-4409-91d9-1276f6c61861 None None] MFA Rules not processed for user `0f7119f8fa5c4445977671140ded91a4`. Rule list: `[]` (Enabled: `True`). {{(pid=92057) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 12:40:03.674999 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.fernet_utils [None req-0ca4af67-7374-4409-91d9-1276f6c61861 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92057) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 12:40:03.682932 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 3/7] 10.222.0.30 () {58 vars in 1023 bytes} [Mon May 23 12:40:03 2022] POST /identity/v3/auth/tokens => generated 914 bytes in 35 msecs (HTTP/1.1 201) 6 headers in 384 bytes (1 switches on core 0) May 23 12:40:03.705938 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49c506fc-1f94-4ebf-b4b4-1178a85cb819 None None] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:40:03.718492 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49c506fc-1f94-4ebf-b4b4-1178a85cb819 None None] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:40:03.722773 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1b205f8-602d-46f1-8f88-11ff5ccf747e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:40:03.723403 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1b205f8-602d-46f1-8f88-11ff5ccf747e admin admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:40:03.723693 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1b205f8-602d-46f1-8f88-11ff5ccf747e admin admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:40:03.723970 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1b205f8-602d-46f1-8f88-11ff5ccf747e admin admin] PATH_INFO: `/v3/auth/tokens` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:40:03.731559 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1b205f8-602d-46f1-8f88-11ff5ccf747e admin admin] RBAC: Authorizing `identity:validate_token()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:40:03.733337 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1b205f8-602d-46f1-8f88-11ff5ccf747e admin admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:40:03.741942 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 5/8] 10.222.0.30 () {58 vars in 1324 bytes} [Mon May 23 12:40:03 2022] GET /identity/v3/auth/tokens => generated 914 bytes in 37 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) May 23 12:40:03.747356 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ca4af67-7374-4409-91d9-1276f6c61861 None None] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:40:03.756740 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ca4af67-7374-4409-91d9-1276f6c61861 None None] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:40:03.761068 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57209776-56e6-4a7a-a991-f9eeccafc563 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:40:03.761960 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57209776-56e6-4a7a-a991-f9eeccafc563 admin admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:40:03.762245 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57209776-56e6-4a7a-a991-f9eeccafc563 admin admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:40:03.762525 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57209776-56e6-4a7a-a991-f9eeccafc563 admin admin] PATH_INFO: `/v3/users/0f7119f8fa5c4445977671140ded91a4` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:40:03.764789 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57209776-56e6-4a7a-a991-f9eeccafc563 admin admin] RBAC: Authorizing `identity:get_user(user_id=0f7119f8fa5c4445977671140ded91a4)` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:40:03.766643 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57209776-56e6-4a7a-a991-f9eeccafc563 admin admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:40:03.768236 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 4/9] 10.222.0.30 () {56 vars in 1198 bytes} [Mon May 23 12:40:03 2022] GET /identity/v3/users/0f7119f8fa5c4445977671140ded91a4 => generated 247 bytes in 21 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) May 23 12:40:06.020887 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcab65b9-8b13-4311-a408-3d1bbb2f6933 None None] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:40:06.021403 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcab65b9-8b13-4311-a408-3d1bbb2f6933 None None] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:40:06.021403 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcab65b9-8b13-4311-a408-3d1bbb2f6933 None None] PATH_INFO: `/` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:40:06.022831 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 6/10] 10.222.0.30 () {54 vars in 926 bytes} [Mon May 23 12:40:06 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) May 23 12:40:06.029110 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4dc45b6a-acd9-4f8a-9916-5e52d820ebd3 None None] REQUEST_METHOD: `POST` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:40:06.029594 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4dc45b6a-acd9-4f8a-9916-5e52d820ebd3 None None] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:40:06.029725 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4dc45b6a-acd9-4f8a-9916-5e52d820ebd3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:40:06.052898 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.auth.core [None req-4dc45b6a-acd9-4f8a-9916-5e52d820ebd3 None None] MFA Rules not processed for user `0f7119f8fa5c4445977671140ded91a4`. Rule list: `[]` (Enabled: `True`). {{(pid=92057) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 12:40:06.054607 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.fernet_utils [None req-4dc45b6a-acd9-4f8a-9916-5e52d820ebd3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92057) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 12:40:06.067034 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b171813d-2b13-42bd-b31c-3492a11f688f None None] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:40:06.067312 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b171813d-2b13-42bd-b31c-3492a11f688f None None] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:40:06.067588 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b171813d-2b13-42bd-b31c-3492a11f688f None None] PATH_INFO: `/` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:40:06.068946 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 7/11] 10.222.0.30 () {54 vars in 926 bytes} [Mon May 23 12:40:06 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) May 23 12:40:06.074390 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b05cf1f7-992f-4635-94b9-849fbab81b10 None None] REQUEST_METHOD: `POST` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:40:06.074679 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b05cf1f7-992f-4635-94b9-849fbab81b10 None None] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:40:06.074957 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b05cf1f7-992f-4635-94b9-849fbab81b10 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:40:06.116138 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG passlib.handlers.bcrypt [None req-b05cf1f7-992f-4635-94b9-849fbab81b10 None None] detected 'bcrypt' backend, version '3.2.2' {{(pid=92056) _load_backend_mixin /usr/local/lib/python3.8/dist-packages/passlib/handlers/bcrypt.py:625}} May 23 12:40:06.116560 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG passlib.handlers.bcrypt [None req-b05cf1f7-992f-4635-94b9-849fbab81b10 None None] 'bcrypt' backend lacks $2$ support, enabling workaround {{(pid=92056) _finalize_backend_mixin /usr/local/lib/python3.8/dist-packages/passlib/handlers/bcrypt.py:406}} May 23 12:40:06.154748 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 5/12] 10.222.0.30 () {58 vars in 1023 bytes} [Mon May 23 12:40:06 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 127 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) May 23 12:40:06.164367 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.auth.core [None req-b05cf1f7-992f-4635-94b9-849fbab81b10 None None] MFA Rules not processed for user `0f7119f8fa5c4445977671140ded91a4`. Rule list: `[]` (Enabled: `True`). {{(pid=92056) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 12:40:06.166011 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.fernet_utils [None req-b05cf1f7-992f-4635-94b9-849fbab81b10 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92056) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 12:40:06.182688 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4dc45b6a-acd9-4f8a-9916-5e52d820ebd3 None None] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:40:06.211045 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=100 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:40:06.285125 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 8/13] 10.222.0.30 () {58 vars in 1023 bytes} [Mon May 23 12:40:06 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 211 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) May 23 12:40:06.292369 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4dc45b6a-acd9-4f8a-9916-5e52d820ebd3 None None] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:40:06.309120 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b05cf1f7-992f-4635-94b9-849fbab81b10 None None] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:40:06.327340 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa1360d4-70a5-4cdb-b26d-55979d79a301 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:40:06.328162 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa1360d4-70a5-4cdb-b26d-55979d79a301 None admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:40:06.328443 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa1360d4-70a5-4cdb-b26d-55979d79a301 None admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:40:06.328720 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa1360d4-70a5-4cdb-b26d-55979d79a301 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:40:06.342837 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa1360d4-70a5-4cdb-b26d-55979d79a301 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} May 23 12:40:06.358902 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application [None req-fa1360d4-70a5-4cdb-b26d-55979d79a301 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. May 23 12:40:06.358902 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application Traceback (most recent call last): May 23 12:40:06.358902 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request May 23 12:40:06.358902 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application rv = self.dispatch_request() May 23 12:40:06.358902 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request May 23 12:40:06.358902 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) May 23 12:40:06.358902 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper May 23 12:40:06.358902 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) May 23 12:40:06.358902 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view May 23 12:40:06.358902 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) May 23 12:40:06.358902 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request May 23 12:40:06.358902 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) May 23 12:40:06.358902 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get May 23 12:40:06.358902 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self._get_role(role_id) May 23 12:40:06.358902 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role May 23 12:40:06.358902 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') May 23 12:40:06.359953 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call May 23 12:40:06.359953 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( May 23 12:40:06.359953 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data May 23 12:40:06.359953 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) May 23 12:40:06.359953 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped May 23 12:40:06.359953 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) May 23 12:40:06.359953 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun May 23 12:40:06.359953 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) May 23 12:40:06.359953 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func May 23 12:40:06.359953 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self.get_or_create( May 23 12:40:06.359953 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create May 23 12:40:06.359953 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application with Lock( May 23 12:40:06.359953 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ May 23 12:40:06.359953 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self._enter() May 23 12:40:06.359953 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter May 23 12:40:06.359953 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) May 23 12:40:06.361359 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create May 23 12:40:06.361359 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self.creator() May 23 12:40:06.361359 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value May 23 12:40:06.361359 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application created_value = creator( May 23 12:40:06.361359 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role May 23 12:40:06.361359 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) May 23 12:40:06.361359 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role May 23 12:40:06.361359 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() May 23 12:40:06.361359 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role May 23 12:40:06.361359 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) May 23 12:40:06.361359 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. May 23 12:40:06.361359 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application May 23 12:40:06.361359 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 6/14] 10.222.0.30 () {56 vars in 1096 bytes} [Mon May 23 12:40:06 2022] GET /identity/v3/roles/admin => generated 83 bytes in 179 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) May 23 12:40:06.366696 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa1360d4-70a5-4cdb-b26d-55979d79a301 None admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:40:06.461539 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=100 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:40:06.480412 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa1360d4-70a5-4cdb-b26d-55979d79a301 None admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:40:06.521527 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-715b9c70-5076-45a8-904f-d2185144b911 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:40:06.522612 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-715b9c70-5076-45a8-904f-d2185144b911 None admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:40:06.522922 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-715b9c70-5076-45a8-904f-d2185144b911 None admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:40:06.523781 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-715b9c70-5076-45a8-904f-d2185144b911 None admin] PATH_INFO: `/v3/roles` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:40:06.524132 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-715b9c70-5076-45a8-904f-d2185144b911 None admin] RBAC: Adding query filter params (name=admin) {{(pid=92057) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} May 23 12:40:06.524482 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-715b9c70-5076-45a8-904f-d2185144b911 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:40:06.525925 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-715b9c70-5076-45a8-904f-d2185144b911 None admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:40:06.541739 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 7/15] 10.222.0.30 () {56 vars in 1099 bytes} [Mon May 23 12:40:06 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 175 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) May 23 12:40:06.548366 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-715b9c70-5076-45a8-904f-d2185144b911 None admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:40:06.549600 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b05cf1f7-992f-4635-94b9-849fbab81b10 None None] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:40:06.552135 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=100 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:40:06.586899 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad12b349-a372-4009-8173-2f6113987460 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:40:06.587793 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad12b349-a372-4009-8173-2f6113987460 None admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:40:06.588076 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad12b349-a372-4009-8173-2f6113987460 None admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:40:06.588351 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad12b349-a372-4009-8173-2f6113987460 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:40:06.663605 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad12b349-a372-4009-8173-2f6113987460 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:40:06.664247 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-715b9c70-5076-45a8-904f-d2185144b911 None admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:40:06.665130 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad12b349-a372-4009-8173-2f6113987460 None admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:40:06.716992 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b807748-5ca2-4943-9cf2-b4d3cc2a414f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:40:06.717866 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b807748-5ca2-4943-9cf2-b4d3cc2a414f None admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:40:06.718182 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b807748-5ca2-4943-9cf2-b4d3cc2a414f None admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:40:06.718459 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b807748-5ca2-4943-9cf2-b4d3cc2a414f None admin] PATH_INFO: `/v3/users/0f7119f8fa5c4445977671140ded91a4` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:40:06.719682 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b807748-5ca2-4943-9cf2-b4d3cc2a414f None admin] RBAC: Authorizing `identity:get_user(user_id=0f7119f8fa5c4445977671140ded91a4)` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:40:06.721168 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b807748-5ca2-4943-9cf2-b4d3cc2a414f None admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:40:06.722837 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 8/16] 10.222.0.30 () {56 vars in 1177 bytes} [Mon May 23 12:40:06 2022] GET /identity/v3/users/0f7119f8fa5c4445977671140ded91a4 => generated 247 bytes in 175 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) May 23 12:40:06.728072 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b807748-5ca2-4943-9cf2-b4d3cc2a414f None admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:40:06.742686 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=100 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:40:06.781452 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 9/17] 10.222.0.30 () {58 vars in 1282 bytes} [Mon May 23 12:40:06 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 473 msecs (HTTP/1.1 200) 6 headers in 358 bytes (2 switches on core 0) May 23 12:40:06.787088 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad12b349-a372-4009-8173-2f6113987460 None admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:40:06.789953 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=100 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:40:06.830275 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b807748-5ca2-4943-9cf2-b4d3cc2a414f None admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:40:06.864446 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d229f61-79b2-4458-938f-7d816bcc0da5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:40:06.865222 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d229f61-79b2-4458-938f-7d816bcc0da5 None admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:40:06.865525 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d229f61-79b2-4458-938f-7d816bcc0da5 None admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:40:06.865796 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d229f61-79b2-4458-938f-7d816bcc0da5 None admin] PATH_INFO: `/v3/domains/default` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:40:06.866992 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d229f61-79b2-4458-938f-7d816bcc0da5 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:40:06.868448 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d229f61-79b2-4458-938f-7d816bcc0da5 None admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:40:06.870244 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 9/18] 10.222.0.30 () {56 vars in 1108 bytes} [Mon May 23 12:40:06 2022] GET /identity/v3/domains/default => generated 199 bytes in 142 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) May 23 12:40:06.875677 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d229f61-79b2-4458-938f-7d816bcc0da5 None admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:40:06.883860 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad12b349-a372-4009-8173-2f6113987460 None admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:40:06.917465 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-096419fa-39e8-4764-a6e8-e80f444d917a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:40:06.918310 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-096419fa-39e8-4764-a6e8-e80f444d917a None admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:40:06.918593 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-096419fa-39e8-4764-a6e8-e80f444d917a None admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:40:06.918868 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-096419fa-39e8-4764-a6e8-e80f444d917a None admin] PATH_INFO: `/v3/domains/Default` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:40:06.940572 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-096419fa-39e8-4764-a6e8-e80f444d917a None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:40:06.942112 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-096419fa-39e8-4764-a6e8-e80f444d917a None admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:40:06.964740 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application [None req-096419fa-39e8-4764-a6e8-e80f444d917a None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. May 23 12:40:06.964740 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application Traceback (most recent call last): May 23 12:40:06.964740 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request May 23 12:40:06.964740 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application rv = self.dispatch_request() May 23 12:40:06.964740 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request May 23 12:40:06.964740 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) May 23 12:40:06.964740 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper May 23 12:40:06.964740 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) May 23 12:40:06.964740 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view May 23 12:40:06.964740 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) May 23 12:40:06.964740 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request May 23 12:40:06.964740 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) May 23 12:40:06.964740 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get May 23 12:40:06.964740 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self._get_domain(domain_id) May 23 12:40:06.964740 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain May 23 12:40:06.965924 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) May 23 12:40:06.965924 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped May 23 12:40:06.965924 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) May 23 12:40:06.965924 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun May 23 12:40:06.965924 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) May 23 12:40:06.965924 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func May 23 12:40:06.965924 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self.get_or_create( May 23 12:40:06.965924 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create May 23 12:40:06.965924 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application with Lock( May 23 12:40:06.965924 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ May 23 12:40:06.965924 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self._enter() May 23 12:40:06.965924 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter May 23 12:40:06.965924 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) May 23 12:40:06.965924 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create May 23 12:40:06.965924 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self.creator() May 23 12:40:06.965924 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value May 23 12:40:06.965924 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application created_value = creator( May 23 12:40:06.967617 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain May 23 12:40:06.967617 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) May 23 12:40:06.967617 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. May 23 12:40:06.967617 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application May 23 12:40:06.967617 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 10/19] 10.222.0.30 () {56 vars in 1108 bytes} [Mon May 23 12:40:06 2022] GET /identity/v3/domains/Default => generated 87 bytes in 180 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) May 23 12:40:06.973565 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-096419fa-39e8-4764-a6e8-e80f444d917a None admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:40:07.035274 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=100 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:40:07.069348 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=100 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:40:07.079441 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-096419fa-39e8-4764-a6e8-e80f444d917a None admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:40:07.090632 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d229f61-79b2-4458-938f-7d816bcc0da5 None admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:40:07.112496 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb628ff5-c7aa-4f25-96e6-599dbe581345 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:40:07.113160 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb628ff5-c7aa-4f25-96e6-599dbe581345 None admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:40:07.113442 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb628ff5-c7aa-4f25-96e6-599dbe581345 None admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:40:07.113726 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb628ff5-c7aa-4f25-96e6-599dbe581345 None admin] PATH_INFO: `/v3/domains` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:40:07.114466 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb628ff5-c7aa-4f25-96e6-599dbe581345 None admin] RBAC: Adding query filter params (name=Default) {{(pid=92056) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} May 23 12:40:07.114772 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb628ff5-c7aa-4f25-96e6-599dbe581345 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:40:07.116153 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb628ff5-c7aa-4f25-96e6-599dbe581345 None admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:40:07.124526 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9adc03e-87a4-4214-9f31-c81831669d99 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:40:07.125177 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9adc03e-87a4-4214-9f31-c81831669d99 None admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:40:07.125462 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9adc03e-87a4-4214-9f31-c81831669d99 None admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:40:07.125738 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9adc03e-87a4-4214-9f31-c81831669d99 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:40:07.126610 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9adc03e-87a4-4214-9f31-c81831669d99 None admin] RBAC: Adding query filter params (role.id=bc3c3c9d654940cbaa5177a6be23d1bc, scope.domain.id=default, user.id=0f7119f8fa5c4445977671140ded91a4) {{(pid=92057) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} May 23 12:40:07.126931 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9adc03e-87a4-4214-9f31-c81831669d99 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:40:07.128355 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9adc03e-87a4-4214-9f31-c81831669d99 None admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:40:07.133544 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 11/20] 10.222.0.30 () {56 vars in 1109 bytes} [Mon May 23 12:40:06 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 161 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) May 23 12:40:07.134943 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 10/21] 10.222.0.30 () {56 vars in 1322 bytes} [Mon May 23 12:40:06 2022] GET /identity/v3/role_assignments?user.id=0f7119f8fa5c4445977671140ded91a4&scope.domain.id=default&role.id=bc3c3c9d654940cbaa5177a6be23d1bc => generated 233 bytes in 260 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) May 23 12:40:09.555521 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42e4e1b8-1ed6-4fdf-82d6-a74a1fb8dc0d None None] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:40:09.556059 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42e4e1b8-1ed6-4fdf-82d6-a74a1fb8dc0d None None] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:40:09.556059 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42e4e1b8-1ed6-4fdf-82d6-a74a1fb8dc0d None None] PATH_INFO: `/` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:40:09.557739 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 12/22] 10.222.0.30 () {54 vars in 926 bytes} [Mon May 23 12:40:09 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) May 23 12:40:09.563729 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd320105-a30a-4bdb-90b3-af7441634290 None None] REQUEST_METHOD: `POST` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:40:09.564162 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd320105-a30a-4bdb-90b3-af7441634290 None None] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:40:09.564482 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd320105-a30a-4bdb-90b3-af7441634290 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:40:09.592416 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.auth.core [None req-bd320105-a30a-4bdb-90b3-af7441634290 None None] MFA Rules not processed for user `0f7119f8fa5c4445977671140ded91a4`. Rule list: `[]` (Enabled: `True`). {{(pid=92057) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 12:40:09.593648 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.fernet_utils [None req-bd320105-a30a-4bdb-90b3-af7441634290 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92057) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 12:40:09.680876 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 11/23] 10.222.0.30 () {58 vars in 1023 bytes} [Mon May 23 12:40:09 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 118 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) May 23 12:40:09.740855 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42e4e1b8-1ed6-4fdf-82d6-a74a1fb8dc0d None None] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:40:09.842158 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc2f27c9-7bdc-4802-a876-b379cef00e4f None None] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:40:09.842344 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc2f27c9-7bdc-4802-a876-b379cef00e4f None None] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:40:09.842617 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc2f27c9-7bdc-4802-a876-b379cef00e4f None None] PATH_INFO: `/` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:40:09.844017 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 12/24] 10.222.0.30 () {54 vars in 926 bytes} [Mon May 23 12:40:09 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) May 23 12:40:09.850010 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab10768c-596c-4bea-a0d5-20216dbf2954 None None] REQUEST_METHOD: `POST` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:40:09.850440 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab10768c-596c-4bea-a0d5-20216dbf2954 None None] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:40:09.850818 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab10768c-596c-4bea-a0d5-20216dbf2954 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:40:09.886949 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.auth.core [None req-ab10768c-596c-4bea-a0d5-20216dbf2954 None None] MFA Rules not processed for user `0f7119f8fa5c4445977671140ded91a4`. Rule list: `[]` (Enabled: `True`). {{(pid=92057) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 12:40:09.888868 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.fernet_utils [None req-ab10768c-596c-4bea-a0d5-20216dbf2954 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92057) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 12:40:09.918573 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42e4e1b8-1ed6-4fdf-82d6-a74a1fb8dc0d None None] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:40:09.953183 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=100 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:40:09.996694 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-448cfc92-db2f-49b3-a987-dd8b0966c732 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:40:09.998733 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-448cfc92-db2f-49b3-a987-dd8b0966c732 None admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:40:09.999188 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-448cfc92-db2f-49b3-a987-dd8b0966c732 None admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:40:09.999598 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-448cfc92-db2f-49b3-a987-dd8b0966c732 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:40:10.019099 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-448cfc92-db2f-49b3-a987-dd8b0966c732 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:40:10.020767 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-448cfc92-db2f-49b3-a987-dd8b0966c732 None admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:40:10.048683 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 13/25] 10.222.0.30 () {58 vars in 1023 bytes} [Mon May 23 12:40:09 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 200 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) May 23 12:40:10.056703 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application [None req-448cfc92-db2f-49b3-a987-dd8b0966c732 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. May 23 12:40:10.056703 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application Traceback (most recent call last): May 23 12:40:10.056703 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request May 23 12:40:10.056703 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application rv = self.dispatch_request() May 23 12:40:10.056703 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request May 23 12:40:10.056703 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) May 23 12:40:10.056703 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper May 23 12:40:10.056703 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) May 23 12:40:10.056703 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view May 23 12:40:10.056703 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) May 23 12:40:10.056703 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request May 23 12:40:10.056703 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) May 23 12:40:10.056703 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get May 23 12:40:10.056703 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self._get_domain(domain_id) May 23 12:40:10.056703 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain May 23 12:40:10.057641 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) May 23 12:40:10.057641 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped May 23 12:40:10.057641 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) May 23 12:40:10.057641 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun May 23 12:40:10.057641 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) May 23 12:40:10.057641 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func May 23 12:40:10.057641 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self.get_or_create( May 23 12:40:10.057641 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create May 23 12:40:10.057641 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application with Lock( May 23 12:40:10.057641 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ May 23 12:40:10.057641 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self._enter() May 23 12:40:10.057641 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter May 23 12:40:10.057641 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) May 23 12:40:10.057641 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create May 23 12:40:10.057641 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self.creator() May 23 12:40:10.057641 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value May 23 12:40:10.057641 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application created_value = creator( May 23 12:40:10.058504 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain May 23 12:40:10.058504 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) May 23 12:40:10.058504 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. May 23 12:40:10.058504 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application May 23 12:40:10.066116 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 13/26] 10.222.0.30 () {56 vars in 1108 bytes} [Mon May 23 12:40:09 2022] GET /identity/v3/domains/Default => generated 87 bytes in 326 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) May 23 12:40:10.072213 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab10768c-596c-4bea-a0d5-20216dbf2954 None None] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:40:10.077111 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=100 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:40:10.078084 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-448cfc92-db2f-49b3-a987-dd8b0966c732 None admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:40:10.283607 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-448cfc92-db2f-49b3-a987-dd8b0966c732 None admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:40:10.296011 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab10768c-596c-4bea-a0d5-20216dbf2954 None None] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:40:10.337330 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9f19749-d4bf-4d9d-9be1-15fcfb6dca9e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:40:10.338030 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9f19749-d4bf-4d9d-9be1-15fcfb6dca9e None admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:40:10.338451 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9f19749-d4bf-4d9d-9be1-15fcfb6dca9e None admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:40:10.338738 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9f19749-d4bf-4d9d-9be1-15fcfb6dca9e None admin] PATH_INFO: `/v3/domains` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:40:10.339473 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9f19749-d4bf-4d9d-9be1-15fcfb6dca9e None admin] RBAC: Adding query filter params (name=Default) {{(pid=92057) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} May 23 12:40:10.339785 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9f19749-d4bf-4d9d-9be1-15fcfb6dca9e None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:40:10.341975 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9f19749-d4bf-4d9d-9be1-15fcfb6dca9e None admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:40:10.361138 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-882eef06-7fd4-4426-8a41-237999ff73fd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:40:10.361954 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-882eef06-7fd4-4426-8a41-237999ff73fd None admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:40:10.362236 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-882eef06-7fd4-4426-8a41-237999ff73fd None admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:40:10.362505 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-882eef06-7fd4-4426-8a41-237999ff73fd None admin] PATH_INFO: `/v3/roles/admin` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:40:10.368544 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 14/27] 10.222.0.30 () {56 vars in 1109 bytes} [Mon May 23 12:40:10 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 297 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) May 23 12:40:10.369984 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edec39c5-f7bd-401f-abc6-ec02303113c4 None None] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:40:10.370289 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edec39c5-f7bd-401f-abc6-ec02303113c4 None None] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:40:10.370571 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edec39c5-f7bd-401f-abc6-ec02303113c4 None None] PATH_INFO: `/` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:40:10.371826 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 15/28] 10.222.0.30 () {54 vars in 926 bytes} [Mon May 23 12:40:10 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) May 23 12:40:10.376502 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-882eef06-7fd4-4426-8a41-237999ff73fd None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} May 23 12:40:10.378774 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-edec39c5-f7bd-401f-abc6-ec02303113c4 None None] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:40:10.386525 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application [None req-882eef06-7fd4-4426-8a41-237999ff73fd None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. May 23 12:40:10.386525 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application Traceback (most recent call last): May 23 12:40:10.386525 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request May 23 12:40:10.386525 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application rv = self.dispatch_request() May 23 12:40:10.386525 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request May 23 12:40:10.386525 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) May 23 12:40:10.386525 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper May 23 12:40:10.386525 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) May 23 12:40:10.386525 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view May 23 12:40:10.386525 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) May 23 12:40:10.386525 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request May 23 12:40:10.386525 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) May 23 12:40:10.386525 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get May 23 12:40:10.386525 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self._get_role(role_id) May 23 12:40:10.386525 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role May 23 12:40:10.386525 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') May 23 12:40:10.388136 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call May 23 12:40:10.388136 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( May 23 12:40:10.388136 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data May 23 12:40:10.388136 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) May 23 12:40:10.388136 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped May 23 12:40:10.388136 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) May 23 12:40:10.388136 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun May 23 12:40:10.388136 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) May 23 12:40:10.388136 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func May 23 12:40:10.388136 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self.get_or_create( May 23 12:40:10.388136 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create May 23 12:40:10.388136 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application with Lock( May 23 12:40:10.388136 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ May 23 12:40:10.388136 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self._enter() May 23 12:40:10.388136 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter May 23 12:40:10.388136 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) May 23 12:40:10.389176 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create May 23 12:40:10.389176 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self.creator() May 23 12:40:10.389176 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value May 23 12:40:10.389176 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application created_value = creator( May 23 12:40:10.389176 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role May 23 12:40:10.389176 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) May 23 12:40:10.389176 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role May 23 12:40:10.389176 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() May 23 12:40:10.389176 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role May 23 12:40:10.389176 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) May 23 12:40:10.389176 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. May 23 12:40:10.389176 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application May 23 12:40:10.395627 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 14/29] 10.222.0.30 () {56 vars in 1096 bytes} [Mon May 23 12:40:10 2022] GET /identity/v3/roles/admin => generated 83 bytes in 315 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) May 23 12:40:10.397205 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cb7cb5c-5330-4e35-a51f-07a58bda3dfa None None] REQUEST_METHOD: `POST` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:40:10.397630 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cb7cb5c-5330-4e35-a51f-07a58bda3dfa None None] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:40:10.398067 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cb7cb5c-5330-4e35-a51f-07a58bda3dfa None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:40:10.435563 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.auth.core [None req-5cb7cb5c-5330-4e35-a51f-07a58bda3dfa None None] MFA Rules not processed for user `0f7119f8fa5c4445977671140ded91a4`. Rule list: `[]` (Enabled: `True`). {{(pid=92056) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 12:40:10.437228 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.fernet_utils [None req-5cb7cb5c-5330-4e35-a51f-07a58bda3dfa None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92056) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 12:40:10.485319 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=100 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:40:10.510157 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=100 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:40:10.543718 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-edec39c5-f7bd-401f-abc6-ec02303113c4 None None] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:40:10.574873 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 15/30] 10.222.0.30 () {58 vars in 1023 bytes} [Mon May 23 12:40:10 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 178 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) May 23 12:40:10.575930 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5cb7cb5c-5330-4e35-a51f-07a58bda3dfa None None] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:40:10.590254 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d81edd9-3210-4a70-8405-1140c271a687 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:40:10.591911 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d81edd9-3210-4a70-8405-1140c271a687 None admin] REQUEST_METHOD: `POST` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:40:10.592782 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d81edd9-3210-4a70-8405-1140c271a687 None admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:40:10.593357 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d81edd9-3210-4a70-8405-1140c271a687 None admin] PATH_INFO: `/v3/projects` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:40:10.594340 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d81edd9-3210-4a70-8405-1140c271a687 None admin] RBAC: Authorizing `identity:create_project()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:40:10.596662 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d81edd9-3210-4a70-8405-1140c271a687 None admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:40:10.689148 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5cb7cb5c-5330-4e35-a51f-07a58bda3dfa None None] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:40:10.734500 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99e03803-57f6-4dea-9f01-66e877d3ef7f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:40:10.735715 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99e03803-57f6-4dea-9f01-66e877d3ef7f None admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:40:10.736224 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99e03803-57f6-4dea-9f01-66e877d3ef7f None admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:40:10.736652 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99e03803-57f6-4dea-9f01-66e877d3ef7f None admin] PATH_INFO: `/v3/roles` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:40:10.737656 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99e03803-57f6-4dea-9f01-66e877d3ef7f None admin] RBAC: Adding query filter params (name=admin) {{(pid=92056) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} May 23 12:40:10.738162 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99e03803-57f6-4dea-9f01-66e877d3ef7f None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:40:10.739821 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99e03803-57f6-4dea-9f01-66e877d3ef7f None admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:40:10.753936 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 16/31] 10.222.0.30 () {56 vars in 1099 bytes} [Mon May 23 12:40:10 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 179 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) May 23 12:40:10.755539 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e027a5e5-8267-4c4b-afb9-4d48dcd016b4 None None] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:40:10.756067 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e027a5e5-8267-4c4b-afb9-4d48dcd016b4 None None] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:40:10.756535 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e027a5e5-8267-4c4b-afb9-4d48dcd016b4 None None] PATH_INFO: `/` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:40:10.758472 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 17/32] 10.222.0.30 () {54 vars in 926 bytes} [Mon May 23 12:40:10 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) May 23 12:40:10.764105 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e027a5e5-8267-4c4b-afb9-4d48dcd016b4 None None] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:40:10.788588 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=100 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:40:10.843401 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 16/33] 10.222.0.30 () {60 vars in 1141 bytes} [Mon May 23 12:40:10 2022] POST /identity/v3/projects => generated 301 bytes in 465 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) May 23 12:40:10.845100 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2c21b03-5972-49e9-8dd1-38f6ab38f1a6 None None] REQUEST_METHOD: `POST` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:40:10.845394 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2c21b03-5972-49e9-8dd1-38f6ab38f1a6 None None] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:40:10.845676 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2c21b03-5972-49e9-8dd1-38f6ab38f1a6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:40:10.864902 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.auth.core [None req-a2c21b03-5972-49e9-8dd1-38f6ab38f1a6 None None] MFA Rules not processed for user `0f7119f8fa5c4445977671140ded91a4`. Rule list: `[]` (Enabled: `True`). {{(pid=92057) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 12:40:10.866438 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.fernet_utils [None req-a2c21b03-5972-49e9-8dd1-38f6ab38f1a6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92057) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 12:40:10.891675 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e027a5e5-8267-4c4b-afb9-4d48dcd016b4 None None] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:40:10.949402 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5f57a0f-e599-45ac-a2ec-bb671364e306 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:40:10.951231 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5f57a0f-e599-45ac-a2ec-bb671364e306 None admin] REQUEST_METHOD: `POST` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:40:10.953284 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5f57a0f-e599-45ac-a2ec-bb671364e306 None admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:40:10.953861 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5f57a0f-e599-45ac-a2ec-bb671364e306 None admin] PATH_INFO: `/v3/roles` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:40:10.955000 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a5f57a0f-e599-45ac-a2ec-bb671364e306 None admin] RBAC: Authorizing `identity:create_role()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:40:10.959936 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a5f57a0f-e599-45ac-a2ec-bb671364e306 None admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:40:10.981963 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 17/34] 10.222.0.30 () {58 vars in 1023 bytes} [Mon May 23 12:40:10 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 138 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) May 23 12:40:10.988987 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2c21b03-5972-49e9-8dd1-38f6ab38f1a6 None None] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:40:11.063662 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:40:11.132985 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2c21b03-5972-49e9-8dd1-38f6ab38f1a6 None None] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:40:11.170605 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-028695af-1fb3-4e42-ade7-5954499a9bd7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:40:11.171856 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-028695af-1fb3-4e42-ade7-5954499a9bd7 None admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:40:11.172344 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-028695af-1fb3-4e42-ade7-5954499a9bd7 None admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:40:11.172809 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-028695af-1fb3-4e42-ade7-5954499a9bd7 None admin] PATH_INFO: `/v3/users/0f7119f8fa5c4445977671140ded91a4` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:40:11.176605 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-028695af-1fb3-4e42-ade7-5954499a9bd7 None admin] RBAC: Authorizing `identity:get_user(user_id=0f7119f8fa5c4445977671140ded91a4)` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:40:11.176864 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-028695af-1fb3-4e42-ade7-5954499a9bd7 None admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:40:11.178840 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 18/35] 10.222.0.30 () {56 vars in 1177 bytes} [Mon May 23 12:40:10 2022] GET /identity/v3/users/0f7119f8fa5c4445977671140ded91a4 => generated 247 bytes in 196 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) May 23 12:40:11.179478 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-028695af-1fb3-4e42-ade7-5954499a9bd7 None admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:40:11.206963 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 18/36] 10.222.0.30 () {60 vars in 1131 bytes} [Mon May 23 12:40:10 2022] POST /identity/v3/roles => generated 219 bytes in 443 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) May 23 12:40:11.208505 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74e030cb-18ed-4ee4-bf07-cee5dc60de15 None None] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:40:11.208802 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74e030cb-18ed-4ee4-bf07-cee5dc60de15 None None] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:40:11.209082 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74e030cb-18ed-4ee4-bf07-cee5dc60de15 None None] PATH_INFO: `/` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:40:11.210439 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 19/37] 10.222.0.30 () {54 vars in 926 bytes} [Mon May 23 12:40:11 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) May 23 12:40:11.211185 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74e030cb-18ed-4ee4-bf07-cee5dc60de15 None None] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:40:11.308970 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-028695af-1fb3-4e42-ade7-5954499a9bd7 None admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:40:11.314078 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:40:11.336758 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74e030cb-18ed-4ee4-bf07-cee5dc60de15 None None] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:40:11.346303 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=100 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:40:11.352944 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-928e5f49-ae0b-427d-8a56-80cd515a3669 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:40:11.354067 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-928e5f49-ae0b-427d-8a56-80cd515a3669 None admin] REQUEST_METHOD: `POST` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:40:11.354384 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-928e5f49-ae0b-427d-8a56-80cd515a3669 None admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:40:11.354667 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-928e5f49-ae0b-427d-8a56-80cd515a3669 None admin] PATH_INFO: `/v3/roles` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:40:11.355752 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-928e5f49-ae0b-427d-8a56-80cd515a3669 None admin] RBAC: Authorizing `identity:create_role()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:40:11.359846 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-928e5f49-ae0b-427d-8a56-80cd515a3669 None admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:40:11.378355 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57a2c3ef-612f-4169-81b3-8a08bc3e4945 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:40:11.379263 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57a2c3ef-612f-4169-81b3-8a08bc3e4945 None admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:40:11.379651 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57a2c3ef-612f-4169-81b3-8a08bc3e4945 None admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:40:11.380046 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57a2c3ef-612f-4169-81b3-8a08bc3e4945 None admin] PATH_INFO: `/v3/domains/default` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:40:11.381707 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57a2c3ef-612f-4169-81b3-8a08bc3e4945 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:40:11.383478 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57a2c3ef-612f-4169-81b3-8a08bc3e4945 None admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:40:11.390081 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 20/38] 10.222.0.30 () {56 vars in 1108 bytes} [Mon May 23 12:40:11 2022] GET /identity/v3/domains/default => generated 199 bytes in 174 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) May 23 12:40:11.390081 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-439da47f-84b9-4fab-b5e8-24a711e28d64 None None] REQUEST_METHOD: `POST` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:40:11.390081 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-439da47f-84b9-4fab-b5e8-24a711e28d64 None None] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:40:11.390081 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-439da47f-84b9-4fab-b5e8-24a711e28d64 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:40:11.408617 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.auth.core [None req-439da47f-84b9-4fab-b5e8-24a711e28d64 None None] MFA Rules not processed for user `0f7119f8fa5c4445977671140ded91a4`. Rule list: `[]` (Enabled: `True`). {{(pid=92056) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 12:40:11.410183 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.fernet_utils [None req-439da47f-84b9-4fab-b5e8-24a711e28d64 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92056) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 12:40:11.456595 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 19/39] 10.222.0.30 () {60 vars in 1131 bytes} [Mon May 23 12:40:11 2022] POST /identity/v3/roles => generated 225 bytes in 277 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) May 23 12:40:11.457467 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-928e5f49-ae0b-427d-8a56-80cd515a3669 None admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:40:11.503141 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 21/40] 10.222.0.30 () {58 vars in 1023 bytes} [Mon May 23 12:40:11 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 117 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) May 23 12:40:11.527764 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-439da47f-84b9-4fab-b5e8-24a711e28d64 None None] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:40:11.612222 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-928e5f49-ae0b-427d-8a56-80cd515a3669 None admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:40:11.647489 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=100 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:40:11.651482 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee3c2a37-bda8-49c3-93c5-fa540960b617 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:40:11.652365 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee3c2a37-bda8-49c3-93c5-fa540960b617 None admin] REQUEST_METHOD: `PUT` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:40:11.652653 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee3c2a37-bda8-49c3-93c5-fa540960b617 None admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:40:11.652939 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee3c2a37-bda8-49c3-93c5-fa540960b617 None admin] PATH_INFO: `/v3/domains/default/users/0f7119f8fa5c4445977671140ded91a4/roles/bc3c3c9d654940cbaa5177a6be23d1bc` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:40:11.656338 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee3c2a37-bda8-49c3-93c5-fa540960b617 None admin] RBAC: Authorizing `identity:create_grant(domain_id=default, user_id=0f7119f8fa5c4445977671140ded91a4, role_id=bc3c3c9d654940cbaa5177a6be23d1bc)` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:40:11.657800 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee3c2a37-bda8-49c3-93c5-fa540960b617 None admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:40:11.672974 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-439da47f-84b9-4fab-b5e8-24a711e28d64 None None] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:40:11.711689 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76fef56b-27b9-4ee4-8213-d5ad8c1ccd74 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:40:11.712842 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76fef56b-27b9-4ee4-8213-d5ad8c1ccd74 None admin] REQUEST_METHOD: `POST` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:40:11.713368 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76fef56b-27b9-4ee4-8213-d5ad8c1ccd74 None admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:40:11.713905 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76fef56b-27b9-4ee4-8213-d5ad8c1ccd74 None admin] PATH_INFO: `/v3/roles` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:40:11.715397 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76fef56b-27b9-4ee4-8213-d5ad8c1ccd74 None admin] RBAC: Authorizing `identity:create_role()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:40:11.718127 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76fef56b-27b9-4ee4-8213-d5ad8c1ccd74 None admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:40:11.775820 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 20/41] 10.222.0.30 () {58 vars in 1361 bytes} [Mon May 23 12:40:11 2022] PUT /identity/v3/domains/default/users/0f7119f8fa5c4445977671140ded91a4/roles/bc3c3c9d654940cbaa5177a6be23d1bc => generated 0 bytes in 319 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) May 23 12:40:11.777235 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9667f47-098c-41db-b410-937e05f38137 None None] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:40:11.777529 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9667f47-098c-41db-b410-937e05f38137 None None] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:40:11.777817 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9667f47-098c-41db-b410-937e05f38137 None None] PATH_INFO: `/` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:40:11.779280 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 21/42] 10.222.0.30 () {54 vars in 926 bytes} [Mon May 23 12:40:11 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) May 23 12:40:11.785291 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1017e3e-da0e-4cd7-9fe8-f2b6c6d92a4b None None] REQUEST_METHOD: `POST` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:40:11.785585 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1017e3e-da0e-4cd7-9fe8-f2b6c6d92a4b None None] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:40:11.785890 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1017e3e-da0e-4cd7-9fe8-f2b6c6d92a4b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:40:11.804363 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.auth.core [None req-b1017e3e-da0e-4cd7-9fe8-f2b6c6d92a4b None None] MFA Rules not processed for user `0f7119f8fa5c4445977671140ded91a4`. Rule list: `[]` (Enabled: `True`). {{(pid=92057) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 12:40:11.805426 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.fernet_utils [None req-b1017e3e-da0e-4cd7-9fe8-f2b6c6d92a4b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92057) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 12:40:11.808709 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 22/43] 10.222.0.30 () {60 vars in 1131 bytes} [Mon May 23 12:40:11 2022] POST /identity/v3/roles => generated 223 bytes in 279 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) May 23 12:40:11.843552 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:40:11.890631 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 22/44] 10.222.0.30 () {58 vars in 1023 bytes} [Mon May 23 12:40:11 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 106 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) May 23 12:40:11.913824 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76fef56b-27b9-4ee4-8213-d5ad8c1ccd74 None admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:40:12.031107 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76fef56b-27b9-4ee4-8213-d5ad8c1ccd74 None admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:40:12.068675 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-adb9d98f-bc80-4d8d-afa4-30d8b7f4612d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:40:12.069738 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-adb9d98f-bc80-4d8d-afa4-30d8b7f4612d None admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:40:12.070184 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-adb9d98f-bc80-4d8d-afa4-30d8b7f4612d None admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:40:12.070476 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-adb9d98f-bc80-4d8d-afa4-30d8b7f4612d None admin] PATH_INFO: `/v3/domains/default` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:40:12.071677 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-adb9d98f-bc80-4d8d-afa4-30d8b7f4612d None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:40:12.073144 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-adb9d98f-bc80-4d8d-afa4-30d8b7f4612d None admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:40:12.075015 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 23/45] 10.222.0.30 () {56 vars in 1108 bytes} [Mon May 23 12:40:11 2022] GET /identity/v3/domains/default => generated 199 bytes in 162 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) May 23 12:40:12.080471 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1017e3e-da0e-4cd7-9fe8-f2b6c6d92a4b None None] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:40:12.177498 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1017e3e-da0e-4cd7-9fe8-f2b6c6d92a4b None None] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:40:12.210406 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8dc24055-bd2c-435d-b61a-bcbfe96a588d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:40:12.211065 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8dc24055-bd2c-435d-b61a-bcbfe96a588d None admin] REQUEST_METHOD: `POST` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:40:12.211350 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8dc24055-bd2c-435d-b61a-bcbfe96a588d None admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:40:12.211630 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8dc24055-bd2c-435d-b61a-bcbfe96a588d None admin] PATH_INFO: `/v3/projects` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:40:12.212347 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8dc24055-bd2c-435d-b61a-bcbfe96a588d None admin] RBAC: Authorizing `identity:create_project()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:40:12.213797 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8dc24055-bd2c-435d-b61a-bcbfe96a588d None admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:40:12.297007 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 23/46] 10.222.0.30 () {60 vars in 1141 bytes} [Mon May 23 12:40:12 2022] POST /identity/v3/projects => generated 312 bytes in 217 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) May 23 12:40:14.009773 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83c9a8ce-7a23-4ccf-9e04-05f490d7b6b1 None None] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:40:14.009773 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83c9a8ce-7a23-4ccf-9e04-05f490d7b6b1 None None] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:40:14.009773 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83c9a8ce-7a23-4ccf-9e04-05f490d7b6b1 None None] PATH_INFO: `/` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:40:14.010948 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 24/47] 10.222.0.30 () {54 vars in 926 bytes} [Mon May 23 12:40:14 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) May 23 12:40:14.016432 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e43d44d1-0330-4477-9f3d-e3b5ec40bca0 None None] REQUEST_METHOD: `POST` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:40:14.016836 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e43d44d1-0330-4477-9f3d-e3b5ec40bca0 None None] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:40:14.017058 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e43d44d1-0330-4477-9f3d-e3b5ec40bca0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:40:14.034581 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.auth.core [None req-e43d44d1-0330-4477-9f3d-e3b5ec40bca0 None None] MFA Rules not processed for user `0f7119f8fa5c4445977671140ded91a4`. Rule list: `[]` (Enabled: `True`). {{(pid=92057) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 12:40:14.035749 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.fernet_utils [None req-e43d44d1-0330-4477-9f3d-e3b5ec40bca0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92057) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 12:40:14.046672 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:40:14.102395 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 24/48] 10.222.0.30 () {58 vars in 1023 bytes} [Mon May 23 12:40:14 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 87 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) May 23 12:40:14.126401 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83c9a8ce-7a23-4ccf-9e04-05f490d7b6b1 None None] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:40:14.135832 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:40:14.229820 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83c9a8ce-7a23-4ccf-9e04-05f490d7b6b1 None None] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:40:14.263038 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06cb871f-a863-47ac-a209-db28638de442 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:40:14.263754 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06cb871f-a863-47ac-a209-db28638de442 None admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:40:14.264043 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06cb871f-a863-47ac-a209-db28638de442 None admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:40:14.264325 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06cb871f-a863-47ac-a209-db28638de442 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:40:14.274294 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06cb871f-a863-47ac-a209-db28638de442 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} May 23 12:40:14.282784 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application [None req-06cb871f-a863-47ac-a209-db28638de442 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. May 23 12:40:14.282784 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application Traceback (most recent call last): May 23 12:40:14.282784 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request May 23 12:40:14.282784 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application rv = self.dispatch_request() May 23 12:40:14.282784 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request May 23 12:40:14.282784 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) May 23 12:40:14.282784 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper May 23 12:40:14.282784 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) May 23 12:40:14.282784 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view May 23 12:40:14.282784 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) May 23 12:40:14.282784 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request May 23 12:40:14.282784 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) May 23 12:40:14.282784 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get May 23 12:40:14.282784 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self._get_role(role_id) May 23 12:40:14.282784 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role May 23 12:40:14.282784 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') May 23 12:40:14.283740 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call May 23 12:40:14.283740 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( May 23 12:40:14.283740 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data May 23 12:40:14.283740 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) May 23 12:40:14.283740 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped May 23 12:40:14.283740 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) May 23 12:40:14.283740 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun May 23 12:40:14.283740 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) May 23 12:40:14.283740 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func May 23 12:40:14.283740 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self.get_or_create( May 23 12:40:14.283740 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create May 23 12:40:14.283740 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application with Lock( May 23 12:40:14.283740 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ May 23 12:40:14.283740 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self._enter() May 23 12:40:14.283740 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter May 23 12:40:14.283740 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) May 23 12:40:14.284518 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create May 23 12:40:14.284518 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self.creator() May 23 12:40:14.284518 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value May 23 12:40:14.284518 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application created_value = creator( May 23 12:40:14.284518 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role May 23 12:40:14.284518 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) May 23 12:40:14.284518 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role May 23 12:40:14.284518 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() May 23 12:40:14.284518 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role May 23 12:40:14.284518 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) May 23 12:40:14.284518 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. May 23 12:40:14.284518 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application May 23 12:40:14.284518 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 25/49] 10.222.0.30 () {56 vars in 1096 bytes} [Mon May 23 12:40:14 2022] GET /identity/v3/roles/admin => generated 83 bytes in 158 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) May 23 12:40:14.289973 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e43d44d1-0330-4477-9f3d-e3b5ec40bca0 None None] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:40:14.392777 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e43d44d1-0330-4477-9f3d-e3b5ec40bca0 None None] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:40:14.426091 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65dd48b0-e24c-4a48-8600-2d5166e79a62 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:40:14.426757 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65dd48b0-e24c-4a48-8600-2d5166e79a62 None admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:40:14.427049 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65dd48b0-e24c-4a48-8600-2d5166e79a62 None admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:40:14.427328 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65dd48b0-e24c-4a48-8600-2d5166e79a62 None admin] PATH_INFO: `/v3/roles` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:40:14.428079 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65dd48b0-e24c-4a48-8600-2d5166e79a62 None admin] RBAC: Adding query filter params (name=admin) {{(pid=92057) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} May 23 12:40:14.428394 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65dd48b0-e24c-4a48-8600-2d5166e79a62 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:40:14.429800 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65dd48b0-e24c-4a48-8600-2d5166e79a62 None admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:40:14.437896 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 25/50] 10.222.0.30 () {56 vars in 1099 bytes} [Mon May 23 12:40:14 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 148 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) May 23 12:40:14.443555 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06cb871f-a863-47ac-a209-db28638de442 None admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:40:14.484599 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1453163e-f26d-485c-9d25-5bd4abc78bc5 None None] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:40:14.485004 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1453163e-f26d-485c-9d25-5bd4abc78bc5 None None] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:40:14.485287 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1453163e-f26d-485c-9d25-5bd4abc78bc5 None None] PATH_INFO: `/` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:40:14.486738 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 26/51] 10.222.0.30 () {54 vars in 926 bytes} [Mon May 23 12:40:14 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) May 23 12:40:14.491577 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b8f3974-111a-491a-8172-a62be694e66d None None] REQUEST_METHOD: `POST` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:40:14.491863 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b8f3974-111a-491a-8172-a62be694e66d None None] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:40:14.492138 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b8f3974-111a-491a-8172-a62be694e66d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:40:14.509021 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.auth.core [None req-9b8f3974-111a-491a-8172-a62be694e66d None None] MFA Rules not processed for user `0f7119f8fa5c4445977671140ded91a4`. Rule list: `[]` (Enabled: `True`). {{(pid=92057) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 12:40:14.510255 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.fernet_utils [None req-9b8f3974-111a-491a-8172-a62be694e66d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92057) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 12:40:14.511935 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=100 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:40:14.512190 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=100 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:40:14.547137 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06cb871f-a863-47ac-a209-db28638de442 None admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:40:14.579472 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 27/52] 10.222.0.30 () {58 vars in 1023 bytes} [Mon May 23 12:40:14 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 88 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) May 23 12:40:14.581355 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eed005f6-32e9-4cd4-89ef-7506e49dfbd4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:40:14.582258 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eed005f6-32e9-4cd4-89ef-7506e49dfbd4 None admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:40:14.582549 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eed005f6-32e9-4cd4-89ef-7506e49dfbd4 None admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:40:14.582825 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eed005f6-32e9-4cd4-89ef-7506e49dfbd4 None admin] PATH_INFO: `/v3/users/0f7119f8fa5c4445977671140ded91a4` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:40:14.584257 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eed005f6-32e9-4cd4-89ef-7506e49dfbd4 None admin] RBAC: Authorizing `identity:get_user(user_id=0f7119f8fa5c4445977671140ded91a4)` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:40:14.585720 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eed005f6-32e9-4cd4-89ef-7506e49dfbd4 None admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:40:14.587374 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 26/53] 10.222.0.30 () {56 vars in 1177 bytes} [Mon May 23 12:40:14 2022] GET /identity/v3/users/0f7119f8fa5c4445977671140ded91a4 => generated 247 bytes in 144 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) May 23 12:40:14.592786 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b8f3974-111a-491a-8172-a62be694e66d None None] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:40:14.602571 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eed005f6-32e9-4cd4-89ef-7506e49dfbd4 None admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:40:14.695108 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b8f3974-111a-491a-8172-a62be694e66d None None] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:40:14.705223 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eed005f6-32e9-4cd4-89ef-7506e49dfbd4 None admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:40:14.733010 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2fff6a2-8a57-4179-b8f6-2b0ff9646501 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:40:14.733803 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2fff6a2-8a57-4179-b8f6-2b0ff9646501 None admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:40:14.734116 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2fff6a2-8a57-4179-b8f6-2b0ff9646501 None admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:40:14.734394 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2fff6a2-8a57-4179-b8f6-2b0ff9646501 None admin] PATH_INFO: `/v3/domains/default` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:40:14.735527 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2fff6a2-8a57-4179-b8f6-2b0ff9646501 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:40:14.737002 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2fff6a2-8a57-4179-b8f6-2b0ff9646501 None admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:40:14.738057 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1434a950-db57-4f85-a5cb-2fdc2ea9ff83 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:40:14.738656 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 28/54] 10.222.0.30 () {56 vars in 1108 bytes} [Mon May 23 12:40:14 2022] GET /identity/v3/domains/default => generated 199 bytes in 146 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) May 23 12:40:14.738848 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1434a950-db57-4f85-a5cb-2fdc2ea9ff83 None admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:40:14.739135 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1434a950-db57-4f85-a5cb-2fdc2ea9ff83 None admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:40:14.739415 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1434a950-db57-4f85-a5cb-2fdc2ea9ff83 None admin] PATH_INFO: `/v3/domains/default` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:40:14.740603 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1434a950-db57-4f85-a5cb-2fdc2ea9ff83 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:40:14.742085 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1434a950-db57-4f85-a5cb-2fdc2ea9ff83 None admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:40:14.748554 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 27/55] 10.222.0.30 () {56 vars in 1108 bytes} [Mon May 23 12:40:14 2022] GET /identity/v3/domains/default => generated 199 bytes in 142 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) May 23 12:40:14.748676 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2fff6a2-8a57-4179-b8f6-2b0ff9646501 None admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:40:14.750369 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1434a950-db57-4f85-a5cb-2fdc2ea9ff83 None admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:40:14.753054 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=100 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:40:14.754893 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=100 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:40:14.849203 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2fff6a2-8a57-4179-b8f6-2b0ff9646501 None admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:40:14.850881 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1434a950-db57-4f85-a5cb-2fdc2ea9ff83 None admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:40:14.882947 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1b4998c-8ff3-4fc0-8745-bed8ec65eb76 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:40:14.883599 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1b4998c-8ff3-4fc0-8745-bed8ec65eb76 None admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:40:14.883888 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1b4998c-8ff3-4fc0-8745-bed8ec65eb76 None admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:40:14.884168 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1b4998c-8ff3-4fc0-8745-bed8ec65eb76 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:40:14.885001 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1b4998c-8ff3-4fc0-8745-bed8ec65eb76 None admin] RBAC: Adding query filter params (role.id=bc3c3c9d654940cbaa5177a6be23d1bc, scope.domain.id=default, user.id=0f7119f8fa5c4445977671140ded91a4) {{(pid=92057) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} May 23 12:40:14.885087 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aecab48c-2e25-43cc-84fc-0bb0769bc270 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:40:14.885316 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1b4998c-8ff3-4fc0-8745-bed8ec65eb76 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:40:14.885663 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aecab48c-2e25-43cc-84fc-0bb0769bc270 None admin] REQUEST_METHOD: `POST` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:40:14.885990 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aecab48c-2e25-43cc-84fc-0bb0769bc270 None admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:40:14.886273 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aecab48c-2e25-43cc-84fc-0bb0769bc270 None admin] PATH_INFO: `/v3/projects` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:40:14.886800 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1b4998c-8ff3-4fc0-8745-bed8ec65eb76 None admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:40:14.887029 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aecab48c-2e25-43cc-84fc-0bb0769bc270 None admin] RBAC: Authorizing `identity:create_project()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:40:14.889105 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aecab48c-2e25-43cc-84fc-0bb0769bc270 None admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:40:14.893567 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 29/56] 10.222.0.30 () {56 vars in 1322 bytes} [Mon May 23 12:40:14 2022] GET /identity/v3/role_assignments?user.id=0f7119f8fa5c4445977671140ded91a4&scope.domain.id=default&role.id=bc3c3c9d654940cbaa5177a6be23d1bc => generated 530 bytes in 150 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) May 23 12:40:15.004548 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 28/57] 10.222.0.30 () {60 vars in 1140 bytes} [Mon May 23 12:40:14 2022] POST /identity/v3/projects => generated 298 bytes in 255 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) May 23 12:40:17.184055 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abcb5e8b-c6df-4df9-94a2-60096984aa34 None None] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:40:17.184597 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abcb5e8b-c6df-4df9-94a2-60096984aa34 None None] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:40:17.184874 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abcb5e8b-c6df-4df9-94a2-60096984aa34 None None] PATH_INFO: `/` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:40:17.186313 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 30/58] 10.222.0.30 () {54 vars in 926 bytes} [Mon May 23 12:40:17 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) May 23 12:40:17.191453 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8db4dc8-4537-465d-9efe-2cf22f098764 None None] REQUEST_METHOD: `POST` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:40:17.191812 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8db4dc8-4537-465d-9efe-2cf22f098764 None None] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:40:17.192018 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8db4dc8-4537-465d-9efe-2cf22f098764 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:40:17.208949 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.auth.core [None req-a8db4dc8-4537-465d-9efe-2cf22f098764 None None] MFA Rules not processed for user `0f7119f8fa5c4445977671140ded91a4`. Rule list: `[]` (Enabled: `True`). {{(pid=92056) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 12:40:17.210203 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.fernet_utils [None req-a8db4dc8-4537-465d-9efe-2cf22f098764 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92056) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 12:40:17.257546 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:40:17.274520 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 29/59] 10.222.0.30 () {58 vars in 1023 bytes} [Mon May 23 12:40:17 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 84 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) May 23 12:40:17.299354 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-abcb5e8b-c6df-4df9-94a2-60096984aa34 None None] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:40:17.367938 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=100 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:40:17.398476 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-abcb5e8b-c6df-4df9-94a2-60096984aa34 None None] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:40:17.431573 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a78d3724-86e9-4de7-bb73-321fb1d19a0c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:40:17.432360 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a78d3724-86e9-4de7-bb73-321fb1d19a0c None admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:40:17.432650 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a78d3724-86e9-4de7-bb73-321fb1d19a0c None admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:40:17.432933 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a78d3724-86e9-4de7-bb73-321fb1d19a0c None admin] PATH_INFO: `/v3/domains/default` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:40:17.434228 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a78d3724-86e9-4de7-bb73-321fb1d19a0c None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:40:17.435716 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a78d3724-86e9-4de7-bb73-321fb1d19a0c None admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:40:17.437362 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 31/60] 10.222.0.30 () {56 vars in 1108 bytes} [Mon May 23 12:40:17 2022] GET /identity/v3/domains/default => generated 199 bytes in 138 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) May 23 12:40:17.442864 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8db4dc8-4537-465d-9efe-2cf22f098764 None None] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:40:17.545833 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8db4dc8-4537-465d-9efe-2cf22f098764 None None] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:40:17.579347 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7be5fb57-2a0c-4b3e-a1d5-eb68c12747ce None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:40:17.580007 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7be5fb57-2a0c-4b3e-a1d5-eb68c12747ce None admin] REQUEST_METHOD: `POST` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:40:17.580288 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7be5fb57-2a0c-4b3e-a1d5-eb68c12747ce None admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:40:17.580565 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7be5fb57-2a0c-4b3e-a1d5-eb68c12747ce None admin] PATH_INFO: `/v3/users` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:40:17.581277 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7be5fb57-2a0c-4b3e-a1d5-eb68c12747ce None admin] RBAC: Authorizing `identity:create_user()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:40:17.582751 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7be5fb57-2a0c-4b3e-a1d5-eb68c12747ce None admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:40:17.654887 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 30/61] 10.222.0.30 () {60 vars in 1132 bytes} [Mon May 23 12:40:17 2022] POST /identity/v3/users => generated 275 bytes in 212 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) May 23 12:40:20.964194 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6304ad5b-deed-4f02-869e-9d09ff727e3f None None] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:40:20.964671 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6304ad5b-deed-4f02-869e-9d09ff727e3f None None] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:40:20.964671 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6304ad5b-deed-4f02-869e-9d09ff727e3f None None] PATH_INFO: `/` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:40:20.966143 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 32/62] 10.222.0.30 () {54 vars in 926 bytes} [Mon May 23 12:40:20 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) May 23 12:40:20.971578 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e4daab3-baf8-4028-904d-7073964a6de8 None None] REQUEST_METHOD: `POST` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:40:20.972431 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e4daab3-baf8-4028-904d-7073964a6de8 None None] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:40:20.974134 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b7a4516-8454-4f46-b8aa-68dd8016f9a0 None None] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:40:20.974624 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b7a4516-8454-4f46-b8aa-68dd8016f9a0 None None] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:40:20.975096 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b7a4516-8454-4f46-b8aa-68dd8016f9a0 None None] PATH_INFO: `/` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:40:20.976602 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 33/63] 10.222.0.30 () {54 vars in 926 bytes} [Mon May 23 12:40:20 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) May 23 12:40:20.976989 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e4daab3-baf8-4028-904d-7073964a6de8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:40:20.982680 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d9a4a8b-0b67-41a6-83de-98ea498eadfe None None] REQUEST_METHOD: `POST` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:40:20.983289 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d9a4a8b-0b67-41a6-83de-98ea498eadfe None None] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:40:20.983837 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d9a4a8b-0b67-41a6-83de-98ea498eadfe None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:40:21.003927 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.auth.core [None req-1e4daab3-baf8-4028-904d-7073964a6de8 None None] MFA Rules not processed for user `0f7119f8fa5c4445977671140ded91a4`. Rule list: `[]` (Enabled: `True`). {{(pid=92056) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 12:40:21.005120 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.fernet_utils [None req-1e4daab3-baf8-4028-904d-7073964a6de8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92056) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 12:40:21.011513 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.auth.core [None req-1d9a4a8b-0b67-41a6-83de-98ea498eadfe None None] MFA Rules not processed for user `0f7119f8fa5c4445977671140ded91a4`. Rule list: `[]` (Enabled: `True`). {{(pid=92057) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 12:40:21.013230 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.fernet_utils [None req-1d9a4a8b-0b67-41a6-83de-98ea498eadfe None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92057) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 12:40:21.073056 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:40:21.122097 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 31/64] 10.222.0.30 () {58 vars in 1023 bytes} [Mon May 23 12:40:20 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 151 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) May 23 12:40:21.144829 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e4daab3-baf8-4028-904d-7073964a6de8 None None] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:40:21.161226 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 34/65] 10.222.0.30 () {58 vars in 1023 bytes} [Mon May 23 12:40:20 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 180 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) May 23 12:40:21.187413 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d9a4a8b-0b67-41a6-83de-98ea498eadfe None None] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:40:21.295029 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=100 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:40:21.315391 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e4daab3-baf8-4028-904d-7073964a6de8 None None] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:40:21.341548 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d9a4a8b-0b67-41a6-83de-98ea498eadfe None None] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:40:21.361192 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4552da41-c586-403c-8aa2-a2a4c2ddc9ef None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:40:21.362348 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4552da41-c586-403c-8aa2-a2a4c2ddc9ef None admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:40:21.362844 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4552da41-c586-403c-8aa2-a2a4c2ddc9ef None admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:40:21.363423 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4552da41-c586-403c-8aa2-a2a4c2ddc9ef None admin] PATH_INFO: `/v3/roles/admin` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:40:21.375487 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4552da41-c586-403c-8aa2-a2a4c2ddc9ef None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} May 23 12:40:21.382865 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b5ccf2c-ff74-4699-9db2-a63bd506507e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:40:21.383668 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b5ccf2c-ff74-4699-9db2-a63bd506507e None admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:40:21.383950 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b5ccf2c-ff74-4699-9db2-a63bd506507e None admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:40:21.384227 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b5ccf2c-ff74-4699-9db2-a63bd506507e None admin] PATH_INFO: `/v3/roles/member` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:40:21.384969 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application [None req-4552da41-c586-403c-8aa2-a2a4c2ddc9ef None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. May 23 12:40:21.384969 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application Traceback (most recent call last): May 23 12:40:21.384969 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request May 23 12:40:21.384969 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application rv = self.dispatch_request() May 23 12:40:21.384969 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request May 23 12:40:21.384969 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) May 23 12:40:21.384969 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper May 23 12:40:21.384969 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) May 23 12:40:21.384969 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view May 23 12:40:21.384969 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) May 23 12:40:21.384969 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request May 23 12:40:21.384969 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) May 23 12:40:21.384969 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get May 23 12:40:21.384969 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self._get_role(role_id) May 23 12:40:21.384969 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role May 23 12:40:21.384969 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') May 23 12:40:21.385805 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call May 23 12:40:21.385805 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( May 23 12:40:21.385805 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data May 23 12:40:21.385805 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) May 23 12:40:21.385805 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped May 23 12:40:21.385805 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) May 23 12:40:21.385805 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun May 23 12:40:21.385805 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) May 23 12:40:21.385805 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func May 23 12:40:21.385805 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self.get_or_create( May 23 12:40:21.385805 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create May 23 12:40:21.385805 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application with Lock( May 23 12:40:21.385805 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ May 23 12:40:21.385805 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self._enter() May 23 12:40:21.385805 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter May 23 12:40:21.385805 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) May 23 12:40:21.386624 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create May 23 12:40:21.386624 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self.creator() May 23 12:40:21.386624 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value May 23 12:40:21.386624 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application created_value = creator( May 23 12:40:21.386624 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role May 23 12:40:21.386624 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) May 23 12:40:21.386624 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role May 23 12:40:21.386624 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() May 23 12:40:21.386624 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role May 23 12:40:21.386624 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) May 23 12:40:21.386624 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. May 23 12:40:21.386624 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application May 23 12:40:21.389477 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 32/66] 10.222.0.30 () {56 vars in 1096 bytes} [Mon May 23 12:40:21 2022] GET /identity/v3/roles/admin => generated 83 bytes in 245 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) May 23 12:40:21.391104 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e7a6225-c541-4c13-9634-ecc8255c50b0 None None] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:40:21.391610 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e7a6225-c541-4c13-9634-ecc8255c50b0 None None] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:40:21.392074 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e7a6225-c541-4c13-9634-ecc8255c50b0 None None] PATH_INFO: `/` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:40:21.397984 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 33/67] 10.222.0.30 () {54 vars in 926 bytes} [Mon May 23 12:40:21 2022] GET /identity => generated 270 bytes in 8 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) May 23 12:40:21.398996 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e7a6225-c541-4c13-9634-ecc8255c50b0 None None] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:40:21.400311 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b5ccf2c-ff74-4699-9db2-a63bd506507e None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} May 23 12:40:21.410737 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application [None req-7b5ccf2c-ff74-4699-9db2-a63bd506507e None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. May 23 12:40:21.410737 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application Traceback (most recent call last): May 23 12:40:21.410737 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request May 23 12:40:21.410737 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application rv = self.dispatch_request() May 23 12:40:21.410737 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request May 23 12:40:21.410737 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) May 23 12:40:21.410737 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper May 23 12:40:21.410737 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) May 23 12:40:21.410737 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view May 23 12:40:21.410737 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) May 23 12:40:21.410737 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request May 23 12:40:21.410737 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) May 23 12:40:21.410737 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get May 23 12:40:21.410737 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self._get_role(role_id) May 23 12:40:21.410737 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role May 23 12:40:21.410737 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') May 23 12:40:21.411690 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call May 23 12:40:21.411690 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( May 23 12:40:21.411690 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data May 23 12:40:21.411690 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) May 23 12:40:21.411690 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped May 23 12:40:21.411690 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) May 23 12:40:21.411690 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun May 23 12:40:21.411690 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) May 23 12:40:21.411690 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func May 23 12:40:21.411690 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self.get_or_create( May 23 12:40:21.411690 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create May 23 12:40:21.411690 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application with Lock( May 23 12:40:21.411690 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ May 23 12:40:21.411690 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self._enter() May 23 12:40:21.411690 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter May 23 12:40:21.411690 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) May 23 12:40:21.412463 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create May 23 12:40:21.412463 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self.creator() May 23 12:40:21.412463 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value May 23 12:40:21.412463 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application created_value = creator( May 23 12:40:21.412463 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role May 23 12:40:21.412463 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) May 23 12:40:21.412463 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role May 23 12:40:21.412463 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() May 23 12:40:21.412463 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role May 23 12:40:21.412463 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) May 23 12:40:21.412463 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. May 23 12:40:21.412463 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application May 23 12:40:21.412463 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 35/68] 10.222.0.30 () {56 vars in 1099 bytes} [Mon May 23 12:40:21 2022] GET /identity/v3/roles/member => generated 84 bytes in 225 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) May 23 12:40:21.413159 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8b4a965-18ed-47d8-a523-f91df05b6d5d None None] REQUEST_METHOD: `POST` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:40:21.413603 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8b4a965-18ed-47d8-a523-f91df05b6d5d None None] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:40:21.414132 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8b4a965-18ed-47d8-a523-f91df05b6d5d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:40:21.437763 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.auth.core [None req-c8b4a965-18ed-47d8-a523-f91df05b6d5d None None] MFA Rules not processed for user `0f7119f8fa5c4445977671140ded91a4`. Rule list: `[]` (Enabled: `True`). {{(pid=92057) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 12:40:21.439292 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.fernet_utils [None req-c8b4a965-18ed-47d8-a523-f91df05b6d5d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92057) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 12:40:21.445995 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=100 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:40:21.525368 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 36/69] 10.222.0.30 () {58 vars in 1023 bytes} [Mon May 23 12:40:21 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 113 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) May 23 12:40:21.525988 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8b4a965-18ed-47d8-a523-f91df05b6d5d None None] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:40:21.534387 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e7a6225-c541-4c13-9634-ecc8255c50b0 None None] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:40:21.576563 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e341f79d-cc60-42f2-b536-a996185a1a63 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:40:21.577150 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e341f79d-cc60-42f2-b536-a996185a1a63 None admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:40:21.577439 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e341f79d-cc60-42f2-b536-a996185a1a63 None admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:40:21.577719 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e341f79d-cc60-42f2-b536-a996185a1a63 None admin] PATH_INFO: `/v3/roles` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:40:21.578516 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e341f79d-cc60-42f2-b536-a996185a1a63 None admin] RBAC: Adding query filter params (name=admin) {{(pid=92056) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} May 23 12:40:21.578825 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e341f79d-cc60-42f2-b536-a996185a1a63 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:40:21.580227 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e341f79d-cc60-42f2-b536-a996185a1a63 None admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:40:21.588832 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 34/70] 10.222.0.30 () {56 vars in 1099 bytes} [Mon May 23 12:40:21 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 190 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) May 23 12:40:21.594229 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-743922d8-0a00-46f2-b566-0e272ccbb2d4 None None] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:40:21.594229 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-743922d8-0a00-46f2-b566-0e272ccbb2d4 None None] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:40:21.594229 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-743922d8-0a00-46f2-b566-0e272ccbb2d4 None None] PATH_INFO: `/` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:40:21.594229 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 35/71] 10.222.0.30 () {54 vars in 926 bytes} [Mon May 23 12:40:21 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) May 23 12:40:21.594229 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-743922d8-0a00-46f2-b566-0e272ccbb2d4 None None] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:40:21.598083 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=100 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:40:21.644854 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8b4a965-18ed-47d8-a523-f91df05b6d5d None None] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:40:21.691499 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d54c380a-3d3c-47d8-95ee-8a23890f6966 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:40:21.692177 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d54c380a-3d3c-47d8-95ee-8a23890f6966 None admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:40:21.692673 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d54c380a-3d3c-47d8-95ee-8a23890f6966 None admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:40:21.693155 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d54c380a-3d3c-47d8-95ee-8a23890f6966 None admin] PATH_INFO: `/v3/roles` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:40:21.694524 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d54c380a-3d3c-47d8-95ee-8a23890f6966 None admin] RBAC: Adding query filter params (name=member) {{(pid=92057) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} May 23 12:40:21.694524 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d54c380a-3d3c-47d8-95ee-8a23890f6966 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:40:21.695831 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d54c380a-3d3c-47d8-95ee-8a23890f6966 None admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:40:21.703532 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=100 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:40:21.709713 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 37/72] 10.222.0.30 () {56 vars in 1101 bytes} [Mon May 23 12:40:21 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 180 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) May 23 12:40:21.709713 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d54c380a-3d3c-47d8-95ee-8a23890f6966 None admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:40:21.717658 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-743922d8-0a00-46f2-b566-0e272ccbb2d4 None None] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:40:21.755622 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a2f485f-61c2-489b-b6d7-2573c546c8cc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:40:21.756371 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a2f485f-61c2-489b-b6d7-2573c546c8cc None admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:40:21.756660 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a2f485f-61c2-489b-b6d7-2573c546c8cc None admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:40:21.756941 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a2f485f-61c2-489b-b6d7-2573c546c8cc None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:40:21.769392 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a2f485f-61c2-489b-b6d7-2573c546c8cc None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} May 23 12:40:21.778891 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application [None req-2a2f485f-61c2-489b-b6d7-2573c546c8cc None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. May 23 12:40:21.778891 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application Traceback (most recent call last): May 23 12:40:21.778891 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request May 23 12:40:21.778891 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application rv = self.dispatch_request() May 23 12:40:21.778891 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request May 23 12:40:21.778891 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) May 23 12:40:21.778891 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper May 23 12:40:21.778891 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) May 23 12:40:21.778891 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view May 23 12:40:21.778891 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) May 23 12:40:21.778891 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request May 23 12:40:21.778891 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) May 23 12:40:21.778891 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get May 23 12:40:21.778891 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self._get_role(role_id) May 23 12:40:21.778891 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role May 23 12:40:21.778891 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') May 23 12:40:21.779946 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call May 23 12:40:21.779946 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( May 23 12:40:21.779946 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data May 23 12:40:21.779946 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) May 23 12:40:21.779946 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped May 23 12:40:21.779946 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) May 23 12:40:21.779946 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun May 23 12:40:21.779946 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) May 23 12:40:21.779946 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func May 23 12:40:21.779946 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self.get_or_create( May 23 12:40:21.779946 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create May 23 12:40:21.779946 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application with Lock( May 23 12:40:21.779946 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ May 23 12:40:21.779946 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self._enter() May 23 12:40:21.779946 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter May 23 12:40:21.779946 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) May 23 12:40:21.781027 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create May 23 12:40:21.781027 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self.creator() May 23 12:40:21.781027 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value May 23 12:40:21.781027 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application created_value = creator( May 23 12:40:21.781027 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role May 23 12:40:21.781027 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) May 23 12:40:21.781027 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role May 23 12:40:21.781027 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() May 23 12:40:21.781027 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role May 23 12:40:21.781027 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) May 23 12:40:21.781027 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. May 23 12:40:21.781027 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application May 23 12:40:21.781027 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 36/73] 10.222.0.30 () {56 vars in 1114 bytes} [Mon May 23 12:40:21 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 187 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) May 23 12:40:21.781691 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f86eb6b0-b982-4d22-b1ae-8a8d743bdf7f None None] REQUEST_METHOD: `POST` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:40:21.781844 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f86eb6b0-b982-4d22-b1ae-8a8d743bdf7f None None] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:40:21.782143 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f86eb6b0-b982-4d22-b1ae-8a8d743bdf7f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:40:21.799575 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.auth.core [None req-f86eb6b0-b982-4d22-b1ae-8a8d743bdf7f None None] MFA Rules not processed for user `0f7119f8fa5c4445977671140ded91a4`. Rule list: `[]` (Enabled: `True`). {{(pid=92056) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 12:40:21.801227 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.fernet_utils [None req-f86eb6b0-b982-4d22-b1ae-8a8d743bdf7f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92056) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 12:40:21.820478 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d54c380a-3d3c-47d8-95ee-8a23890f6966 None admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:40:21.841694 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=100 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:40:21.856593 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e270d16e-2863-43d1-a7e0-36b4d65e1d33 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:40:21.857470 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e270d16e-2863-43d1-a7e0-36b4d65e1d33 None admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:40:21.857768 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e270d16e-2863-43d1-a7e0-36b4d65e1d33 None admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:40:21.858076 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e270d16e-2863-43d1-a7e0-36b4d65e1d33 None admin] PATH_INFO: `/v3/users/0f7119f8fa5c4445977671140ded91a4` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:40:21.859291 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e270d16e-2863-43d1-a7e0-36b4d65e1d33 None admin] RBAC: Authorizing `identity:get_user(user_id=0f7119f8fa5c4445977671140ded91a4)` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:40:21.860745 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e270d16e-2863-43d1-a7e0-36b4d65e1d33 None admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:40:21.862409 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 38/74] 10.222.0.30 () {56 vars in 1177 bytes} [Mon May 23 12:40:21 2022] GET /identity/v3/users/0f7119f8fa5c4445977671140ded91a4 => generated 247 bytes in 156 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) May 23 12:40:21.863170 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e270d16e-2863-43d1-a7e0-36b4d65e1d33 None admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:40:21.877381 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 37/75] 10.222.0.30 () {58 vars in 1023 bytes} [Mon May 23 12:40:21 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 96 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) May 23 12:40:21.878271 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f86eb6b0-b982-4d22-b1ae-8a8d743bdf7f None None] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:40:21.966184 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=100 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:40:21.967628 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e270d16e-2863-43d1-a7e0-36b4d65e1d33 None admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:40:21.983853 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f86eb6b0-b982-4d22-b1ae-8a8d743bdf7f None None] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:40:22.001115 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-204ff937-2e84-4ff4-a9da-833286d819fe None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:40:22.002023 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-204ff937-2e84-4ff4-a9da-833286d819fe None admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:40:22.002314 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-204ff937-2e84-4ff4-a9da-833286d819fe None admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:40:22.002596 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-204ff937-2e84-4ff4-a9da-833286d819fe None admin] PATH_INFO: `/v3/users/b3478ea957234bb99fde27e6c3d0dfd3` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:40:22.016811 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-204ff937-2e84-4ff4-a9da-833286d819fe None admin] RBAC: Authorizing `identity:get_user(user_id=b3478ea957234bb99fde27e6c3d0dfd3)` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:40:22.018313 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-204ff937-2e84-4ff4-a9da-833286d819fe None admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:40:22.019020 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66b1e4a0-408b-443a-a042-11d39861d66c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:40:22.019684 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66b1e4a0-408b-443a-a042-11d39861d66c None admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:40:22.019968 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66b1e4a0-408b-443a-a042-11d39861d66c None admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:40:22.020049 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 39/76] 10.222.0.30 () {56 vars in 1177 bytes} [Mon May 23 12:40:21 2022] GET /identity/v3/users/b3478ea957234bb99fde27e6c3d0dfd3 => generated 275 bytes in 157 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) May 23 12:40:22.020244 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66b1e4a0-408b-443a-a042-11d39861d66c None admin] PATH_INFO: `/v3/roles` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:40:22.020987 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66b1e4a0-408b-443a-a042-11d39861d66c None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=92056) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} May 23 12:40:22.021208 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-887713b2-6560-4039-b9c6-06d9ea3ecfeb None None] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:40:22.021296 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66b1e4a0-408b-443a-a042-11d39861d66c None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:40:22.021497 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-887713b2-6560-4039-b9c6-06d9ea3ecfeb None None] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:40:22.021779 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-887713b2-6560-4039-b9c6-06d9ea3ecfeb None None] PATH_INFO: `/` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:40:22.022797 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66b1e4a0-408b-443a-a042-11d39861d66c None admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:40:22.023279 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 40/77] 10.222.0.30 () {54 vars in 926 bytes} [Mon May 23 12:40:22 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) May 23 12:40:22.023825 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-887713b2-6560-4039-b9c6-06d9ea3ecfeb None None] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:40:22.031427 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 38/78] 10.222.0.30 () {56 vars in 1111 bytes} [Mon May 23 12:40:21 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 153 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) May 23 12:40:22.032234 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66b1e4a0-408b-443a-a042-11d39861d66c None admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:40:22.075321 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=100 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:40:22.129604 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-887713b2-6560-4039-b9c6-06d9ea3ecfeb None None] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:40:22.131792 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66b1e4a0-408b-443a-a042-11d39861d66c None admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:40:22.162937 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e161c24-c682-4ae0-90f3-738c26c48da2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:40:22.163578 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e161c24-c682-4ae0-90f3-738c26c48da2 None admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:40:22.163863 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e161c24-c682-4ae0-90f3-738c26c48da2 None admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:40:22.164139 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e161c24-c682-4ae0-90f3-738c26c48da2 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:40:22.165058 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb4dadb3-7e16-4edc-a89a-e3d5adb00f11 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:40:22.165849 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb4dadb3-7e16-4edc-a89a-e3d5adb00f11 None admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:40:22.166167 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb4dadb3-7e16-4edc-a89a-e3d5adb00f11 None admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:40:22.166447 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb4dadb3-7e16-4edc-a89a-e3d5adb00f11 None admin] PATH_INFO: `/v3/roles/member` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:40:22.176745 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb4dadb3-7e16-4edc-a89a-e3d5adb00f11 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} May 23 12:40:22.185021 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application [None req-fb4dadb3-7e16-4edc-a89a-e3d5adb00f11 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. May 23 12:40:22.185021 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application Traceback (most recent call last): May 23 12:40:22.185021 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request May 23 12:40:22.185021 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application rv = self.dispatch_request() May 23 12:40:22.185021 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request May 23 12:40:22.185021 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) May 23 12:40:22.185021 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper May 23 12:40:22.185021 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) May 23 12:40:22.185021 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view May 23 12:40:22.185021 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) May 23 12:40:22.185021 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request May 23 12:40:22.185021 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) May 23 12:40:22.185021 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get May 23 12:40:22.185021 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self._get_role(role_id) May 23 12:40:22.185021 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role May 23 12:40:22.185021 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') May 23 12:40:22.186171 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call May 23 12:40:22.186171 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( May 23 12:40:22.186171 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data May 23 12:40:22.186171 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) May 23 12:40:22.186171 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped May 23 12:40:22.186171 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) May 23 12:40:22.186171 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun May 23 12:40:22.186171 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) May 23 12:40:22.186171 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func May 23 12:40:22.186171 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self.get_or_create( May 23 12:40:22.186171 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create May 23 12:40:22.186171 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application with Lock( May 23 12:40:22.186171 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ May 23 12:40:22.186171 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self._enter() May 23 12:40:22.186171 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter May 23 12:40:22.186171 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) May 23 12:40:22.186957 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create May 23 12:40:22.186957 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self.creator() May 23 12:40:22.186957 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value May 23 12:40:22.186957 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application created_value = creator( May 23 12:40:22.186957 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role May 23 12:40:22.186957 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) May 23 12:40:22.186957 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role May 23 12:40:22.186957 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() May 23 12:40:22.186957 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role May 23 12:40:22.186957 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) May 23 12:40:22.186957 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. May 23 12:40:22.186957 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application May 23 12:40:22.186957 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 39/79] 10.222.0.30 () {56 vars in 1099 bytes} [Mon May 23 12:40:22 2022] GET /identity/v3/roles/member => generated 84 bytes in 154 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) May 23 12:40:22.187629 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7db5c62b-6737-4ba9-8205-5b3f27dbbb97 None None] REQUEST_METHOD: `POST` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:40:22.187695 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7db5c62b-6737-4ba9-8205-5b3f27dbbb97 None None] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:40:22.187964 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7db5c62b-6737-4ba9-8205-5b3f27dbbb97 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:40:22.194802 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=100 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:40:22.205911 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.auth.core [None req-7db5c62b-6737-4ba9-8205-5b3f27dbbb97 None None] MFA Rules not processed for user `0f7119f8fa5c4445977671140ded91a4`. Rule list: `[]` (Enabled: `True`). {{(pid=92056) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 12:40:22.210471 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.fernet_utils [None req-7db5c62b-6737-4ba9-8205-5b3f27dbbb97 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92056) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 12:40:22.239476 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e161c24-c682-4ae0-90f3-738c26c48da2 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:40:22.240967 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e161c24-c682-4ae0-90f3-738c26c48da2 None admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:40:22.273058 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 40/80] 10.222.0.30 () {58 vars in 1023 bytes} [Mon May 23 12:40:22 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) May 23 12:40:22.273929 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7db5c62b-6737-4ba9-8205-5b3f27dbbb97 None None] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:40:22.286962 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=100 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:40:22.341275 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 41/81] 10.222.0.30 () {58 vars in 1282 bytes} [Mon May 23 12:40:22 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 317 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) May 23 12:40:22.341966 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e161c24-c682-4ae0-90f3-738c26c48da2 None admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:40:22.382773 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7db5c62b-6737-4ba9-8205-5b3f27dbbb97 None None] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:40:22.416327 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0fd3e7bd-d1fb-4d5c-ac31-3470b82a7858 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:40:22.416974 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fd3e7bd-d1fb-4d5c-ac31-3470b82a7858 None admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:40:22.417262 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fd3e7bd-d1fb-4d5c-ac31-3470b82a7858 None admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:40:22.417550 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fd3e7bd-d1fb-4d5c-ac31-3470b82a7858 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:40:22.421047 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=100 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:40:22.440613 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e161c24-c682-4ae0-90f3-738c26c48da2 None admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:40:22.473437 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9121671-ada6-4db9-8038-fc5ff31021af None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:40:22.474331 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9121671-ada6-4db9-8038-fc5ff31021af None admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:40:22.474617 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9121671-ada6-4db9-8038-fc5ff31021af None admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:40:22.474900 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9121671-ada6-4db9-8038-fc5ff31021af None admin] PATH_INFO: `/v3/users/b3478ea957234bb99fde27e6c3d0dfd3` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:40:22.476093 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9121671-ada6-4db9-8038-fc5ff31021af None admin] RBAC: Authorizing `identity:get_user(user_id=b3478ea957234bb99fde27e6c3d0dfd3)` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:40:22.477565 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9121671-ada6-4db9-8038-fc5ff31021af None admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:40:22.479228 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 42/82] 10.222.0.30 () {56 vars in 1177 bytes} [Mon May 23 12:40:22 2022] GET /identity/v3/users/b3478ea957234bb99fde27e6c3d0dfd3 => generated 275 bytes in 138 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) May 23 12:40:22.479994 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9121671-ada6-4db9-8038-fc5ff31021af None admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:40:22.487560 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0fd3e7bd-d1fb-4d5c-ac31-3470b82a7858 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:40:22.489042 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0fd3e7bd-d1fb-4d5c-ac31-3470b82a7858 None admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:40:22.495122 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=100 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:40:22.580443 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9121671-ada6-4db9-8038-fc5ff31021af None admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:40:22.595051 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 41/83] 10.222.0.30 () {58 vars in 1282 bytes} [Mon May 23 12:40:22 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 321 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) May 23 12:40:22.595924 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0fd3e7bd-d1fb-4d5c-ac31-3470b82a7858 None admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:40:22.619501 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ccf33fbe-64d6-452f-b82e-7d05eaede45e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:40:22.620168 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccf33fbe-64d6-452f-b82e-7d05eaede45e None admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:40:22.620458 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccf33fbe-64d6-452f-b82e-7d05eaede45e None admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:40:22.620738 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccf33fbe-64d6-452f-b82e-7d05eaede45e None admin] PATH_INFO: `/v3/roles` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:40:22.622150 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ccf33fbe-64d6-452f-b82e-7d05eaede45e None admin] RBAC: Adding query filter params (name=member) {{(pid=92057) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} May 23 12:40:22.622500 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ccf33fbe-64d6-452f-b82e-7d05eaede45e None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:40:22.623926 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ccf33fbe-64d6-452f-b82e-7d05eaede45e None admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:40:22.628602 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=100 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:40:22.632130 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 43/84] 10.222.0.30 () {56 vars in 1101 bytes} [Mon May 23 12:40:22 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 152 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) May 23 12:40:22.632985 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ccf33fbe-64d6-452f-b82e-7d05eaede45e None admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:40:22.695282 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0fd3e7bd-d1fb-4d5c-ac31-3470b82a7858 None admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:40:22.714221 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=100 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:40:22.730314 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ddfbc7e1-a0e7-4246-9065-e3bb32898628 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:40:22.731295 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddfbc7e1-a0e7-4246-9065-e3bb32898628 None admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:40:22.731593 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddfbc7e1-a0e7-4246-9065-e3bb32898628 None admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:40:22.731870 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddfbc7e1-a0e7-4246-9065-e3bb32898628 None admin] PATH_INFO: `/v3/domains/default` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:40:22.733045 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ddfbc7e1-a0e7-4246-9065-e3bb32898628 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:40:22.734620 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ddfbc7e1-a0e7-4246-9065-e3bb32898628 None admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:40:22.736236 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 42/85] 10.222.0.30 () {56 vars in 1108 bytes} [Mon May 23 12:40:22 2022] GET /identity/v3/domains/default => generated 199 bytes in 141 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) May 23 12:40:22.736863 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ccf33fbe-64d6-452f-b82e-7d05eaede45e None admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:40:22.737044 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ddfbc7e1-a0e7-4246-9065-e3bb32898628 None admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:40:22.771957 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45def34b-5d5c-4a83-aa47-e9e62044449a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:40:22.772813 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45def34b-5d5c-4a83-aa47-e9e62044449a None admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:40:22.773101 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45def34b-5d5c-4a83-aa47-e9e62044449a None admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:40:22.773375 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45def34b-5d5c-4a83-aa47-e9e62044449a None admin] PATH_INFO: `/v3/projects/486401ae2e5f40d5a5f060242896ae53` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:40:22.774597 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45def34b-5d5c-4a83-aa47-e9e62044449a None admin] RBAC: Authorizing `identity:get_project(project_id=486401ae2e5f40d5a5f060242896ae53)` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:40:22.776048 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45def34b-5d5c-4a83-aa47-e9e62044449a None admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:40:22.777931 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 44/86] 10.222.0.30 () {56 vars in 1186 bytes} [Mon May 23 12:40:22 2022] GET /identity/v3/projects/486401ae2e5f40d5a5f060242896ae53 => generated 298 bytes in 145 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) May 23 12:40:22.778602 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45def34b-5d5c-4a83-aa47-e9e62044449a None admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:40:22.838453 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ddfbc7e1-a0e7-4246-9065-e3bb32898628 None admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:40:22.840050 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=100 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:40:22.874896 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7bff06e-5ecb-4b1e-ada1-20bd9cd4a64c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:40:22.875553 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7bff06e-5ecb-4b1e-ada1-20bd9cd4a64c None admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:40:22.875838 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7bff06e-5ecb-4b1e-ada1-20bd9cd4a64c None admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:40:22.876119 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7bff06e-5ecb-4b1e-ada1-20bd9cd4a64c None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:40:22.880601 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45def34b-5d5c-4a83-aa47-e9e62044449a None admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:40:22.913648 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51964fd1-3b82-4409-b605-48769d086a7c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:40:22.914469 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51964fd1-3b82-4409-b605-48769d086a7c None admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:40:22.914758 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51964fd1-3b82-4409-b605-48769d086a7c None admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:40:22.915034 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51964fd1-3b82-4409-b605-48769d086a7c None admin] PATH_INFO: `/v3/projects/486401ae2e5f40d5a5f060242896ae53` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:40:22.916214 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51964fd1-3b82-4409-b605-48769d086a7c None admin] RBAC: Authorizing `identity:get_project(project_id=486401ae2e5f40d5a5f060242896ae53)` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:40:22.917664 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51964fd1-3b82-4409-b605-48769d086a7c None admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:40:22.919421 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 45/87] 10.222.0.30 () {56 vars in 1186 bytes} [Mon May 23 12:40:22 2022] GET /identity/v3/projects/486401ae2e5f40d5a5f060242896ae53 => generated 298 bytes in 141 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) May 23 12:40:22.920184 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51964fd1-3b82-4409-b605-48769d086a7c None admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:40:22.937412 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=100 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:40:22.944689 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c7bff06e-5ecb-4b1e-ada1-20bd9cd4a64c None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:40:22.946193 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c7bff06e-5ecb-4b1e-ada1-20bd9cd4a64c None admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:40:23.021492 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51964fd1-3b82-4409-b605-48769d086a7c None admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:40:23.043416 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 43/88] 10.222.0.30 () {58 vars in 1282 bytes} [Mon May 23 12:40:22 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 307 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) May 23 12:40:23.044333 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7bff06e-5ecb-4b1e-ada1-20bd9cd4a64c None admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:40:23.046603 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=100 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:40:23.061324 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12f341e3-044f-471a-bbf1-5ae5e5c593f6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:40:23.062240 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12f341e3-044f-471a-bbf1-5ae5e5c593f6 None admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:40:23.062528 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12f341e3-044f-471a-bbf1-5ae5e5c593f6 None admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:40:23.062810 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12f341e3-044f-471a-bbf1-5ae5e5c593f6 None admin] PATH_INFO: `/v3/users/b3478ea957234bb99fde27e6c3d0dfd3` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:40:23.064011 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12f341e3-044f-471a-bbf1-5ae5e5c593f6 None admin] RBAC: Authorizing `identity:get_user(user_id=b3478ea957234bb99fde27e6c3d0dfd3)` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:40:23.065488 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12f341e3-044f-471a-bbf1-5ae5e5c593f6 None admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:40:23.067165 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 46/89] 10.222.0.30 () {56 vars in 1177 bytes} [Mon May 23 12:40:22 2022] GET /identity/v3/users/b3478ea957234bb99fde27e6c3d0dfd3 => generated 275 bytes in 147 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) May 23 12:40:23.068008 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12f341e3-044f-471a-bbf1-5ae5e5c593f6 None admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:40:23.143512 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7bff06e-5ecb-4b1e-ada1-20bd9cd4a64c None admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:40:23.149639 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=100 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:40:23.170328 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12f341e3-044f-471a-bbf1-5ae5e5c593f6 None admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:40:23.177275 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0618c412-e98f-4304-b7ca-a8cfa352c765 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:40:23.177973 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0618c412-e98f-4304-b7ca-a8cfa352c765 None admin] REQUEST_METHOD: `POST` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:40:23.178257 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0618c412-e98f-4304-b7ca-a8cfa352c765 None admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:40:23.178531 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0618c412-e98f-4304-b7ca-a8cfa352c765 None admin] PATH_INFO: `/v3/users` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:40:23.179240 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0618c412-e98f-4304-b7ca-a8cfa352c765 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:40:23.180737 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0618c412-e98f-4304-b7ca-a8cfa352c765 None admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:40:23.203117 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12fb0670-2c75-40e7-9458-f03591bc40be None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:40:23.203767 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12fb0670-2c75-40e7-9458-f03591bc40be None admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:40:23.204057 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12fb0670-2c75-40e7-9458-f03591bc40be None admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:40:23.204371 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12fb0670-2c75-40e7-9458-f03591bc40be None admin] PATH_INFO: `/v3/role_assignments` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:40:23.205195 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12fb0670-2c75-40e7-9458-f03591bc40be None admin] RBAC: Adding query filter params (role.id=bc3c3c9d654940cbaa5177a6be23d1bc, scope.project.id=486401ae2e5f40d5a5f060242896ae53, user.id=0f7119f8fa5c4445977671140ded91a4) {{(pid=92057) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} May 23 12:40:23.205508 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12fb0670-2c75-40e7-9458-f03591bc40be None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:40:23.206974 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12fb0670-2c75-40e7-9458-f03591bc40be None admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:40:23.212312 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 47/90] 10.222.0.30 () {56 vars in 1374 bytes} [Mon May 23 12:40:23 2022] GET /identity/v3/role_assignments?user.id=0f7119f8fa5c4445977671140ded91a4&scope.project.id=486401ae2e5f40d5a5f060242896ae53&role.id=bc3c3c9d654940cbaa5177a6be23d1bc => generated 259 bytes in 145 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) May 23 12:40:23.213085 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12fb0670-2c75-40e7-9458-f03591bc40be None admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:40:23.290705 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 44/91] 10.222.0.30 () {60 vars in 1132 bytes} [Mon May 23 12:40:23 2022] POST /identity/v3/users => generated 289 bytes in 247 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) May 23 12:40:23.291894 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0618c412-e98f-4304-b7ca-a8cfa352c765 None admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:40:23.345146 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12fb0670-2c75-40e7-9458-f03591bc40be None admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:40:23.387027 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:40:23.399354 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5231d424-1306-4780-9d4a-28ae201102b0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:40:23.401379 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5231d424-1306-4780-9d4a-28ae201102b0 None admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:40:23.401379 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5231d424-1306-4780-9d4a-28ae201102b0 None admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:40:23.401379 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5231d424-1306-4780-9d4a-28ae201102b0 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:40:23.402841 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5231d424-1306-4780-9d4a-28ae201102b0 None admin] RBAC: Adding query filter params (role.id=80fb22507c9a4ff09d44d8f12f6fff1a, scope.project.id=486401ae2e5f40d5a5f060242896ae53, user.id=b3478ea957234bb99fde27e6c3d0dfd3) {{(pid=92057) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} May 23 12:40:23.403318 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5231d424-1306-4780-9d4a-28ae201102b0 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:40:23.404833 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5231d424-1306-4780-9d4a-28ae201102b0 None admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:40:23.413740 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 48/92] 10.222.0.30 () {56 vars in 1374 bytes} [Mon May 23 12:40:23 2022] GET /identity/v3/role_assignments?user.id=b3478ea957234bb99fde27e6c3d0dfd3&scope.project.id=486401ae2e5f40d5a5f060242896ae53&role.id=80fb22507c9a4ff09d44d8f12f6fff1a => generated 259 bytes in 201 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) May 23 12:40:23.414531 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5231d424-1306-4780-9d4a-28ae201102b0 None admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:40:23.441745 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=100 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:40:23.455383 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0618c412-e98f-4304-b7ca-a8cfa352c765 None admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:40:23.536936 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5231d424-1306-4780-9d4a-28ae201102b0 None admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:40:23.539619 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5bbd1502-d138-4696-a307-2c98e53fb2f2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:40:23.540791 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5bbd1502-d138-4696-a307-2c98e53fb2f2 None admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:40:23.541275 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5bbd1502-d138-4696-a307-2c98e53fb2f2 None admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:40:23.541802 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5bbd1502-d138-4696-a307-2c98e53fb2f2 None admin] PATH_INFO: `/v3/projects/486401ae2e5f40d5a5f060242896ae53` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:40:23.544361 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5bbd1502-d138-4696-a307-2c98e53fb2f2 None admin] RBAC: Authorizing `identity:get_project(project_id=486401ae2e5f40d5a5f060242896ae53)` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:40:23.546179 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5bbd1502-d138-4696-a307-2c98e53fb2f2 None admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:40:23.548258 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 45/93] 10.222.0.30 () {56 vars in 1186 bytes} [Mon May 23 12:40:23 2022] GET /identity/v3/projects/486401ae2e5f40d5a5f060242896ae53 => generated 298 bytes in 257 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) May 23 12:40:23.554668 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5bbd1502-d138-4696-a307-2c98e53fb2f2 None admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:40:23.577698 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f6a0fce-abe7-4543-9b12-b8f573c2bfaa None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:40:23.578396 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f6a0fce-abe7-4543-9b12-b8f573c2bfaa None admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:40:23.578683 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f6a0fce-abe7-4543-9b12-b8f573c2bfaa None admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:40:23.578963 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f6a0fce-abe7-4543-9b12-b8f573c2bfaa None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:40:23.674794 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f6a0fce-abe7-4543-9b12-b8f573c2bfaa None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:40:23.676780 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f6a0fce-abe7-4543-9b12-b8f573c2bfaa None admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:40:23.695816 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5bbd1502-d138-4696-a307-2c98e53fb2f2 None admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:40:23.698722 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=100 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:40:23.738300 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=100 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:40:23.751523 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cadf69a1-e7ca-434b-a1c2-16ab380866da None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:40:23.752634 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cadf69a1-e7ca-434b-a1c2-16ab380866da None admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:40:23.753112 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cadf69a1-e7ca-434b-a1c2-16ab380866da None admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:40:23.753608 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cadf69a1-e7ca-434b-a1c2-16ab380866da None admin] PATH_INFO: `/v3/role_assignments` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:40:23.754769 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cadf69a1-e7ca-434b-a1c2-16ab380866da None admin] RBAC: Adding query filter params (role.id=e3b90948645643e888620ab1e5c774bf, scope.project.id=486401ae2e5f40d5a5f060242896ae53, user.id=b3478ea957234bb99fde27e6c3d0dfd3) {{(pid=92056) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} May 23 12:40:23.755425 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cadf69a1-e7ca-434b-a1c2-16ab380866da None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:40:23.757265 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cadf69a1-e7ca-434b-a1c2-16ab380866da None admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:40:23.769643 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 46/94] 10.222.0.30 () {56 vars in 1374 bytes} [Mon May 23 12:40:23 2022] GET /identity/v3/role_assignments?user.id=b3478ea957234bb99fde27e6c3d0dfd3&scope.project.id=486401ae2e5f40d5a5f060242896ae53&role.id=e3b90948645643e888620ab1e5c774bf => generated 259 bytes in 216 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) May 23 12:40:23.814008 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 49/95] 10.222.0.30 () {58 vars in 1282 bytes} [Mon May 23 12:40:23 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 400 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) May 23 12:40:23.819983 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cadf69a1-e7ca-434b-a1c2-16ab380866da None admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:40:23.938032 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cadf69a1-e7ca-434b-a1c2-16ab380866da None admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:40:23.982371 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-076cd1d4-9c8b-4587-a8e5-3a392b42b292 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:40:23.983213 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-076cd1d4-9c8b-4587-a8e5-3a392b42b292 None admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:40:23.983508 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-076cd1d4-9c8b-4587-a8e5-3a392b42b292 None admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:40:23.983785 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-076cd1d4-9c8b-4587-a8e5-3a392b42b292 None admin] PATH_INFO: `/v3/projects/40ed642151e34cbfa138265d51948402` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:40:23.986971 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-076cd1d4-9c8b-4587-a8e5-3a392b42b292 None admin] RBAC: Authorizing `identity:get_project(project_id=40ed642151e34cbfa138265d51948402)` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:40:23.988488 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-076cd1d4-9c8b-4587-a8e5-3a392b42b292 None admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:40:23.990659 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 47/96] 10.222.0.30 () {56 vars in 1186 bytes} [Mon May 23 12:40:23 2022] GET /identity/v3/projects/40ed642151e34cbfa138265d51948402 => generated 312 bytes in 171 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) May 23 12:40:23.997274 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f6a0fce-abe7-4543-9b12-b8f573c2bfaa None admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:40:24.131318 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f6a0fce-abe7-4543-9b12-b8f573c2bfaa None admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:40:24.146813 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=100 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:40:24.176247 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d1fab7e-0f07-4301-a68c-aca0c8d85977 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:40:24.176909 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d1fab7e-0f07-4301-a68c-aca0c8d85977 None admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:40:24.177194 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d1fab7e-0f07-4301-a68c-aca0c8d85977 None admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:40:24.177471 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d1fab7e-0f07-4301-a68c-aca0c8d85977 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:40:24.178355 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d1fab7e-0f07-4301-a68c-aca0c8d85977 None admin] RBAC: Adding query filter params (role.id=80fb22507c9a4ff09d44d8f12f6fff1a, scope.project.id=40ed642151e34cbfa138265d51948402, user.id=b3478ea957234bb99fde27e6c3d0dfd3) {{(pid=92057) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} May 23 12:40:24.178670 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d1fab7e-0f07-4301-a68c-aca0c8d85977 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:40:24.180159 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d1fab7e-0f07-4301-a68c-aca0c8d85977 None admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:40:24.191880 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 50/97] 10.222.0.30 () {56 vars in 1374 bytes} [Mon May 23 12:40:23 2022] GET /identity/v3/role_assignments?user.id=b3478ea957234bb99fde27e6c3d0dfd3&scope.project.id=40ed642151e34cbfa138265d51948402&role.id=80fb22507c9a4ff09d44d8f12f6fff1a => generated 259 bytes in 196 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) May 23 12:40:26.186278 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2753f87c-a066-4870-b921-31a97bcbe424 None None] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:40:26.186867 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2753f87c-a066-4870-b921-31a97bcbe424 None None] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:40:26.186867 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2753f87c-a066-4870-b921-31a97bcbe424 None None] PATH_INFO: `/` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:40:26.188488 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 48/98] 10.222.0.30 () {54 vars in 926 bytes} [Mon May 23 12:40:26 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) May 23 12:40:26.193936 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b8995ee-12f0-4046-9fdc-4377a98f1a74 None None] REQUEST_METHOD: `POST` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:40:26.194414 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b8995ee-12f0-4046-9fdc-4377a98f1a74 None None] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:40:26.194494 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b8995ee-12f0-4046-9fdc-4377a98f1a74 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:40:26.218779 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.auth.core [None req-6b8995ee-12f0-4046-9fdc-4377a98f1a74 None None] MFA Rules not processed for user `0f7119f8fa5c4445977671140ded91a4`. Rule list: `[]` (Enabled: `True`). {{(pid=92057) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 12:40:26.220030 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.fernet_utils [None req-6b8995ee-12f0-4046-9fdc-4377a98f1a74 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92057) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 12:40:26.316131 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 51/99] 10.222.0.30 () {58 vars in 1023 bytes} [Mon May 23 12:40:26 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 123 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) May 23 12:40:26.350812 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2753f87c-a066-4870-b921-31a97bcbe424 None None] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:40:26.444302 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=100 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:40:26.485261 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-955b032d-0326-4d16-80aa-a6051c3e44ee None None] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:40:26.485560 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-955b032d-0326-4d16-80aa-a6051c3e44ee None None] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:40:26.485842 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-955b032d-0326-4d16-80aa-a6051c3e44ee None None] PATH_INFO: `/` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:40:26.487526 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 52/100] 10.222.0.30 () {54 vars in 926 bytes} [Mon May 23 12:40:26 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) May 23 12:40:26.492664 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-898caf88-cb86-4663-8d11-1d067637380a None None] REQUEST_METHOD: `POST` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:40:26.492958 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-898caf88-cb86-4663-8d11-1d067637380a None None] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:40:26.493245 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-898caf88-cb86-4663-8d11-1d067637380a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:40:26.514974 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2753f87c-a066-4870-b921-31a97bcbe424 None None] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:40:26.525995 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.auth.core [None req-898caf88-cb86-4663-8d11-1d067637380a None None] MFA Rules not processed for user `0f7119f8fa5c4445977671140ded91a4`. Rule list: `[]` (Enabled: `True`). {{(pid=92057) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 12:40:26.527772 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.fernet_utils [None req-898caf88-cb86-4663-8d11-1d067637380a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92057) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 12:40:26.585678 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4427f46-f447-4c47-99bd-20deba73b960 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:40:26.586509 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4427f46-f447-4c47-99bd-20deba73b960 None admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:40:26.586793 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4427f46-f447-4c47-99bd-20deba73b960 None admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:40:26.587071 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4427f46-f447-4c47-99bd-20deba73b960 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:40:26.618645 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4427f46-f447-4c47-99bd-20deba73b960 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} May 23 12:40:26.626971 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application [None req-e4427f46-f447-4c47-99bd-20deba73b960 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. May 23 12:40:26.626971 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application Traceback (most recent call last): May 23 12:40:26.626971 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request May 23 12:40:26.626971 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application rv = self.dispatch_request() May 23 12:40:26.626971 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request May 23 12:40:26.626971 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) May 23 12:40:26.626971 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper May 23 12:40:26.626971 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) May 23 12:40:26.626971 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view May 23 12:40:26.626971 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) May 23 12:40:26.626971 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request May 23 12:40:26.626971 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) May 23 12:40:26.626971 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get May 23 12:40:26.626971 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self._get_role(role_id) May 23 12:40:26.626971 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role May 23 12:40:26.626971 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') May 23 12:40:26.627942 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call May 23 12:40:26.627942 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( May 23 12:40:26.627942 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data May 23 12:40:26.627942 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) May 23 12:40:26.627942 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped May 23 12:40:26.627942 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) May 23 12:40:26.627942 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun May 23 12:40:26.627942 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) May 23 12:40:26.627942 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func May 23 12:40:26.627942 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self.get_or_create( May 23 12:40:26.627942 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create May 23 12:40:26.627942 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application with Lock( May 23 12:40:26.627942 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ May 23 12:40:26.627942 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self._enter() May 23 12:40:26.627942 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter May 23 12:40:26.627942 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) May 23 12:40:26.629172 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create May 23 12:40:26.629172 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self.creator() May 23 12:40:26.629172 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value May 23 12:40:26.629172 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application created_value = creator( May 23 12:40:26.629172 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role May 23 12:40:26.629172 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) May 23 12:40:26.629172 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role May 23 12:40:26.629172 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() May 23 12:40:26.629172 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role May 23 12:40:26.629172 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) May 23 12:40:26.629172 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. May 23 12:40:26.629172 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application May 23 12:40:26.629172 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 49/101] 10.222.0.30 () {56 vars in 1096 bytes} [Mon May 23 12:40:26 2022] GET /identity/v3/roles/admin => generated 83 bytes in 278 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) May 23 12:40:26.629866 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb75d570-84df-4cd9-a061-418c49fe7c28 None None] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:40:26.629866 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb75d570-84df-4cd9-a061-418c49fe7c28 None None] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:40:26.634118 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb75d570-84df-4cd9-a061-418c49fe7c28 None None] PATH_INFO: `/` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:40:26.635743 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 50/102] 10.222.0.30 () {54 vars in 926 bytes} [Mon May 23 12:40:26 2022] GET /identity => generated 270 bytes in 7 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) May 23 12:40:26.641351 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b4c1955-5491-488b-852e-51d40943e595 None None] REQUEST_METHOD: `POST` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:40:26.641628 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b4c1955-5491-488b-852e-51d40943e595 None None] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:40:26.641928 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b4c1955-5491-488b-852e-51d40943e595 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:40:26.645622 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 53/103] 10.222.0.30 () {58 vars in 1023 bytes} [Mon May 23 12:40:26 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 154 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) May 23 12:40:26.646502 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-898caf88-cb86-4663-8d11-1d067637380a None None] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:40:26.683831 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.auth.core [None req-1b4c1955-5491-488b-852e-51d40943e595 None None] MFA Rules not processed for user `0f7119f8fa5c4445977671140ded91a4`. Rule list: `[]` (Enabled: `True`). {{(pid=92056) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 12:40:26.685514 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.fernet_utils [None req-1b4c1955-5491-488b-852e-51d40943e595 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92056) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 12:40:26.707272 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=100 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:40:26.799039 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 51/104] 10.222.0.30 () {58 vars in 1023 bytes} [Mon May 23 12:40:26 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 158 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) May 23 12:40:26.799794 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b4c1955-5491-488b-852e-51d40943e595 None None] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:40:26.864231 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-898caf88-cb86-4663-8d11-1d067637380a None None] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:40:26.905807 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=100 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:40:26.927385 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc9033ce-49c1-4784-a3fb-dd44c9fbddea None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:40:26.928063 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc9033ce-49c1-4784-a3fb-dd44c9fbddea None admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:40:26.928356 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc9033ce-49c1-4784-a3fb-dd44c9fbddea None admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:40:26.928630 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc9033ce-49c1-4784-a3fb-dd44c9fbddea None admin] PATH_INFO: `/v3/roles` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:40:26.929391 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc9033ce-49c1-4784-a3fb-dd44c9fbddea None admin] RBAC: Adding query filter params (name=admin) {{(pid=92057) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} May 23 12:40:26.929696 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc9033ce-49c1-4784-a3fb-dd44c9fbddea None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:40:26.931348 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc9033ce-49c1-4784-a3fb-dd44c9fbddea None admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:40:26.940321 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 54/105] 10.222.0.30 () {56 vars in 1099 bytes} [Mon May 23 12:40:26 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 294 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) May 23 12:40:26.940950 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc9033ce-49c1-4784-a3fb-dd44c9fbddea None admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:40:26.970337 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b4c1955-5491-488b-852e-51d40943e595 None None] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:40:27.007106 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55e95ac4-fd7a-497c-be35-db893e5a3435 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:40:27.012086 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55e95ac4-fd7a-497c-be35-db893e5a3435 None admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:40:27.012086 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55e95ac4-fd7a-497c-be35-db893e5a3435 None admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:40:27.012086 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55e95ac4-fd7a-497c-be35-db893e5a3435 None admin] PATH_INFO: `/v3/roles/reader` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:40:27.020795 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55e95ac4-fd7a-497c-be35-db893e5a3435 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} May 23 12:40:27.029166 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application [None req-55e95ac4-fd7a-497c-be35-db893e5a3435 None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. May 23 12:40:27.029166 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application Traceback (most recent call last): May 23 12:40:27.029166 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request May 23 12:40:27.029166 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application rv = self.dispatch_request() May 23 12:40:27.029166 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request May 23 12:40:27.029166 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) May 23 12:40:27.029166 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper May 23 12:40:27.029166 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) May 23 12:40:27.029166 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view May 23 12:40:27.029166 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) May 23 12:40:27.029166 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request May 23 12:40:27.029166 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) May 23 12:40:27.029166 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get May 23 12:40:27.029166 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self._get_role(role_id) May 23 12:40:27.029166 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role May 23 12:40:27.029166 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') May 23 12:40:27.030555 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call May 23 12:40:27.030555 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( May 23 12:40:27.030555 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data May 23 12:40:27.030555 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) May 23 12:40:27.030555 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped May 23 12:40:27.030555 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) May 23 12:40:27.030555 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun May 23 12:40:27.030555 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) May 23 12:40:27.030555 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func May 23 12:40:27.030555 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self.get_or_create( May 23 12:40:27.030555 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create May 23 12:40:27.030555 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application with Lock( May 23 12:40:27.030555 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ May 23 12:40:27.030555 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self._enter() May 23 12:40:27.030555 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter May 23 12:40:27.030555 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) May 23 12:40:27.031440 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create May 23 12:40:27.031440 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self.creator() May 23 12:40:27.031440 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value May 23 12:40:27.031440 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application created_value = creator( May 23 12:40:27.031440 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role May 23 12:40:27.031440 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) May 23 12:40:27.031440 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role May 23 12:40:27.031440 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() May 23 12:40:27.031440 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role May 23 12:40:27.031440 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) May 23 12:40:27.031440 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. May 23 12:40:27.031440 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application May 23 12:40:27.033530 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 52/106] 10.222.0.30 () {56 vars in 1099 bytes} [Mon May 23 12:40:26 2022] GET /identity/v3/roles/reader => generated 84 bytes in 234 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) May 23 12:40:27.035220 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63b2b995-4242-4c27-bd38-ea3e3630e8cc None None] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:40:27.035760 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63b2b995-4242-4c27-bd38-ea3e3630e8cc None None] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:40:27.036215 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63b2b995-4242-4c27-bd38-ea3e3630e8cc None None] PATH_INFO: `/` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:40:27.037767 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=100 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:40:27.041925 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 53/107] 10.222.0.30 () {54 vars in 926 bytes} [Mon May 23 12:40:27 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) May 23 12:40:27.048585 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63b2b995-4242-4c27-bd38-ea3e3630e8cc None None] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:40:27.048711 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc9033ce-49c1-4784-a3fb-dd44c9fbddea None admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:40:27.169855 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb8150cc-80fa-427d-8884-d5ae72c59e10 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:40:27.170873 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb8150cc-80fa-427d-8884-d5ae72c59e10 None admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:40:27.171264 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb8150cc-80fa-427d-8884-d5ae72c59e10 None admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:40:27.171648 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb8150cc-80fa-427d-8884-d5ae72c59e10 None admin] PATH_INFO: `/v3/roles/member` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:40:27.185794 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb8150cc-80fa-427d-8884-d5ae72c59e10 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} May 23 12:40:27.196147 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application [None req-eb8150cc-80fa-427d-8884-d5ae72c59e10 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. May 23 12:40:27.196147 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application Traceback (most recent call last): May 23 12:40:27.196147 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request May 23 12:40:27.196147 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application rv = self.dispatch_request() May 23 12:40:27.196147 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request May 23 12:40:27.196147 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) May 23 12:40:27.196147 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper May 23 12:40:27.196147 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) May 23 12:40:27.196147 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view May 23 12:40:27.196147 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) May 23 12:40:27.196147 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request May 23 12:40:27.196147 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) May 23 12:40:27.196147 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get May 23 12:40:27.196147 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self._get_role(role_id) May 23 12:40:27.196147 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role May 23 12:40:27.196147 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') May 23 12:40:27.197505 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call May 23 12:40:27.197505 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( May 23 12:40:27.197505 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data May 23 12:40:27.197505 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) May 23 12:40:27.197505 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped May 23 12:40:27.197505 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) May 23 12:40:27.197505 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun May 23 12:40:27.197505 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) May 23 12:40:27.197505 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func May 23 12:40:27.197505 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self.get_or_create( May 23 12:40:27.197505 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create May 23 12:40:27.197505 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application with Lock( May 23 12:40:27.197505 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ May 23 12:40:27.197505 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self._enter() May 23 12:40:27.197505 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter May 23 12:40:27.197505 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) May 23 12:40:27.198648 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create May 23 12:40:27.198648 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self.creator() May 23 12:40:27.198648 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value May 23 12:40:27.198648 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application created_value = creator( May 23 12:40:27.198648 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role May 23 12:40:27.198648 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) May 23 12:40:27.198648 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role May 23 12:40:27.198648 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() May 23 12:40:27.198648 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role May 23 12:40:27.198648 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) May 23 12:40:27.198648 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. May 23 12:40:27.198648 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application May 23 12:40:27.201957 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 55/108] 10.222.0.30 () {56 vars in 1099 bytes} [Mon May 23 12:40:26 2022] GET /identity/v3/roles/member => generated 84 bytes in 257 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) May 23 12:40:27.203407 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb8150cc-80fa-427d-8884-d5ae72c59e10 None admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:40:27.251198 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63b2b995-4242-4c27-bd38-ea3e3630e8cc None None] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:40:27.260497 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=100 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:40:27.286964 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8fd702fa-bbe9-4c96-b2c3-9d9acb1a3fdd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:40:27.287960 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fd702fa-bbe9-4c96-b2c3-9d9acb1a3fdd None admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:40:27.288323 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fd702fa-bbe9-4c96-b2c3-9d9acb1a3fdd None admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:40:27.288678 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fd702fa-bbe9-4c96-b2c3-9d9acb1a3fdd None admin] PATH_INFO: `/v3/users/0f7119f8fa5c4445977671140ded91a4` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:40:27.290247 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8fd702fa-bbe9-4c96-b2c3-9d9acb1a3fdd None admin] RBAC: Authorizing `identity:get_user(user_id=0f7119f8fa5c4445977671140ded91a4)` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:40:27.291730 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8fd702fa-bbe9-4c96-b2c3-9d9acb1a3fdd None admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:40:27.293667 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 54/109] 10.222.0.30 () {56 vars in 1177 bytes} [Mon May 23 12:40:27 2022] GET /identity/v3/users/0f7119f8fa5c4445977671140ded91a4 => generated 247 bytes in 251 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) May 23 12:40:27.294888 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2625574-8670-4958-8523-2b5eca5bbb44 None None] REQUEST_METHOD: `POST` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:40:27.295186 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2625574-8670-4958-8523-2b5eca5bbb44 None None] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:40:27.295467 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2625574-8670-4958-8523-2b5eca5bbb44 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:40:27.313094 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.auth.core [None req-f2625574-8670-4958-8523-2b5eca5bbb44 None None] MFA Rules not processed for user `0f7119f8fa5c4445977671140ded91a4`. Rule list: `[]` (Enabled: `True`). {{(pid=92056) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 12:40:27.314359 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.fernet_utils [None req-f2625574-8670-4958-8523-2b5eca5bbb44 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92056) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 12:40:27.319354 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb8150cc-80fa-427d-8884-d5ae72c59e10 None admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:40:27.354927 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92840a65-46f1-46aa-a3e1-10aee39c0324 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:40:27.355740 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92840a65-46f1-46aa-a3e1-10aee39c0324 None admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:40:27.356031 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92840a65-46f1-46aa-a3e1-10aee39c0324 None admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:40:27.356313 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92840a65-46f1-46aa-a3e1-10aee39c0324 None admin] PATH_INFO: `/v3/roles` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:40:27.357067 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92840a65-46f1-46aa-a3e1-10aee39c0324 None admin] RBAC: Adding query filter params (name=reader) {{(pid=92057) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} May 23 12:40:27.357378 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92840a65-46f1-46aa-a3e1-10aee39c0324 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:40:27.358799 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92840a65-46f1-46aa-a3e1-10aee39c0324 None admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:40:27.372145 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 56/110] 10.222.0.30 () {56 vars in 1101 bytes} [Mon May 23 12:40:27 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 165 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) May 23 12:40:27.372145 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92840a65-46f1-46aa-a3e1-10aee39c0324 None admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:40:27.387274 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=100 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:40:27.403925 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 55/111] 10.222.0.30 () {58 vars in 1023 bytes} [Mon May 23 12:40:27 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 103 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) May 23 12:40:27.403925 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03ec2419-1ea9-43a6-8167-6d9261af19a4 None None] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:40:27.403925 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03ec2419-1ea9-43a6-8167-6d9261af19a4 None None] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:40:27.403925 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03ec2419-1ea9-43a6-8167-6d9261af19a4 None None] PATH_INFO: `/` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:40:27.403925 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 56/112] 10.222.0.30 () {54 vars in 926 bytes} [Mon May 23 12:40:27 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) May 23 12:40:27.405110 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03ec2419-1ea9-43a6-8167-6d9261af19a4 None None] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:40:27.484883 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92840a65-46f1-46aa-a3e1-10aee39c0324 None admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:40:27.515178 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03ec2419-1ea9-43a6-8167-6d9261af19a4 None None] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:40:27.522122 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b1fb624-829f-4bd1-ba6d-56df73b44dc1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:40:27.522754 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=100 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:40:27.523843 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b1fb624-829f-4bd1-ba6d-56df73b44dc1 None admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:40:27.524305 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b1fb624-829f-4bd1-ba6d-56df73b44dc1 None admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:40:27.524787 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b1fb624-829f-4bd1-ba6d-56df73b44dc1 None admin] PATH_INFO: `/v3/roles` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:40:27.525743 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b1fb624-829f-4bd1-ba6d-56df73b44dc1 None admin] RBAC: Adding query filter params (name=member) {{(pid=92057) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} May 23 12:40:27.526230 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b1fb624-829f-4bd1-ba6d-56df73b44dc1 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:40:27.528609 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b1fb624-829f-4bd1-ba6d-56df73b44dc1 None admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:40:27.537775 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 57/113] 10.222.0.30 () {56 vars in 1101 bytes} [Mon May 23 12:40:27 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 170 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) May 23 12:40:27.538860 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b1fb624-829f-4bd1-ba6d-56df73b44dc1 None admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:40:27.555450 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0001a51-e32c-42c4-b2db-ce30ec16a861 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:40:27.556592 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0001a51-e32c-42c4-b2db-ce30ec16a861 None admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:40:27.557084 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0001a51-e32c-42c4-b2db-ce30ec16a861 None admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:40:27.557625 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0001a51-e32c-42c4-b2db-ce30ec16a861 None admin] PATH_INFO: `/v3/projects/486401ae2e5f40d5a5f060242896ae53` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:40:27.559618 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0001a51-e32c-42c4-b2db-ce30ec16a861 None admin] RBAC: Authorizing `identity:get_project(project_id=486401ae2e5f40d5a5f060242896ae53)` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:40:27.561079 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0001a51-e32c-42c4-b2db-ce30ec16a861 None admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:40:27.562831 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 57/114] 10.222.0.30 () {56 vars in 1186 bytes} [Mon May 23 12:40:27 2022] GET /identity/v3/projects/486401ae2e5f40d5a5f060242896ae53 => generated 298 bytes in 163 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) May 23 12:40:27.567431 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ace4a458-105e-4da8-8aa5-dc25bcb74b28 None None] REQUEST_METHOD: `POST` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:40:27.567431 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ace4a458-105e-4da8-8aa5-dc25bcb74b28 None None] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:40:27.567431 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ace4a458-105e-4da8-8aa5-dc25bcb74b28 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:40:27.580564 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.auth.core [None req-ace4a458-105e-4da8-8aa5-dc25bcb74b28 None None] MFA Rules not processed for user `0f7119f8fa5c4445977671140ded91a4`. Rule list: `[]` (Enabled: `True`). {{(pid=92056) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 12:40:27.581734 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.fernet_utils [None req-ace4a458-105e-4da8-8aa5-dc25bcb74b28 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92056) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 12:40:27.628444 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=100 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:40:27.657020 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 58/115] 10.222.0.30 () {58 vars in 1023 bytes} [Mon May 23 12:40:27 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 89 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) May 23 12:40:27.657242 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b1fb624-829f-4bd1-ba6d-56df73b44dc1 None admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:40:27.657339 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ace4a458-105e-4da8-8aa5-dc25bcb74b28 None None] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:40:27.701397 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d726852-2e9a-459d-803d-a206f8e2dd89 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:40:27.702328 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d726852-2e9a-459d-803d-a206f8e2dd89 None admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:40:27.702622 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d726852-2e9a-459d-803d-a206f8e2dd89 None admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:40:27.702907 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d726852-2e9a-459d-803d-a206f8e2dd89 None admin] PATH_INFO: `/v3/users/5c3e3c32e6c8497c8e3d98e09b8de7be` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:40:27.718545 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d726852-2e9a-459d-803d-a206f8e2dd89 None admin] RBAC: Authorizing `identity:get_user(user_id=5c3e3c32e6c8497c8e3d98e09b8de7be)` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:40:27.720054 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d726852-2e9a-459d-803d-a206f8e2dd89 None admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:40:27.726278 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 58/116] 10.222.0.30 () {56 vars in 1177 bytes} [Mon May 23 12:40:27 2022] GET /identity/v3/users/5c3e3c32e6c8497c8e3d98e09b8de7be => generated 289 bytes in 188 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) May 23 12:40:27.727221 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d726852-2e9a-459d-803d-a206f8e2dd89 None admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:40:27.780907 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ace4a458-105e-4da8-8aa5-dc25bcb74b28 None None] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:40:27.784270 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=100 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:40:27.815731 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef35ede6-dfc5-438b-a239-3c3bf26f0546 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:40:27.816538 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef35ede6-dfc5-438b-a239-3c3bf26f0546 None admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:40:27.816821 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef35ede6-dfc5-438b-a239-3c3bf26f0546 None admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:40:27.817103 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef35ede6-dfc5-438b-a239-3c3bf26f0546 None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:40:27.827874 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef35ede6-dfc5-438b-a239-3c3bf26f0546 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} May 23 12:40:27.832216 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d726852-2e9a-459d-803d-a206f8e2dd89 None admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:40:27.836154 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application [None req-ef35ede6-dfc5-438b-a239-3c3bf26f0546 None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. May 23 12:40:27.836154 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application Traceback (most recent call last): May 23 12:40:27.836154 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request May 23 12:40:27.836154 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application rv = self.dispatch_request() May 23 12:40:27.836154 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request May 23 12:40:27.836154 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) May 23 12:40:27.836154 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper May 23 12:40:27.836154 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) May 23 12:40:27.836154 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view May 23 12:40:27.836154 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) May 23 12:40:27.836154 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request May 23 12:40:27.836154 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) May 23 12:40:27.836154 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get May 23 12:40:27.836154 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self._get_role(role_id) May 23 12:40:27.836154 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role May 23 12:40:27.836154 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') May 23 12:40:27.837217 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call May 23 12:40:27.837217 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( May 23 12:40:27.837217 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data May 23 12:40:27.837217 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) May 23 12:40:27.837217 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped May 23 12:40:27.837217 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) May 23 12:40:27.837217 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun May 23 12:40:27.837217 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) May 23 12:40:27.837217 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func May 23 12:40:27.837217 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self.get_or_create( May 23 12:40:27.837217 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create May 23 12:40:27.837217 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application with Lock( May 23 12:40:27.837217 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ May 23 12:40:27.837217 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self._enter() May 23 12:40:27.837217 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter May 23 12:40:27.837217 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) May 23 12:40:27.838285 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create May 23 12:40:27.838285 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self.creator() May 23 12:40:27.838285 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value May 23 12:40:27.838285 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application created_value = creator( May 23 12:40:27.838285 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role May 23 12:40:27.838285 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) May 23 12:40:27.838285 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role May 23 12:40:27.838285 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() May 23 12:40:27.838285 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role May 23 12:40:27.838285 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) May 23 12:40:27.838285 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. May 23 12:40:27.838285 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application May 23 12:40:27.838285 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 59/117] 10.222.0.30 () {56 vars in 1114 bytes} [Mon May 23 12:40:27 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 185 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) May 23 12:40:27.838285 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef35ede6-dfc5-438b-a239-3c3bf26f0546 None admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:40:27.878215 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4959dac-8020-4308-bf8a-66ea0248cb12 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:40:27.879086 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4959dac-8020-4308-bf8a-66ea0248cb12 None admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:40:27.879370 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4959dac-8020-4308-bf8a-66ea0248cb12 None admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:40:27.879647 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4959dac-8020-4308-bf8a-66ea0248cb12 None admin] PATH_INFO: `/v3/users/b3478ea957234bb99fde27e6c3d0dfd3` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:40:27.880824 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4959dac-8020-4308-bf8a-66ea0248cb12 None admin] RBAC: Authorizing `identity:get_user(user_id=b3478ea957234bb99fde27e6c3d0dfd3)` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:40:27.882307 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4959dac-8020-4308-bf8a-66ea0248cb12 None admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:40:27.883940 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 59/118] 10.222.0.30 () {56 vars in 1177 bytes} [Mon May 23 12:40:27 2022] GET /identity/v3/users/b3478ea957234bb99fde27e6c3d0dfd3 => generated 275 bytes in 157 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) May 23 12:40:27.889553 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4959dac-8020-4308-bf8a-66ea0248cb12 None admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:40:27.892957 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=100 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:40:27.941304 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef35ede6-dfc5-438b-a239-3c3bf26f0546 None admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:40:27.975401 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2f678b6-e8d5-450c-b062-328b54066208 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:40:27.976292 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2f678b6-e8d5-450c-b062-328b54066208 None admin] REQUEST_METHOD: `PUT` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:40:27.976582 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2f678b6-e8d5-450c-b062-328b54066208 None admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:40:27.976867 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2f678b6-e8d5-450c-b062-328b54066208 None admin] PATH_INFO: `/v3/projects/486401ae2e5f40d5a5f060242896ae53/users/0f7119f8fa5c4445977671140ded91a4/roles/bc3c3c9d654940cbaa5177a6be23d1bc` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:40:27.978876 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2f678b6-e8d5-450c-b062-328b54066208 None admin] RBAC: Authorizing `identity:create_grant(project_id=486401ae2e5f40d5a5f060242896ae53, user_id=0f7119f8fa5c4445977671140ded91a4, role_id=bc3c3c9d654940cbaa5177a6be23d1bc)` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:40:27.980399 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2f678b6-e8d5-450c-b062-328b54066208 None admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:40:27.989309 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4959dac-8020-4308-bf8a-66ea0248cb12 None admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:40:28.024056 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4dde6e8-8bee-41ee-8f7b-b9ce314ac9f5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:40:28.024839 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4dde6e8-8bee-41ee-8f7b-b9ce314ac9f5 None admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:40:28.025127 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4dde6e8-8bee-41ee-8f7b-b9ce314ac9f5 None admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:40:28.025406 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4dde6e8-8bee-41ee-8f7b-b9ce314ac9f5 None admin] PATH_INFO: `/v3/domains/default` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:40:28.026594 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c4dde6e8-8bee-41ee-8f7b-b9ce314ac9f5 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:40:28.028564 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c4dde6e8-8bee-41ee-8f7b-b9ce314ac9f5 None admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:40:28.030480 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 60/119] 10.222.0.30 () {56 vars in 1108 bytes} [Mon May 23 12:40:27 2022] GET /identity/v3/domains/default => generated 199 bytes in 141 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) May 23 12:40:28.032915 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4dde6e8-8bee-41ee-8f7b-b9ce314ac9f5 None admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:40:28.099021 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=100 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:40:28.100204 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 60/120] 10.222.0.30 () {58 vars in 1439 bytes} [Mon May 23 12:40:27 2022] PUT /identity/v3/projects/486401ae2e5f40d5a5f060242896ae53/users/0f7119f8fa5c4445977671140ded91a4/roles/bc3c3c9d654940cbaa5177a6be23d1bc => generated 0 bytes in 262 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) May 23 12:40:28.101615 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa30c65f-0283-46a6-84ce-ffb18ec3ea2c None None] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:40:28.101925 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa30c65f-0283-46a6-84ce-ffb18ec3ea2c None None] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:40:28.106112 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa30c65f-0283-46a6-84ce-ffb18ec3ea2c None None] PATH_INFO: `/` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:40:28.106112 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 61/121] 10.222.0.30 () {54 vars in 926 bytes} [Mon May 23 12:40:28 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) May 23 12:40:28.106112 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa30c65f-0283-46a6-84ce-ffb18ec3ea2c None None] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:40:28.134805 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:40:28.141929 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4dde6e8-8bee-41ee-8f7b-b9ce314ac9f5 None admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:40:28.181119 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aead9146-00c8-44a3-a143-efd0816cb685 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:40:28.181791 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aead9146-00c8-44a3-a143-efd0816cb685 None admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:40:28.182111 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aead9146-00c8-44a3-a143-efd0816cb685 None admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:40:28.182391 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aead9146-00c8-44a3-a143-efd0816cb685 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:40:28.225030 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa30c65f-0283-46a6-84ce-ffb18ec3ea2c None None] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:40:28.256330 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aead9146-00c8-44a3-a143-efd0816cb685 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:40:28.257917 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aead9146-00c8-44a3-a143-efd0816cb685 None admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:40:28.264136 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d36d2c95-edd9-43b2-9ac3-056fa02e09f6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:40:28.264826 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d36d2c95-edd9-43b2-9ac3-056fa02e09f6 None admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:40:28.265104 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d36d2c95-edd9-43b2-9ac3-056fa02e09f6 None admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:40:28.265379 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d36d2c95-edd9-43b2-9ac3-056fa02e09f6 None admin] PATH_INFO: `/v3/roles` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:40:28.266170 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d36d2c95-edd9-43b2-9ac3-056fa02e09f6 None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=92056) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} May 23 12:40:28.266480 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d36d2c95-edd9-43b2-9ac3-056fa02e09f6 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:40:28.267878 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d36d2c95-edd9-43b2-9ac3-056fa02e09f6 None admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:40:28.277509 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 62/122] 10.222.0.30 () {56 vars in 1111 bytes} [Mon May 23 12:40:28 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 173 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) May 23 12:40:28.278300 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d36d2c95-edd9-43b2-9ac3-056fa02e09f6 None admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:40:28.341737 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=100 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:40:28.372393 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 61/123] 10.222.0.30 () {58 vars in 1282 bytes} [Mon May 23 12:40:28 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 341 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) May 23 12:40:28.373522 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aead9146-00c8-44a3-a143-efd0816cb685 None admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:40:28.383223 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=100 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:40:28.399073 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d36d2c95-edd9-43b2-9ac3-056fa02e09f6 None admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:40:28.435822 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c051681-f1c7-4823-80ed-1fd901eaac11 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:40:28.436691 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c051681-f1c7-4823-80ed-1fd901eaac11 None admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:40:28.436976 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c051681-f1c7-4823-80ed-1fd901eaac11 None admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:40:28.437261 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c051681-f1c7-4823-80ed-1fd901eaac11 None admin] PATH_INFO: `/v3/projects/486401ae2e5f40d5a5f060242896ae53` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:40:28.438503 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c051681-f1c7-4823-80ed-1fd901eaac11 None admin] RBAC: Authorizing `identity:get_project(project_id=486401ae2e5f40d5a5f060242896ae53)` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:40:28.439964 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c051681-f1c7-4823-80ed-1fd901eaac11 None admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:40:28.441716 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 63/124] 10.222.0.30 () {56 vars in 1186 bytes} [Mon May 23 12:40:28 2022] GET /identity/v3/projects/486401ae2e5f40d5a5f060242896ae53 => generated 298 bytes in 164 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) May 23 12:40:28.442899 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10210d2e-52b5-4147-ba8b-0cca67466535 None None] REQUEST_METHOD: `POST` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:40:28.443185 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10210d2e-52b5-4147-ba8b-0cca67466535 None None] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:40:28.443465 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10210d2e-52b5-4147-ba8b-0cca67466535 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:40:28.459862 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.auth.core [None req-10210d2e-52b5-4147-ba8b-0cca67466535 None None] MFA Rules not processed for user `0f7119f8fa5c4445977671140ded91a4`. Rule list: `[]` (Enabled: `True`). {{(pid=92056) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 12:40:28.461064 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.fernet_utils [None req-10210d2e-52b5-4147-ba8b-0cca67466535 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92056) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 12:40:28.490485 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aead9146-00c8-44a3-a143-efd0816cb685 None admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:40:28.532280 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96d42f82-45ae-4359-ba81-7a210b82cb19 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:40:28.533204 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96d42f82-45ae-4359-ba81-7a210b82cb19 None admin] REQUEST_METHOD: `POST` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:40:28.533640 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96d42f82-45ae-4359-ba81-7a210b82cb19 None admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:40:28.534087 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96d42f82-45ae-4359-ba81-7a210b82cb19 None admin] PATH_INFO: `/v3/projects` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:40:28.534937 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96d42f82-45ae-4359-ba81-7a210b82cb19 None admin] RBAC: Authorizing `identity:create_project()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:40:28.536474 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96d42f82-45ae-4359-ba81-7a210b82cb19 None admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:40:28.549537 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 64/125] 10.222.0.30 () {58 vars in 1023 bytes} [Mon May 23 12:40:28 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 107 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) May 23 12:40:28.554755 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10210d2e-52b5-4147-ba8b-0cca67466535 None None] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:40:28.624102 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 62/126] 10.222.0.30 () {60 vars in 1141 bytes} [Mon May 23 12:40:28 2022] POST /identity/v3/projects => generated 302 bytes in 251 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) May 23 12:40:28.631030 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96d42f82-45ae-4359-ba81-7a210b82cb19 None admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:40:28.647906 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=100 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:40:28.669991 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:40:28.676460 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10210d2e-52b5-4147-ba8b-0cca67466535 None None] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:40:28.731052 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d351ca7-b0b8-4840-9a0b-01adfc87f342 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:40:28.731966 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d351ca7-b0b8-4840-9a0b-01adfc87f342 None admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:40:28.732622 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d351ca7-b0b8-4840-9a0b-01adfc87f342 None admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:40:28.732622 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d351ca7-b0b8-4840-9a0b-01adfc87f342 None admin] PATH_INFO: `/v3/users/b3478ea957234bb99fde27e6c3d0dfd3` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:40:28.733843 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d351ca7-b0b8-4840-9a0b-01adfc87f342 None admin] RBAC: Authorizing `identity:get_user(user_id=b3478ea957234bb99fde27e6c3d0dfd3)` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:40:28.735803 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d351ca7-b0b8-4840-9a0b-01adfc87f342 None admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:40:28.737772 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 65/127] 10.222.0.30 () {56 vars in 1177 bytes} [Mon May 23 12:40:28 2022] GET /identity/v3/users/b3478ea957234bb99fde27e6c3d0dfd3 => generated 275 bytes in 188 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) May 23 12:40:28.738582 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d351ca7-b0b8-4840-9a0b-01adfc87f342 None admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:40:28.795968 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96d42f82-45ae-4359-ba81-7a210b82cb19 None admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:40:28.849634 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4252193d-56c1-4109-be81-a5ef6dee397a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:40:28.850570 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4252193d-56c1-4109-be81-a5ef6dee397a None admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:40:28.851648 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4252193d-56c1-4109-be81-a5ef6dee397a None admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:40:28.851648 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4252193d-56c1-4109-be81-a5ef6dee397a None admin] PATH_INFO: `/v3/projects/486401ae2e5f40d5a5f060242896ae53` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:40:28.852720 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4252193d-56c1-4109-be81-a5ef6dee397a None admin] RBAC: Authorizing `identity:get_project(project_id=486401ae2e5f40d5a5f060242896ae53)` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:40:28.854381 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4252193d-56c1-4109-be81-a5ef6dee397a None admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:40:28.856887 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 63/128] 10.222.0.30 () {56 vars in 1186 bytes} [Mon May 23 12:40:28 2022] GET /identity/v3/projects/486401ae2e5f40d5a5f060242896ae53 => generated 298 bytes in 226 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) May 23 12:40:28.857731 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4252193d-56c1-4109-be81-a5ef6dee397a None admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:40:28.866968 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d351ca7-b0b8-4840-9a0b-01adfc87f342 None admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:40:28.912597 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b146ebf-b6b2-4d74-924d-59b0322ba3df None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:40:28.913816 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b146ebf-b6b2-4d74-924d-59b0322ba3df None admin] REQUEST_METHOD: `PUT` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:40:28.914281 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b146ebf-b6b2-4d74-924d-59b0322ba3df None admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:40:28.914790 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b146ebf-b6b2-4d74-924d-59b0322ba3df None admin] PATH_INFO: `/v3/projects/486401ae2e5f40d5a5f060242896ae53/users/b3478ea957234bb99fde27e6c3d0dfd3/roles/80fb22507c9a4ff09d44d8f12f6fff1a` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:40:28.917025 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b146ebf-b6b2-4d74-924d-59b0322ba3df None admin] RBAC: Authorizing `identity:create_grant(project_id=486401ae2e5f40d5a5f060242896ae53, user_id=b3478ea957234bb99fde27e6c3d0dfd3, role_id=80fb22507c9a4ff09d44d8f12f6fff1a)` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:40:28.918997 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b146ebf-b6b2-4d74-924d-59b0322ba3df None admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:40:28.963351 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=100 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:40:28.971917 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4252193d-56c1-4109-be81-a5ef6dee397a None admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:40:29.006149 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c8d5c74-711c-4035-bb7f-21ee0c581994 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:40:29.007000 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c8d5c74-711c-4035-bb7f-21ee0c581994 None admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:40:29.007298 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c8d5c74-711c-4035-bb7f-21ee0c581994 None admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:40:29.007581 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c8d5c74-711c-4035-bb7f-21ee0c581994 None admin] PATH_INFO: `/v3/roles/member` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:40:29.017508 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 66/129] 10.222.0.30 () {58 vars in 1439 bytes} [Mon May 23 12:40:28 2022] PUT /identity/v3/projects/486401ae2e5f40d5a5f060242896ae53/users/b3478ea957234bb99fde27e6c3d0dfd3/roles/80fb22507c9a4ff09d44d8f12f6fff1a => generated 0 bytes in 279 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) May 23 12:40:29.019224 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b146ebf-b6b2-4d74-924d-59b0322ba3df None admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:40:29.019644 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c8d5c74-711c-4035-bb7f-21ee0c581994 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} May 23 12:40:29.028366 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application [None req-8c8d5c74-711c-4035-bb7f-21ee0c581994 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. May 23 12:40:29.028366 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application Traceback (most recent call last): May 23 12:40:29.028366 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request May 23 12:40:29.028366 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application rv = self.dispatch_request() May 23 12:40:29.028366 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request May 23 12:40:29.028366 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) May 23 12:40:29.028366 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper May 23 12:40:29.028366 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) May 23 12:40:29.028366 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view May 23 12:40:29.028366 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) May 23 12:40:29.028366 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request May 23 12:40:29.028366 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) May 23 12:40:29.028366 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get May 23 12:40:29.028366 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self._get_role(role_id) May 23 12:40:29.028366 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role May 23 12:40:29.028366 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') May 23 12:40:29.029346 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call May 23 12:40:29.029346 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( May 23 12:40:29.029346 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data May 23 12:40:29.029346 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) May 23 12:40:29.029346 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped May 23 12:40:29.029346 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) May 23 12:40:29.029346 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun May 23 12:40:29.029346 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) May 23 12:40:29.029346 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func May 23 12:40:29.029346 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self.get_or_create( May 23 12:40:29.029346 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create May 23 12:40:29.029346 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application with Lock( May 23 12:40:29.029346 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ May 23 12:40:29.029346 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self._enter() May 23 12:40:29.029346 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter May 23 12:40:29.029346 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) May 23 12:40:29.031429 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create May 23 12:40:29.031429 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self.creator() May 23 12:40:29.031429 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value May 23 12:40:29.031429 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application created_value = creator( May 23 12:40:29.031429 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role May 23 12:40:29.031429 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) May 23 12:40:29.031429 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role May 23 12:40:29.031429 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() May 23 12:40:29.031429 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role May 23 12:40:29.031429 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) May 23 12:40:29.031429 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. May 23 12:40:29.031429 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application May 23 12:40:29.031429 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 64/130] 10.222.0.30 () {56 vars in 1099 bytes} [Mon May 23 12:40:28 2022] GET /identity/v3/roles/member => generated 84 bytes in 172 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) May 23 12:40:29.031429 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c8d5c74-711c-4035-bb7f-21ee0c581994 None admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:40:29.045024 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:40:29.142418 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c8d5c74-711c-4035-bb7f-21ee0c581994 None admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:40:29.150761 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b146ebf-b6b2-4d74-924d-59b0322ba3df None admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:40:29.179498 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d401b9d-080d-4640-ba6e-091a535dd7cf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:40:29.180087 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d401b9d-080d-4640-ba6e-091a535dd7cf None admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:40:29.180377 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d401b9d-080d-4640-ba6e-091a535dd7cf None admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:40:29.180970 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d401b9d-080d-4640-ba6e-091a535dd7cf None admin] PATH_INFO: `/v3/role_assignments` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:40:29.181505 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d401b9d-080d-4640-ba6e-091a535dd7cf None admin] RBAC: Adding query filter params (role.id=c223286ed21a4654a0f2b4e5551831c7, scope.project.id=486401ae2e5f40d5a5f060242896ae53, user.id=5c3e3c32e6c8497c8e3d98e09b8de7be) {{(pid=92057) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} May 23 12:40:29.181821 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d401b9d-080d-4640-ba6e-091a535dd7cf None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:40:29.183421 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d401b9d-080d-4640-ba6e-091a535dd7cf None admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:40:29.193902 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 65/131] 10.222.0.30 () {56 vars in 1374 bytes} [Mon May 23 12:40:29 2022] GET /identity/v3/role_assignments?user.id=5c3e3c32e6c8497c8e3d98e09b8de7be&scope.project.id=486401ae2e5f40d5a5f060242896ae53&role.id=c223286ed21a4654a0f2b4e5551831c7 => generated 259 bytes in 160 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) May 23 12:40:29.193902 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d401b9d-080d-4640-ba6e-091a535dd7cf None admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:40:29.208539 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53b48cb4-76f6-4a20-92a8-d5b84dc5e3e4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:40:29.209441 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53b48cb4-76f6-4a20-92a8-d5b84dc5e3e4 None admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:40:29.209778 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53b48cb4-76f6-4a20-92a8-d5b84dc5e3e4 None admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:40:29.210089 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53b48cb4-76f6-4a20-92a8-d5b84dc5e3e4 None admin] PATH_INFO: `/v3/projects/486401ae2e5f40d5a5f060242896ae53` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:40:29.215141 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53b48cb4-76f6-4a20-92a8-d5b84dc5e3e4 None admin] RBAC: Authorizing `identity:get_project(project_id=486401ae2e5f40d5a5f060242896ae53)` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:40:29.216653 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53b48cb4-76f6-4a20-92a8-d5b84dc5e3e4 None admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:40:29.218939 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 67/132] 10.222.0.30 () {56 vars in 1186 bytes} [Mon May 23 12:40:29 2022] GET /identity/v3/projects/486401ae2e5f40d5a5f060242896ae53 => generated 298 bytes in 200 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) May 23 12:40:29.224621 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53b48cb4-76f6-4a20-92a8-d5b84dc5e3e4 None admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:40:29.238284 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=100 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:40:29.380743 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=100 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:40:29.412221 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53b48cb4-76f6-4a20-92a8-d5b84dc5e3e4 None admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:40:29.441582 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d401b9d-080d-4640-ba6e-091a535dd7cf None admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:40:29.456800 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ddc6b2c5-2462-416b-832e-069b11cb63c5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:40:29.458305 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddc6b2c5-2462-416b-832e-069b11cb63c5 None admin] REQUEST_METHOD: `PUT` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:40:29.458697 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddc6b2c5-2462-416b-832e-069b11cb63c5 None admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:40:29.459079 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddc6b2c5-2462-416b-832e-069b11cb63c5 None admin] PATH_INFO: `/v3/projects/486401ae2e5f40d5a5f060242896ae53/users/b3478ea957234bb99fde27e6c3d0dfd3/roles/e3b90948645643e888620ab1e5c774bf` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:40:29.462376 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ddc6b2c5-2462-416b-832e-069b11cb63c5 None admin] RBAC: Authorizing `identity:create_grant(project_id=486401ae2e5f40d5a5f060242896ae53, user_id=b3478ea957234bb99fde27e6c3d0dfd3, role_id=e3b90948645643e888620ab1e5c774bf)` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:40:29.465153 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ddc6b2c5-2462-416b-832e-069b11cb63c5 None admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:40:29.489679 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6ec5e18-c181-43e0-82db-8ae4256b1947 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:40:29.490398 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6ec5e18-c181-43e0-82db-8ae4256b1947 None admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:40:29.490683 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6ec5e18-c181-43e0-82db-8ae4256b1947 None admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:40:29.490964 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6ec5e18-c181-43e0-82db-8ae4256b1947 None admin] PATH_INFO: `/v3/roles` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:40:29.491705 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6ec5e18-c181-43e0-82db-8ae4256b1947 None admin] RBAC: Adding query filter params (name=member) {{(pid=92057) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} May 23 12:40:29.492013 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6ec5e18-c181-43e0-82db-8ae4256b1947 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:40:29.493413 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6ec5e18-c181-43e0-82db-8ae4256b1947 None admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:40:29.503484 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 66/133] 10.222.0.30 () {56 vars in 1101 bytes} [Mon May 23 12:40:29 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 313 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) May 23 12:40:29.509621 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6ec5e18-c181-43e0-82db-8ae4256b1947 None admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:40:29.551963 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 68/134] 10.222.0.30 () {58 vars in 1439 bytes} [Mon May 23 12:40:29 2022] PUT /identity/v3/projects/486401ae2e5f40d5a5f060242896ae53/users/b3478ea957234bb99fde27e6c3d0dfd3/roles/e3b90948645643e888620ab1e5c774bf => generated 0 bytes in 328 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) May 23 12:40:29.604140 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=100 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:40:29.636819 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6ec5e18-c181-43e0-82db-8ae4256b1947 None admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:40:29.677666 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c5207c0-8180-461a-b87b-4e8167607925 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:40:29.678618 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c5207c0-8180-461a-b87b-4e8167607925 None admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:40:29.678912 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c5207c0-8180-461a-b87b-4e8167607925 None admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:40:29.679196 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c5207c0-8180-461a-b87b-4e8167607925 None admin] PATH_INFO: `/v3/users/b3478ea957234bb99fde27e6c3d0dfd3` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:40:29.680573 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c5207c0-8180-461a-b87b-4e8167607925 None admin] RBAC: Authorizing `identity:get_user(user_id=b3478ea957234bb99fde27e6c3d0dfd3)` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:40:29.682153 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c5207c0-8180-461a-b87b-4e8167607925 None admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:40:29.684592 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 67/135] 10.222.0.30 () {56 vars in 1177 bytes} [Mon May 23 12:40:29 2022] GET /identity/v3/users/b3478ea957234bb99fde27e6c3d0dfd3 => generated 275 bytes in 175 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) May 23 12:40:29.689674 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ddc6b2c5-2462-416b-832e-069b11cb63c5 None admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:40:29.824864 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ddc6b2c5-2462-416b-832e-069b11cb63c5 None admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:40:29.868378 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee3db66e-1ac2-4922-931b-61b1e24a1ee7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:40:29.869282 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee3db66e-1ac2-4922-931b-61b1e24a1ee7 None admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:40:29.869550 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee3db66e-1ac2-4922-931b-61b1e24a1ee7 None admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:40:29.869832 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee3db66e-1ac2-4922-931b-61b1e24a1ee7 None admin] PATH_INFO: `/v3/projects/40ed642151e34cbfa138265d51948402` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:40:29.871120 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee3db66e-1ac2-4922-931b-61b1e24a1ee7 None admin] RBAC: Authorizing `identity:get_project(project_id=40ed642151e34cbfa138265d51948402)` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:40:29.872858 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee3db66e-1ac2-4922-931b-61b1e24a1ee7 None admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:40:29.874952 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 69/136] 10.222.0.30 () {56 vars in 1186 bytes} [Mon May 23 12:40:29 2022] GET /identity/v3/projects/40ed642151e34cbfa138265d51948402 => generated 312 bytes in 186 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) May 23 12:40:29.882116 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c5207c0-8180-461a-b87b-4e8167607925 None admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:40:30.002440 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c5207c0-8180-461a-b87b-4e8167607925 None admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:40:30.043517 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5614cce-e3ce-4f25-b6fa-9901dd1e03a8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:40:30.044420 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5614cce-e3ce-4f25-b6fa-9901dd1e03a8 None admin] REQUEST_METHOD: `PUT` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:40:30.044709 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5614cce-e3ce-4f25-b6fa-9901dd1e03a8 None admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:40:30.044988 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5614cce-e3ce-4f25-b6fa-9901dd1e03a8 None admin] PATH_INFO: `/v3/projects/40ed642151e34cbfa138265d51948402/users/b3478ea957234bb99fde27e6c3d0dfd3/roles/80fb22507c9a4ff09d44d8f12f6fff1a` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:40:30.047129 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5614cce-e3ce-4f25-b6fa-9901dd1e03a8 None admin] RBAC: Authorizing `identity:create_grant(project_id=40ed642151e34cbfa138265d51948402, user_id=b3478ea957234bb99fde27e6c3d0dfd3, role_id=80fb22507c9a4ff09d44d8f12f6fff1a)` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:40:30.048614 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5614cce-e3ce-4f25-b6fa-9901dd1e03a8 None admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:40:30.141455 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 68/137] 10.222.0.30 () {58 vars in 1439 bytes} [Mon May 23 12:40:29 2022] PUT /identity/v3/projects/40ed642151e34cbfa138265d51948402/users/b3478ea957234bb99fde27e6c3d0dfd3/roles/80fb22507c9a4ff09d44d8f12f6fff1a => generated 0 bytes in 260 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) May 23 12:40:30.945398 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96a2de8b-9027-4abe-ba89-3ed476b35058 None None] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:40:30.945942 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96a2de8b-9027-4abe-ba89-3ed476b35058 None None] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:40:30.945942 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96a2de8b-9027-4abe-ba89-3ed476b35058 None None] PATH_INFO: `/` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:40:30.947343 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 70/138] 10.222.0.30 () {54 vars in 926 bytes} [Mon May 23 12:40:30 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) May 23 12:40:30.959933 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df1ce8af-75a6-4d85-beba-8490efcf8342 None None] REQUEST_METHOD: `POST` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:40:30.960429 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df1ce8af-75a6-4d85-beba-8490efcf8342 None None] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:40:30.960501 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df1ce8af-75a6-4d85-beba-8490efcf8342 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:40:30.976686 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:40:30.985952 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.auth.core [None req-df1ce8af-75a6-4d85-beba-8490efcf8342 None None] MFA Rules not processed for user `0f7119f8fa5c4445977671140ded91a4`. Rule list: `[]` (Enabled: `True`). {{(pid=92057) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 12:40:30.987144 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.fernet_utils [None req-df1ce8af-75a6-4d85-beba-8490efcf8342 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92057) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 12:40:31.066345 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 69/139] 10.222.0.30 () {58 vars in 1023 bytes} [Mon May 23 12:40:30 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 107 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) May 23 12:40:31.109825 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96a2de8b-9027-4abe-ba89-3ed476b35058 None None] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:40:31.131891 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:40:31.262566 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96a2de8b-9027-4abe-ba89-3ed476b35058 None None] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:40:31.310203 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44e35b0e-e91e-4903-a7d4-edeb71c7dce0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:40:31.311065 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44e35b0e-e91e-4903-a7d4-edeb71c7dce0 None admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:40:31.311352 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44e35b0e-e91e-4903-a7d4-edeb71c7dce0 None admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:40:31.311633 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44e35b0e-e91e-4903-a7d4-edeb71c7dce0 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:40:31.324504 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44e35b0e-e91e-4903-a7d4-edeb71c7dce0 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} May 23 12:40:31.332341 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application [None req-44e35b0e-e91e-4903-a7d4-edeb71c7dce0 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. May 23 12:40:31.332341 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application Traceback (most recent call last): May 23 12:40:31.332341 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request May 23 12:40:31.332341 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application rv = self.dispatch_request() May 23 12:40:31.332341 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request May 23 12:40:31.332341 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) May 23 12:40:31.332341 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper May 23 12:40:31.332341 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) May 23 12:40:31.332341 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view May 23 12:40:31.332341 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) May 23 12:40:31.332341 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request May 23 12:40:31.332341 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) May 23 12:40:31.332341 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get May 23 12:40:31.332341 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self._get_role(role_id) May 23 12:40:31.332341 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role May 23 12:40:31.332341 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') May 23 12:40:31.333199 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call May 23 12:40:31.333199 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( May 23 12:40:31.333199 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data May 23 12:40:31.333199 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) May 23 12:40:31.333199 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped May 23 12:40:31.333199 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) May 23 12:40:31.333199 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun May 23 12:40:31.333199 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) May 23 12:40:31.333199 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func May 23 12:40:31.333199 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self.get_or_create( May 23 12:40:31.333199 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create May 23 12:40:31.333199 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application with Lock( May 23 12:40:31.333199 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ May 23 12:40:31.333199 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self._enter() May 23 12:40:31.333199 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter May 23 12:40:31.333199 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) May 23 12:40:31.334406 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create May 23 12:40:31.334406 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self.creator() May 23 12:40:31.334406 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value May 23 12:40:31.334406 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application created_value = creator( May 23 12:40:31.334406 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role May 23 12:40:31.334406 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) May 23 12:40:31.334406 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role May 23 12:40:31.334406 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() May 23 12:40:31.334406 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role May 23 12:40:31.334406 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) May 23 12:40:31.334406 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. May 23 12:40:31.334406 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application May 23 12:40:31.334406 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 71/140] 10.222.0.30 () {56 vars in 1096 bytes} [Mon May 23 12:40:31 2022] GET /identity/v3/roles/admin => generated 83 bytes in 224 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) May 23 12:40:31.339521 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df1ce8af-75a6-4d85-beba-8490efcf8342 None None] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:40:31.486040 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df1ce8af-75a6-4d85-beba-8490efcf8342 None None] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:40:31.534710 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5af1e1e-b16d-4f0c-8687-4615b8ac25d8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:40:31.535321 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5af1e1e-b16d-4f0c-8687-4615b8ac25d8 None admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:40:31.535610 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5af1e1e-b16d-4f0c-8687-4615b8ac25d8 None admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:40:31.535885 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5af1e1e-b16d-4f0c-8687-4615b8ac25d8 None admin] PATH_INFO: `/v3/roles` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:40:31.536646 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5af1e1e-b16d-4f0c-8687-4615b8ac25d8 None admin] RBAC: Adding query filter params (name=admin) {{(pid=92057) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} May 23 12:40:31.536958 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5af1e1e-b16d-4f0c-8687-4615b8ac25d8 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:40:31.538413 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5af1e1e-b16d-4f0c-8687-4615b8ac25d8 None admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:40:31.541050 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=100 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:40:31.548274 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 70/141] 10.222.0.30 () {56 vars in 1099 bytes} [Mon May 23 12:40:31 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 209 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) May 23 12:40:31.554026 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44e35b0e-e91e-4903-a7d4-edeb71c7dce0 None admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:40:31.686330 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=100 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:40:31.724715 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44e35b0e-e91e-4903-a7d4-edeb71c7dce0 None admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:40:31.781207 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8287520b-ebd6-4e0e-a815-54152486e917 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:40:31.782192 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8287520b-ebd6-4e0e-a815-54152486e917 None admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:40:31.782477 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8287520b-ebd6-4e0e-a815-54152486e917 None admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:40:31.782757 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8287520b-ebd6-4e0e-a815-54152486e917 None admin] PATH_INFO: `/v3/users/0f7119f8fa5c4445977671140ded91a4` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:40:31.786667 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8287520b-ebd6-4e0e-a815-54152486e917 None admin] RBAC: Authorizing `identity:get_user(user_id=0f7119f8fa5c4445977671140ded91a4)` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:40:31.788103 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8287520b-ebd6-4e0e-a815-54152486e917 None admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:40:31.789948 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 72/142] 10.222.0.30 () {56 vars in 1177 bytes} [Mon May 23 12:40:31 2022] GET /identity/v3/users/0f7119f8fa5c4445977671140ded91a4 => generated 247 bytes in 236 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) May 23 12:40:31.798998 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5af1e1e-b16d-4f0c-8687-4615b8ac25d8 None admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:40:31.870275 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b296b2a6-08c9-4af1-8b36-35832264197a None None] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:40:31.871387 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b296b2a6-08c9-4af1-8b36-35832264197a None None] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:40:31.871674 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b296b2a6-08c9-4af1-8b36-35832264197a None None] PATH_INFO: `/` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:40:31.873051 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 73/143] 10.222.0.30 () {54 vars in 926 bytes} [Mon May 23 12:40:31 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) May 23 12:40:31.878688 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-969c1548-5a79-4311-b9f8-b62b763271f7 None None] REQUEST_METHOD: `POST` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:40:31.879266 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-969c1548-5a79-4311-b9f8-b62b763271f7 None None] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:40:31.879758 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-969c1548-5a79-4311-b9f8-b62b763271f7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:40:31.919031 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.auth.core [None req-969c1548-5a79-4311-b9f8-b62b763271f7 None None] MFA Rules not processed for user `0f7119f8fa5c4445977671140ded91a4`. Rule list: `[]` (Enabled: `True`). {{(pid=92056) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 12:40:31.920374 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.fernet_utils [None req-969c1548-5a79-4311-b9f8-b62b763271f7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92056) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 12:40:31.978834 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5af1e1e-b16d-4f0c-8687-4615b8ac25d8 None admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:40:32.050062 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 74/144] 10.222.0.30 () {58 vars in 1023 bytes} [Mon May 23 12:40:31 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 172 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) May 23 12:40:32.082601 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43087d75-204e-41fb-bc25-30781a9c916e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:40:32.083271 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43087d75-204e-41fb-bc25-30781a9c916e None admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:40:32.083559 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43087d75-204e-41fb-bc25-30781a9c916e None admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:40:32.083843 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43087d75-204e-41fb-bc25-30781a9c916e None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:40:32.094368 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-969c1548-5a79-4311-b9f8-b62b763271f7 None None] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:40:32.206587 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=100 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:40:32.254157 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=100 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:40:32.266951 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43087d75-204e-41fb-bc25-30781a9c916e None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:40:32.267563 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-969c1548-5a79-4311-b9f8-b62b763271f7 None None] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:40:32.268441 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43087d75-204e-41fb-bc25-30781a9c916e None admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:40:32.330849 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b47e1827-e528-4286-9810-2ba01929d9c2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:40:32.331609 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b47e1827-e528-4286-9810-2ba01929d9c2 None admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:40:32.331897 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b47e1827-e528-4286-9810-2ba01929d9c2 None admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:40:32.332186 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b47e1827-e528-4286-9810-2ba01929d9c2 None admin] PATH_INFO: `/v3/domains/default` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:40:32.333553 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b47e1827-e528-4286-9810-2ba01929d9c2 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:40:32.335089 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b47e1827-e528-4286-9810-2ba01929d9c2 None admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:40:32.337043 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 75/145] 10.222.0.30 () {56 vars in 1108 bytes} [Mon May 23 12:40:32 2022] GET /identity/v3/domains/default => generated 199 bytes in 243 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) May 23 12:40:32.342432 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a3cc17d-b56d-4369-8555-d6b38d1eea57 None None] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:40:32.342432 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a3cc17d-b56d-4369-8555-d6b38d1eea57 None None] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:40:32.342432 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a3cc17d-b56d-4369-8555-d6b38d1eea57 None None] PATH_INFO: `/` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:40:32.342432 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 76/146] 10.222.0.30 () {54 vars in 926 bytes} [Mon May 23 12:40:32 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) May 23 12:40:32.346847 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a3cc17d-b56d-4369-8555-d6b38d1eea57 None None] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:40:32.468191 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 71/147] 10.222.0.30 () {58 vars in 1282 bytes} [Mon May 23 12:40:31 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 669 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) May 23 12:40:32.469309 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5586528a-4dc8-4f85-a3e3-21ea73f409fd None None] REQUEST_METHOD: `POST` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:40:32.469607 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5586528a-4dc8-4f85-a3e3-21ea73f409fd None None] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:40:32.469902 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5586528a-4dc8-4f85-a3e3-21ea73f409fd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:40:32.487810 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.auth.core [None req-5586528a-4dc8-4f85-a3e3-21ea73f409fd None None] MFA Rules not processed for user `0f7119f8fa5c4445977671140ded91a4`. Rule list: `[]` (Enabled: `True`). {{(pid=92057) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 12:40:32.488994 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.fernet_utils [None req-5586528a-4dc8-4f85-a3e3-21ea73f409fd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92057) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 12:40:32.506565 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a3cc17d-b56d-4369-8555-d6b38d1eea57 None None] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:40:32.564355 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 72/148] 10.222.0.30 () {58 vars in 1023 bytes} [Mon May 23 12:40:32 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 96 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) May 23 12:40:32.565020 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5586528a-4dc8-4f85-a3e3-21ea73f409fd None None] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:40:32.595439 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69d4549d-822f-4e8b-912a-094dfdf87b86 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:40:32.596044 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69d4549d-822f-4e8b-912a-094dfdf87b86 None admin] REQUEST_METHOD: `POST` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:40:32.596326 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69d4549d-822f-4e8b-912a-094dfdf87b86 None admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:40:32.596698 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69d4549d-822f-4e8b-912a-094dfdf87b86 None admin] PATH_INFO: `/v3/users` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:40:32.597644 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69d4549d-822f-4e8b-912a-094dfdf87b86 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:40:32.599720 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69d4549d-822f-4e8b-912a-094dfdf87b86 None admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:40:32.609299 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=100 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:40:32.622561 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=100 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:40:32.703267 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5586528a-4dc8-4f85-a3e3-21ea73f409fd None None] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:40:32.737685 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5094ec76-0976-4c56-bd40-dd97d8388372 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:40:32.738572 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5094ec76-0976-4c56-bd40-dd97d8388372 None admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:40:32.738950 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5094ec76-0976-4c56-bd40-dd97d8388372 None admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:40:32.742485 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5094ec76-0976-4c56-bd40-dd97d8388372 None admin] PATH_INFO: `/v3/projects/486401ae2e5f40d5a5f060242896ae53` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:40:32.744069 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5094ec76-0976-4c56-bd40-dd97d8388372 None admin] RBAC: Authorizing `identity:get_project(project_id=486401ae2e5f40d5a5f060242896ae53)` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:40:32.746528 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5094ec76-0976-4c56-bd40-dd97d8388372 None admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:40:32.748389 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 73/149] 10.222.0.30 () {56 vars in 1186 bytes} [Mon May 23 12:40:32 2022] GET /identity/v3/projects/486401ae2e5f40d5a5f060242896ae53 => generated 298 bytes in 184 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) May 23 12:40:32.749424 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3605e2a6-964a-4cfc-b5ed-2a370a7c3144 None None] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:40:32.749714 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3605e2a6-964a-4cfc-b5ed-2a370a7c3144 None None] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:40:32.750024 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3605e2a6-964a-4cfc-b5ed-2a370a7c3144 None None] PATH_INFO: `/` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:40:32.751263 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 74/150] 10.222.0.30 () {54 vars in 926 bytes} [Mon May 23 12:40:32 2022] GET /identity => generated 270 bytes in 2 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) May 23 12:40:32.751575 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 77/151] 10.222.0.30 () {60 vars in 1132 bytes} [Mon May 23 12:40:32 2022] POST /identity/v3/users => generated 283 bytes in 405 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) May 23 12:40:32.756728 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3605e2a6-964a-4cfc-b5ed-2a370a7c3144 None None] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:40:32.758118 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69d4549d-822f-4e8b-912a-094dfdf87b86 None admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:40:32.884353 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=100 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:40:32.894127 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3605e2a6-964a-4cfc-b5ed-2a370a7c3144 None None] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:40:32.943897 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1dc32365-2230-4e57-999f-04f5aaa1ac13 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:40:32.944712 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1dc32365-2230-4e57-999f-04f5aaa1ac13 None admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:40:32.944995 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1dc32365-2230-4e57-999f-04f5aaa1ac13 None admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:40:32.945270 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1dc32365-2230-4e57-999f-04f5aaa1ac13 None admin] PATH_INFO: `/v3/roles/member` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:40:32.977114 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1dc32365-2230-4e57-999f-04f5aaa1ac13 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} May 23 12:40:32.989247 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69d4549d-822f-4e8b-912a-094dfdf87b86 None admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:40:32.991439 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application [None req-1dc32365-2230-4e57-999f-04f5aaa1ac13 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. May 23 12:40:32.991439 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application Traceback (most recent call last): May 23 12:40:32.991439 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request May 23 12:40:32.991439 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application rv = self.dispatch_request() May 23 12:40:32.991439 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request May 23 12:40:32.991439 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) May 23 12:40:32.991439 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper May 23 12:40:32.991439 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) May 23 12:40:32.991439 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view May 23 12:40:32.991439 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) May 23 12:40:32.991439 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request May 23 12:40:32.991439 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) May 23 12:40:32.991439 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get May 23 12:40:32.991439 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self._get_role(role_id) May 23 12:40:32.991439 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role May 23 12:40:32.991439 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') May 23 12:40:32.992632 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call May 23 12:40:32.992632 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( May 23 12:40:32.992632 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data May 23 12:40:32.992632 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) May 23 12:40:32.992632 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped May 23 12:40:32.992632 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) May 23 12:40:32.992632 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun May 23 12:40:32.992632 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) May 23 12:40:32.992632 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func May 23 12:40:32.992632 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self.get_or_create( May 23 12:40:32.992632 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create May 23 12:40:32.992632 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application with Lock( May 23 12:40:32.992632 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ May 23 12:40:32.992632 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self._enter() May 23 12:40:32.992632 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter May 23 12:40:32.992632 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) May 23 12:40:32.993451 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create May 23 12:40:32.993451 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self.creator() May 23 12:40:32.993451 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value May 23 12:40:32.993451 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application created_value = creator( May 23 12:40:32.993451 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role May 23 12:40:32.993451 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) May 23 12:40:32.993451 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role May 23 12:40:32.993451 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() May 23 12:40:32.993451 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role May 23 12:40:32.993451 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) May 23 12:40:32.993451 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. May 23 12:40:32.993451 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application May 23 12:40:32.993451 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 75/152] 10.222.0.30 () {56 vars in 1099 bytes} [Mon May 23 12:40:32 2022] GET /identity/v3/roles/member => generated 84 bytes in 241 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) May 23 12:40:32.994139 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-894f83ac-4b59-41b6-821d-7b201aa1c083 None None] REQUEST_METHOD: `POST` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:40:32.994204 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-894f83ac-4b59-41b6-821d-7b201aa1c083 None None] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:40:32.994752 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-894f83ac-4b59-41b6-821d-7b201aa1c083 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:40:33.033605 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.auth.core [None req-894f83ac-4b59-41b6-821d-7b201aa1c083 None None] MFA Rules not processed for user `0f7119f8fa5c4445977671140ded91a4`. Rule list: `[]` (Enabled: `True`). {{(pid=92057) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 12:40:33.034844 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.fernet_utils [None req-894f83ac-4b59-41b6-821d-7b201aa1c083 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92057) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 12:40:33.037716 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d535f49-b1b3-47de-96a4-5a1fb3d07c90 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:40:33.039424 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d535f49-b1b3-47de-96a4-5a1fb3d07c90 None admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:40:33.039713 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d535f49-b1b3-47de-96a4-5a1fb3d07c90 None admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:40:33.039995 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d535f49-b1b3-47de-96a4-5a1fb3d07c90 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:40:33.040904 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d535f49-b1b3-47de-96a4-5a1fb3d07c90 None admin] RBAC: Adding query filter params (role.id=bc3c3c9d654940cbaa5177a6be23d1bc, scope.project.id=486401ae2e5f40d5a5f060242896ae53, user.id=0f7119f8fa5c4445977671140ded91a4) {{(pid=92056) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} May 23 12:40:33.041230 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d535f49-b1b3-47de-96a4-5a1fb3d07c90 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:40:33.043488 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d535f49-b1b3-47de-96a4-5a1fb3d07c90 None admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:40:33.053247 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 78/153] 10.222.0.30 () {56 vars in 1374 bytes} [Mon May 23 12:40:32 2022] GET /identity/v3/role_assignments?user.id=0f7119f8fa5c4445977671140ded91a4&scope.project.id=486401ae2e5f40d5a5f060242896ae53&role.id=bc3c3c9d654940cbaa5177a6be23d1bc => generated 608 bytes in 295 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) May 23 12:40:33.054332 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d535f49-b1b3-47de-96a4-5a1fb3d07c90 None admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:40:33.134564 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:INSERT=3,keystone:SELECT=97 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:40:33.149017 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 76/154] 10.222.0.30 () {58 vars in 1023 bytes} [Mon May 23 12:40:32 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 156 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) May 23 12:40:33.155834 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18d14bf4-0e35-4372-be74-97f812122df1 None None] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:40:33.156139 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18d14bf4-0e35-4372-be74-97f812122df1 None None] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:40:33.156403 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18d14bf4-0e35-4372-be74-97f812122df1 None None] PATH_INFO: `/` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:40:33.157835 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 77/155] 10.222.0.30 () {54 vars in 926 bytes} [Mon May 23 12:40:33 2022] GET /identity => generated 270 bytes in 8 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) May 23 12:40:33.168389 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0776dbbb-28a1-435f-8028-1f8498ada77e None None] REQUEST_METHOD: `POST` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:40:33.168690 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0776dbbb-28a1-435f-8028-1f8498ada77e None None] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:40:33.168958 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0776dbbb-28a1-435f-8028-1f8498ada77e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:40:33.193660 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d535f49-b1b3-47de-96a4-5a1fb3d07c90 None admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:40:33.197402 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.auth.core [None req-0776dbbb-28a1-435f-8028-1f8498ada77e None None] MFA Rules not processed for user `0f7119f8fa5c4445977671140ded91a4`. Rule list: `[]` (Enabled: `True`). {{(pid=92057) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 12:40:33.198689 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.fernet_utils [None req-0776dbbb-28a1-435f-8028-1f8498ada77e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92057) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 12:40:33.233732 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8544ed55-280e-43e0-88d4-82ef8e3efa89 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:40:33.234804 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8544ed55-280e-43e0-88d4-82ef8e3efa89 None admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:40:33.235324 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8544ed55-280e-43e0-88d4-82ef8e3efa89 None admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:40:33.235850 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8544ed55-280e-43e0-88d4-82ef8e3efa89 None admin] PATH_INFO: `/v3/roles` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:40:33.236819 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8544ed55-280e-43e0-88d4-82ef8e3efa89 None admin] RBAC: Adding query filter params (name=member) {{(pid=92056) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} May 23 12:40:33.237347 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8544ed55-280e-43e0-88d4-82ef8e3efa89 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:40:33.239018 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8544ed55-280e-43e0-88d4-82ef8e3efa89 None admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:40:33.249120 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 79/156] 10.222.0.30 () {56 vars in 1101 bytes} [Mon May 23 12:40:33 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 196 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) May 23 12:40:33.250327 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8544ed55-280e-43e0-88d4-82ef8e3efa89 None admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:40:33.284451 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 78/157] 10.222.0.30 () {58 vars in 1023 bytes} [Mon May 23 12:40:33 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 118 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) May 23 12:40:33.285313 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0776dbbb-28a1-435f-8028-1f8498ada77e None None] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:40:33.305322 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=100 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:40:33.387034 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8544ed55-280e-43e0-88d4-82ef8e3efa89 None admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:40:33.411247 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=100 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:40:33.429696 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0776dbbb-28a1-435f-8028-1f8498ada77e None None] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:40:33.434020 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d86d8c7-1b9c-4cda-9f00-d5d86dcb02d1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:40:33.434898 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d86d8c7-1b9c-4cda-9f00-d5d86dcb02d1 None admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:40:33.435186 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d86d8c7-1b9c-4cda-9f00-d5d86dcb02d1 None admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:40:33.435465 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d86d8c7-1b9c-4cda-9f00-d5d86dcb02d1 None admin] PATH_INFO: `/v3/roles/reader` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:40:33.451123 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d86d8c7-1b9c-4cda-9f00-d5d86dcb02d1 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} May 23 12:40:33.461891 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application [None req-3d86d8c7-1b9c-4cda-9f00-d5d86dcb02d1 None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. May 23 12:40:33.461891 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application Traceback (most recent call last): May 23 12:40:33.461891 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request May 23 12:40:33.461891 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application rv = self.dispatch_request() May 23 12:40:33.461891 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request May 23 12:40:33.461891 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) May 23 12:40:33.461891 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper May 23 12:40:33.461891 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) May 23 12:40:33.461891 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view May 23 12:40:33.461891 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) May 23 12:40:33.461891 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request May 23 12:40:33.461891 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) May 23 12:40:33.461891 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get May 23 12:40:33.461891 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self._get_role(role_id) May 23 12:40:33.461891 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role May 23 12:40:33.461891 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') May 23 12:40:33.462863 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call May 23 12:40:33.462863 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( May 23 12:40:33.462863 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data May 23 12:40:33.462863 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) May 23 12:40:33.462863 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped May 23 12:40:33.462863 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) May 23 12:40:33.462863 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun May 23 12:40:33.462863 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) May 23 12:40:33.462863 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func May 23 12:40:33.462863 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self.get_or_create( May 23 12:40:33.462863 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create May 23 12:40:33.462863 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application with Lock( May 23 12:40:33.462863 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ May 23 12:40:33.462863 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self._enter() May 23 12:40:33.462863 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter May 23 12:40:33.462863 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) May 23 12:40:33.463668 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create May 23 12:40:33.463668 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self.creator() May 23 12:40:33.463668 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value May 23 12:40:33.463668 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application created_value = creator( May 23 12:40:33.463668 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role May 23 12:40:33.463668 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) May 23 12:40:33.463668 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role May 23 12:40:33.463668 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() May 23 12:40:33.463668 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role May 23 12:40:33.463668 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) May 23 12:40:33.463668 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. May 23 12:40:33.463668 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application May 23 12:40:33.465380 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 80/158] 10.222.0.30 () {56 vars in 1099 bytes} [Mon May 23 12:40:33 2022] GET /identity/v3/roles/reader => generated 84 bytes in 214 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) May 23 12:40:33.465380 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d86d8c7-1b9c-4cda-9f00-d5d86dcb02d1 None admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:40:33.482003 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-641c2794-b52a-43ac-8a27-a328ff4eabd1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:40:33.482947 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-641c2794-b52a-43ac-8a27-a328ff4eabd1 None admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:40:33.483234 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-641c2794-b52a-43ac-8a27-a328ff4eabd1 None admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:40:33.483559 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-641c2794-b52a-43ac-8a27-a328ff4eabd1 None admin] PATH_INFO: `/v3/users/b3478ea957234bb99fde27e6c3d0dfd3` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:40:33.484820 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-641c2794-b52a-43ac-8a27-a328ff4eabd1 None admin] RBAC: Authorizing `identity:get_user(user_id=b3478ea957234bb99fde27e6c3d0dfd3)` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:40:33.486362 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-641c2794-b52a-43ac-8a27-a328ff4eabd1 None admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:40:33.488338 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 79/159] 10.222.0.30 () {56 vars in 1177 bytes} [Mon May 23 12:40:33 2022] GET /identity/v3/users/b3478ea957234bb99fde27e6c3d0dfd3 => generated 275 bytes in 203 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) May 23 12:40:33.489012 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-641c2794-b52a-43ac-8a27-a328ff4eabd1 None admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:40:33.600503 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d86d8c7-1b9c-4cda-9f00-d5d86dcb02d1 None admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:40:33.613745 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=100 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:40:33.639130 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-641c2794-b52a-43ac-8a27-a328ff4eabd1 None admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:40:33.650633 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34c4cad1-78ae-4b75-866f-172c833c2726 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:40:33.652491 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34c4cad1-78ae-4b75-866f-172c833c2726 None admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:40:33.653090 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34c4cad1-78ae-4b75-866f-172c833c2726 None admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:40:33.654390 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34c4cad1-78ae-4b75-866f-172c833c2726 None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:40:33.666492 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34c4cad1-78ae-4b75-866f-172c833c2726 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} May 23 12:40:33.674997 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application [None req-34c4cad1-78ae-4b75-866f-172c833c2726 None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. May 23 12:40:33.674997 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application Traceback (most recent call last): May 23 12:40:33.674997 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request May 23 12:40:33.674997 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application rv = self.dispatch_request() May 23 12:40:33.674997 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request May 23 12:40:33.674997 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) May 23 12:40:33.674997 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper May 23 12:40:33.674997 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) May 23 12:40:33.674997 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view May 23 12:40:33.674997 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) May 23 12:40:33.674997 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request May 23 12:40:33.674997 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) May 23 12:40:33.674997 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get May 23 12:40:33.674997 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self._get_role(role_id) May 23 12:40:33.674997 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role May 23 12:40:33.674997 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') May 23 12:40:33.675926 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call May 23 12:40:33.675926 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( May 23 12:40:33.675926 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data May 23 12:40:33.675926 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) May 23 12:40:33.675926 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped May 23 12:40:33.675926 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) May 23 12:40:33.675926 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun May 23 12:40:33.675926 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) May 23 12:40:33.675926 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func May 23 12:40:33.675926 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self.get_or_create( May 23 12:40:33.675926 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create May 23 12:40:33.675926 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application with Lock( May 23 12:40:33.675926 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ May 23 12:40:33.675926 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self._enter() May 23 12:40:33.675926 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter May 23 12:40:33.675926 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) May 23 12:40:33.676753 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create May 23 12:40:33.676753 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self.creator() May 23 12:40:33.676753 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value May 23 12:40:33.676753 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application created_value = creator( May 23 12:40:33.676753 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role May 23 12:40:33.676753 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) May 23 12:40:33.676753 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role May 23 12:40:33.676753 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() May 23 12:40:33.676753 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role May 23 12:40:33.676753 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) May 23 12:40:33.676753 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. May 23 12:40:33.676753 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application May 23 12:40:33.676753 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 81/160] 10.222.0.30 () {56 vars in 1114 bytes} [Mon May 23 12:40:33 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 212 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) May 23 12:40:33.677414 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34c4cad1-78ae-4b75-866f-172c833c2726 None admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:40:33.695888 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-abc62e21-1714-4657-867f-eeaf21482ae6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:40:33.696737 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abc62e21-1714-4657-867f-eeaf21482ae6 None admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:40:33.697083 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abc62e21-1714-4657-867f-eeaf21482ae6 None admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:40:33.697422 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abc62e21-1714-4657-867f-eeaf21482ae6 None admin] PATH_INFO: `/v3/roles` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:40:33.698288 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-abc62e21-1714-4657-867f-eeaf21482ae6 None admin] RBAC: Adding query filter params (name=reader) {{(pid=92057) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} May 23 12:40:33.699253 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-abc62e21-1714-4657-867f-eeaf21482ae6 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:40:33.701021 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-abc62e21-1714-4657-867f-eeaf21482ae6 None admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:40:33.711073 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 80/161] 10.222.0.30 () {56 vars in 1101 bytes} [Mon May 23 12:40:33 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 222 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) May 23 12:40:33.712680 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e852fc42-0d72-4716-af67-f9f6553be1ff None None] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:40:33.713106 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e852fc42-0d72-4716-af67-f9f6553be1ff None None] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:40:33.713509 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e852fc42-0d72-4716-af67-f9f6553be1ff None None] PATH_INFO: `/` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:40:33.716111 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 81/162] 10.222.0.30 () {54 vars in 926 bytes} [Mon May 23 12:40:33 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) May 23 12:40:33.717011 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e852fc42-0d72-4716-af67-f9f6553be1ff None None] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:40:33.728514 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=100 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:40:33.822947 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34c4cad1-78ae-4b75-866f-172c833c2726 None admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:40:33.859437 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e852fc42-0d72-4716-af67-f9f6553be1ff None None] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:40:33.897222 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f1f2731-be7f-4317-bd93-542f77ef030a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:40:33.897756 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d02020bc-66dc-49f5-ad7f-9f756ebb7754 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:40:33.897922 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f1f2731-be7f-4317-bd93-542f77ef030a None admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:40:33.898096 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f1f2731-be7f-4317-bd93-542f77ef030a None admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:40:33.898380 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f1f2731-be7f-4317-bd93-542f77ef030a None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:40:33.899226 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d02020bc-66dc-49f5-ad7f-9f756ebb7754 None admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:40:33.899743 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d02020bc-66dc-49f5-ad7f-9f756ebb7754 None admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:40:33.900232 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d02020bc-66dc-49f5-ad7f-9f756ebb7754 None admin] PATH_INFO: `/v3/roles` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:40:33.901311 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d02020bc-66dc-49f5-ad7f-9f756ebb7754 None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=92057) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} May 23 12:40:33.902539 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d02020bc-66dc-49f5-ad7f-9f756ebb7754 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:40:33.904307 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d02020bc-66dc-49f5-ad7f-9f756ebb7754 None admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:40:33.914727 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 82/163] 10.222.0.30 () {56 vars in 1111 bytes} [Mon May 23 12:40:33 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 198 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) May 23 12:40:33.916369 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-696ea976-74f0-4478-812f-cc3bf241ed1d None None] REQUEST_METHOD: `POST` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:40:33.917187 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-696ea976-74f0-4478-812f-cc3bf241ed1d None None] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:40:33.917937 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-696ea976-74f0-4478-812f-cc3bf241ed1d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:40:33.951987 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.auth.core [None req-696ea976-74f0-4478-812f-cc3bf241ed1d None None] MFA Rules not processed for user `0f7119f8fa5c4445977671140ded91a4`. Rule list: `[]` (Enabled: `True`). {{(pid=92057) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 12:40:33.953516 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.fernet_utils [None req-696ea976-74f0-4478-812f-cc3bf241ed1d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92057) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 12:40:33.960337 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=100 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:40:33.983772 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f1f2731-be7f-4317-bd93-542f77ef030a None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:40:33.990191 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f1f2731-be7f-4317-bd93-542f77ef030a None admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:40:34.026253 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=100 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:40:34.090449 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 82/164] 10.222.0.30 () {58 vars in 1282 bytes} [Mon May 23 12:40:33 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 414 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) May 23 12:40:34.091428 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f1f2731-be7f-4317-bd93-542f77ef030a None admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:40:34.146668 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 83/165] 10.222.0.30 () {58 vars in 1023 bytes} [Mon May 23 12:40:33 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 231 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) May 23 12:40:34.147430 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-696ea976-74f0-4478-812f-cc3bf241ed1d None None] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:40:34.223699 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f1f2731-be7f-4317-bd93-542f77ef030a None admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:40:34.288743 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a800c2f-be11-44bd-88e1-6c5546bc9cfc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:40:34.289623 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a800c2f-be11-44bd-88e1-6c5546bc9cfc None admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:40:34.289912 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a800c2f-be11-44bd-88e1-6c5546bc9cfc None admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:40:34.290188 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a800c2f-be11-44bd-88e1-6c5546bc9cfc None admin] PATH_INFO: `/v3/users/5c3e3c32e6c8497c8e3d98e09b8de7be` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:40:34.293075 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a800c2f-be11-44bd-88e1-6c5546bc9cfc None admin] RBAC: Authorizing `identity:get_user(user_id=5c3e3c32e6c8497c8e3d98e09b8de7be)` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:40:34.294779 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a800c2f-be11-44bd-88e1-6c5546bc9cfc None admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:40:34.298922 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 83/166] 10.222.0.30 () {56 vars in 1177 bytes} [Mon May 23 12:40:34 2022] GET /identity/v3/users/5c3e3c32e6c8497c8e3d98e09b8de7be => generated 289 bytes in 207 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) May 23 12:40:34.299702 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a800c2f-be11-44bd-88e1-6c5546bc9cfc None admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:40:34.325602 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-696ea976-74f0-4478-812f-cc3bf241ed1d None None] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:40:34.326239 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=100 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:40:34.361968 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48235924-f6d8-44e1-8afb-7c3441c1cf3c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:40:34.369613 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48235924-f6d8-44e1-8afb-7c3441c1cf3c None admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:40:34.369613 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48235924-f6d8-44e1-8afb-7c3441c1cf3c None admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:40:34.369613 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48235924-f6d8-44e1-8afb-7c3441c1cf3c None admin] PATH_INFO: `/v3/users/b3478ea957234bb99fde27e6c3d0dfd3` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:40:34.369613 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48235924-f6d8-44e1-8afb-7c3441c1cf3c None admin] RBAC: Authorizing `identity:get_user(user_id=b3478ea957234bb99fde27e6c3d0dfd3)` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:40:34.369613 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48235924-f6d8-44e1-8afb-7c3441c1cf3c None admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:40:34.369613 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 84/167] 10.222.0.30 () {56 vars in 1177 bytes} [Mon May 23 12:40:34 2022] GET /identity/v3/users/b3478ea957234bb99fde27e6c3d0dfd3 => generated 275 bytes in 221 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) May 23 12:40:34.370133 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48235924-f6d8-44e1-8afb-7c3441c1cf3c None admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:40:34.378223 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=100 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:40:34.459793 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a800c2f-be11-44bd-88e1-6c5546bc9cfc None admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:40:34.498973 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bdae7be5-e339-4a66-9e68-862793f651ea None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:40:34.499846 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdae7be5-e339-4a66-9e68-862793f651ea None admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:40:34.500126 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdae7be5-e339-4a66-9e68-862793f651ea None admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:40:34.500401 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdae7be5-e339-4a66-9e68-862793f651ea None admin] PATH_INFO: `/v3/projects/486401ae2e5f40d5a5f060242896ae53` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:40:34.501612 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bdae7be5-e339-4a66-9e68-862793f651ea None admin] RBAC: Authorizing `identity:get_project(project_id=486401ae2e5f40d5a5f060242896ae53)` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:40:34.503505 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bdae7be5-e339-4a66-9e68-862793f651ea None admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:40:34.505998 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 84/168] 10.222.0.30 () {56 vars in 1186 bytes} [Mon May 23 12:40:34 2022] GET /identity/v3/projects/486401ae2e5f40d5a5f060242896ae53 => generated 298 bytes in 206 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) May 23 12:40:34.506778 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bdae7be5-e339-4a66-9e68-862793f651ea None admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:40:34.548704 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48235924-f6d8-44e1-8afb-7c3441c1cf3c None admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:40:34.594249 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=100 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:40:34.630870 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2221f1c-fe71-4bc9-8bbd-9cce8b3369f1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:40:34.632365 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2221f1c-fe71-4bc9-8bbd-9cce8b3369f1 None admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:40:34.632654 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2221f1c-fe71-4bc9-8bbd-9cce8b3369f1 None admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:40:34.632933 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2221f1c-fe71-4bc9-8bbd-9cce8b3369f1 None admin] PATH_INFO: `/v3/roles/member` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:40:34.635217 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bdae7be5-e339-4a66-9e68-862793f651ea None admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:40:34.655124 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f2221f1c-fe71-4bc9-8bbd-9cce8b3369f1 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} May 23 12:40:34.662950 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application [None req-f2221f1c-fe71-4bc9-8bbd-9cce8b3369f1 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. May 23 12:40:34.662950 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application Traceback (most recent call last): May 23 12:40:34.662950 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request May 23 12:40:34.662950 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application rv = self.dispatch_request() May 23 12:40:34.662950 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request May 23 12:40:34.662950 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) May 23 12:40:34.662950 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper May 23 12:40:34.662950 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) May 23 12:40:34.662950 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view May 23 12:40:34.662950 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) May 23 12:40:34.662950 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request May 23 12:40:34.662950 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) May 23 12:40:34.662950 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get May 23 12:40:34.662950 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self._get_role(role_id) May 23 12:40:34.662950 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role May 23 12:40:34.662950 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') May 23 12:40:34.663843 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call May 23 12:40:34.663843 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( May 23 12:40:34.663843 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data May 23 12:40:34.663843 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) May 23 12:40:34.663843 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped May 23 12:40:34.663843 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) May 23 12:40:34.663843 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun May 23 12:40:34.663843 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) May 23 12:40:34.663843 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func May 23 12:40:34.663843 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self.get_or_create( May 23 12:40:34.663843 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create May 23 12:40:34.663843 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application with Lock( May 23 12:40:34.663843 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ May 23 12:40:34.663843 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self._enter() May 23 12:40:34.663843 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter May 23 12:40:34.663843 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) May 23 12:40:34.664655 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create May 23 12:40:34.664655 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self.creator() May 23 12:40:34.664655 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value May 23 12:40:34.664655 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application created_value = creator( May 23 12:40:34.664655 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role May 23 12:40:34.664655 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) May 23 12:40:34.664655 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role May 23 12:40:34.664655 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() May 23 12:40:34.664655 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role May 23 12:40:34.664655 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) May 23 12:40:34.664655 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. May 23 12:40:34.664655 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application May 23 12:40:34.664655 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 85/169] 10.222.0.30 () {56 vars in 1099 bytes} [Mon May 23 12:40:34 2022] GET /identity/v3/roles/member => generated 84 bytes in 296 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) May 23 12:40:34.665327 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2221f1c-fe71-4bc9-8bbd-9cce8b3369f1 None admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:40:34.721394 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14519853-cb37-480c-8982-5b54c8ea1a04 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:40:34.722219 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14519853-cb37-480c-8982-5b54c8ea1a04 None admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:40:34.722499 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14519853-cb37-480c-8982-5b54c8ea1a04 None admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:40:34.723088 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14519853-cb37-480c-8982-5b54c8ea1a04 None admin] PATH_INFO: `/v3/projects/486401ae2e5f40d5a5f060242896ae53` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:40:34.725525 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14519853-cb37-480c-8982-5b54c8ea1a04 None admin] RBAC: Authorizing `identity:get_project(project_id=486401ae2e5f40d5a5f060242896ae53)` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:40:34.727072 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14519853-cb37-480c-8982-5b54c8ea1a04 None admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:40:34.734287 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 85/170] 10.222.0.30 () {56 vars in 1186 bytes} [Mon May 23 12:40:34 2022] GET /identity/v3/projects/486401ae2e5f40d5a5f060242896ae53 => generated 298 bytes in 228 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) May 23 12:40:34.735195 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14519853-cb37-480c-8982-5b54c8ea1a04 None admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:40:34.794426 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=100 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:40:34.856166 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2221f1c-fe71-4bc9-8bbd-9cce8b3369f1 None admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:40:34.903419 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14519853-cb37-480c-8982-5b54c8ea1a04 None admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:40:34.935943 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a683ada-7e5c-48ac-885c-8bd872930266 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:40:34.936721 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a683ada-7e5c-48ac-885c-8bd872930266 None admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:40:34.937010 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a683ada-7e5c-48ac-885c-8bd872930266 None admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:40:34.937289 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a683ada-7e5c-48ac-885c-8bd872930266 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:40:34.949959 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca056015-e2af-4806-ac12-5a4fa1844b85 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:40:34.950943 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca056015-e2af-4806-ac12-5a4fa1844b85 None admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:40:34.951428 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca056015-e2af-4806-ac12-5a4fa1844b85 None admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:40:34.951965 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca056015-e2af-4806-ac12-5a4fa1844b85 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:40:34.953017 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca056015-e2af-4806-ac12-5a4fa1844b85 None admin] RBAC: Adding query filter params (role.id=80fb22507c9a4ff09d44d8f12f6fff1a, scope.project.id=486401ae2e5f40d5a5f060242896ae53, user.id=b3478ea957234bb99fde27e6c3d0dfd3) {{(pid=92056) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} May 23 12:40:34.953335 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca056015-e2af-4806-ac12-5a4fa1844b85 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:40:34.954873 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca056015-e2af-4806-ac12-5a4fa1844b85 None admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:40:34.961547 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 86/171] 10.222.0.30 () {56 vars in 1374 bytes} [Mon May 23 12:40:34 2022] GET /identity/v3/role_assignments?user.id=b3478ea957234bb99fde27e6c3d0dfd3&scope.project.id=486401ae2e5f40d5a5f060242896ae53&role.id=80fb22507c9a4ff09d44d8f12f6fff1a => generated 608 bytes in 227 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) May 23 12:40:34.962363 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca056015-e2af-4806-ac12-5a4fa1844b85 None admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:40:34.978398 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=100 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:40:35.085459 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a683ada-7e5c-48ac-885c-8bd872930266 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:40:35.086983 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a683ada-7e5c-48ac-885c-8bd872930266 None admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:40:35.132062 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca056015-e2af-4806-ac12-5a4fa1844b85 None admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:40:35.169403 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5aa7ca4-d2e3-43d9-bae6-feea6df75c62 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:40:35.170134 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5aa7ca4-d2e3-43d9-bae6-feea6df75c62 None admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:40:35.170423 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5aa7ca4-d2e3-43d9-bae6-feea6df75c62 None admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:40:35.170701 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5aa7ca4-d2e3-43d9-bae6-feea6df75c62 None admin] PATH_INFO: `/v3/roles` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:40:35.171476 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5aa7ca4-d2e3-43d9-bae6-feea6df75c62 None admin] RBAC: Adding query filter params (name=member) {{(pid=92056) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} May 23 12:40:35.171785 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5aa7ca4-d2e3-43d9-bae6-feea6df75c62 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:40:35.173206 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5aa7ca4-d2e3-43d9-bae6-feea6df75c62 None admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:40:35.182345 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 87/172] 10.222.0.30 () {56 vars in 1101 bytes} [Mon May 23 12:40:34 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 220 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) May 23 12:40:35.182985 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5aa7ca4-d2e3-43d9-bae6-feea6df75c62 None admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:40:35.279261 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=100 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:40:35.309100 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=100 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:40:35.318551 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 86/173] 10.222.0.30 () {58 vars in 1282 bytes} [Mon May 23 12:40:34 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 654 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) May 23 12:40:35.319461 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a683ada-7e5c-48ac-885c-8bd872930266 None admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:40:35.395950 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5aa7ca4-d2e3-43d9-bae6-feea6df75c62 None admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:40:35.441155 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a683ada-7e5c-48ac-885c-8bd872930266 None admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:40:35.478036 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43976495-904e-43e8-8a8a-0ec09cae5d5a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:40:35.478928 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43976495-904e-43e8-8a8a-0ec09cae5d5a None admin] REQUEST_METHOD: `PUT` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:40:35.479208 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43976495-904e-43e8-8a8a-0ec09cae5d5a None admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:40:35.479527 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43976495-904e-43e8-8a8a-0ec09cae5d5a None admin] PATH_INFO: `/v3/projects/486401ae2e5f40d5a5f060242896ae53/users/5c3e3c32e6c8497c8e3d98e09b8de7be/roles/c223286ed21a4654a0f2b4e5551831c7` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:40:35.482903 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43976495-904e-43e8-8a8a-0ec09cae5d5a None admin] RBAC: Authorizing `identity:create_grant(project_id=486401ae2e5f40d5a5f060242896ae53, user_id=5c3e3c32e6c8497c8e3d98e09b8de7be, role_id=c223286ed21a4654a0f2b4e5551831c7)` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:40:35.484560 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43976495-904e-43e8-8a8a-0ec09cae5d5a None admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:40:35.490569 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c514f3e-1b8d-456b-a176-d9513fdc9418 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:40:35.491440 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c514f3e-1b8d-456b-a176-d9513fdc9418 None admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:40:35.491720 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c514f3e-1b8d-456b-a176-d9513fdc9418 None admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:40:35.491991 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c514f3e-1b8d-456b-a176-d9513fdc9418 None admin] PATH_INFO: `/v3/users/b3478ea957234bb99fde27e6c3d0dfd3` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:40:35.493720 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c514f3e-1b8d-456b-a176-d9513fdc9418 None admin] RBAC: Authorizing `identity:get_user(user_id=b3478ea957234bb99fde27e6c3d0dfd3)` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:40:35.495839 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c514f3e-1b8d-456b-a176-d9513fdc9418 None admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:40:35.498108 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 87/174] 10.222.0.30 () {56 vars in 1177 bytes} [Mon May 23 12:40:35 2022] GET /identity/v3/users/b3478ea957234bb99fde27e6c3d0dfd3 => generated 275 bytes in 179 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) May 23 12:40:35.499977 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c514f3e-1b8d-456b-a176-d9513fdc9418 None admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:40:35.537658 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=100 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:40:35.579562 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 88/175] 10.222.0.30 () {58 vars in 1439 bytes} [Mon May 23 12:40:35 2022] PUT /identity/v3/projects/486401ae2e5f40d5a5f060242896ae53/users/5c3e3c32e6c8497c8e3d98e09b8de7be/roles/c223286ed21a4654a0f2b4e5551831c7 => generated 0 bytes in 397 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) May 23 12:40:35.580484 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43976495-904e-43e8-8a8a-0ec09cae5d5a None admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:40:35.654139 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c514f3e-1b8d-456b-a176-d9513fdc9418 None admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:40:35.741058 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-efb1b4be-4ed2-43c0-9f03-5bdc32a715ba None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:40:35.742057 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efb1b4be-4ed2-43c0-9f03-5bdc32a715ba None admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:40:35.742347 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efb1b4be-4ed2-43c0-9f03-5bdc32a715ba None admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:40:35.742618 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efb1b4be-4ed2-43c0-9f03-5bdc32a715ba None admin] PATH_INFO: `/v3/projects/486401ae2e5f40d5a5f060242896ae53` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:40:35.744640 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-efb1b4be-4ed2-43c0-9f03-5bdc32a715ba None admin] RBAC: Authorizing `identity:get_project(project_id=486401ae2e5f40d5a5f060242896ae53)` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:40:35.750681 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-efb1b4be-4ed2-43c0-9f03-5bdc32a715ba None admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:40:35.752805 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 88/176] 10.222.0.30 () {56 vars in 1186 bytes} [Mon May 23 12:40:35 2022] GET /identity/v3/projects/486401ae2e5f40d5a5f060242896ae53 => generated 298 bytes in 254 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) May 23 12:40:35.758859 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-efb1b4be-4ed2-43c0-9f03-5bdc32a715ba None admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:40:35.820209 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43976495-904e-43e8-8a8a-0ec09cae5d5a None admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:40:35.882171 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:40:35.900171 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a3d79f4-bb0e-4183-851a-9df429374363 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:40:35.900838 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a3d79f4-bb0e-4183-851a-9df429374363 None admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:40:35.901118 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a3d79f4-bb0e-4183-851a-9df429374363 None admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:40:35.901500 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a3d79f4-bb0e-4183-851a-9df429374363 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:40:35.973822 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-efb1b4be-4ed2-43c0-9f03-5bdc32a715ba None admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:40:35.975760 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=100 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:40:36.019899 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2387da9f-f407-4199-8f2e-61f7bf54ad10 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:40:36.021165 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2387da9f-f407-4199-8f2e-61f7bf54ad10 None admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:40:36.021453 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2387da9f-f407-4199-8f2e-61f7bf54ad10 None admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:40:36.022049 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2387da9f-f407-4199-8f2e-61f7bf54ad10 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:40:36.022616 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a3d79f4-bb0e-4183-851a-9df429374363 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:40:36.022891 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2387da9f-f407-4199-8f2e-61f7bf54ad10 None admin] RBAC: Adding query filter params (role.id=e3b90948645643e888620ab1e5c774bf, scope.project.id=486401ae2e5f40d5a5f060242896ae53, user.id=b3478ea957234bb99fde27e6c3d0dfd3) {{(pid=92057) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} May 23 12:40:36.023206 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2387da9f-f407-4199-8f2e-61f7bf54ad10 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:40:36.024190 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a3d79f4-bb0e-4183-851a-9df429374363 None admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:40:36.024673 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2387da9f-f407-4199-8f2e-61f7bf54ad10 None admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:40:36.034889 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 89/177] 10.222.0.30 () {56 vars in 1374 bytes} [Mon May 23 12:40:35 2022] GET /identity/v3/role_assignments?user.id=b3478ea957234bb99fde27e6c3d0dfd3&scope.project.id=486401ae2e5f40d5a5f060242896ae53&role.id=e3b90948645643e888620ab1e5c774bf => generated 608 bytes in 275 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) May 23 12:40:36.167760 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 89/178] 10.222.0.30 () {58 vars in 1282 bytes} [Mon May 23 12:40:35 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 588 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) May 23 12:40:36.173662 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2387da9f-f407-4199-8f2e-61f7bf54ad10 None admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:40:36.261633 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-056bea45-75c2-494d-8a6f-3890ed80b34e None None] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:40:36.261962 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-056bea45-75c2-494d-8a6f-3890ed80b34e None None] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:40:36.262585 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-056bea45-75c2-494d-8a6f-3890ed80b34e None None] PATH_INFO: `/` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:40:36.264304 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 90/179] 10.222.0.30 () {54 vars in 926 bytes} [Mon May 23 12:40:36 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) May 23 12:40:36.275460 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69e65672-2cfb-469b-81e6-c15001a0d51d None None] REQUEST_METHOD: `POST` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:40:36.276125 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69e65672-2cfb-469b-81e6-c15001a0d51d None None] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:40:36.277229 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69e65672-2cfb-469b-81e6-c15001a0d51d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:40:36.305557 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.auth.core [None req-69e65672-2cfb-469b-81e6-c15001a0d51d None None] MFA Rules not processed for user `0f7119f8fa5c4445977671140ded91a4`. Rule list: `[]` (Enabled: `True`). {{(pid=92056) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 12:40:36.307178 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.fernet_utils [None req-69e65672-2cfb-469b-81e6-c15001a0d51d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92056) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 12:40:36.327804 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2387da9f-f407-4199-8f2e-61f7bf54ad10 None admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:40:36.352414 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=100 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:40:36.378566 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-056d1d5b-2925-4677-90c0-35b3cfde6eb0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:40:36.379740 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-056d1d5b-2925-4677-90c0-35b3cfde6eb0 None admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:40:36.380160 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-056d1d5b-2925-4677-90c0-35b3cfde6eb0 None admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:40:36.381090 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-056d1d5b-2925-4677-90c0-35b3cfde6eb0 None admin] PATH_INFO: `/v3/projects/40ed642151e34cbfa138265d51948402` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:40:36.383303 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-056d1d5b-2925-4677-90c0-35b3cfde6eb0 None admin] RBAC: Authorizing `identity:get_project(project_id=40ed642151e34cbfa138265d51948402)` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:40:36.385540 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-056d1d5b-2925-4677-90c0-35b3cfde6eb0 None admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:40:36.391184 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 90/180] 10.222.0.30 () {56 vars in 1186 bytes} [Mon May 23 12:40:36 2022] GET /identity/v3/projects/40ed642151e34cbfa138265d51948402 => generated 312 bytes in 218 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) May 23 12:40:36.403020 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-056d1d5b-2925-4677-90c0-35b3cfde6eb0 None admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:40:36.448989 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 91/181] 10.222.0.30 () {58 vars in 1023 bytes} [Mon May 23 12:40:36 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 174 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) May 23 12:40:36.450696 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0890d070-8896-4b90-b485-aed4d9193c05 None None] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:40:36.451145 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0890d070-8896-4b90-b485-aed4d9193c05 None None] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:40:36.451560 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0890d070-8896-4b90-b485-aed4d9193c05 None None] PATH_INFO: `/` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:40:36.457463 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 92/182] 10.222.0.30 () {54 vars in 926 bytes} [Mon May 23 12:40:36 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) May 23 12:40:36.461471 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c9368ae-92c1-43e9-9a42-b1db2f39a1a5 None None] REQUEST_METHOD: `POST` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:40:36.464519 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c9368ae-92c1-43e9-9a42-b1db2f39a1a5 None None] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:40:36.465646 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c9368ae-92c1-43e9-9a42-b1db2f39a1a5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:40:36.470235 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=100 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:40:36.491225 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.auth.core [None req-8c9368ae-92c1-43e9-9a42-b1db2f39a1a5 None None] MFA Rules not processed for user `0f7119f8fa5c4445977671140ded91a4`. Rule list: `[]` (Enabled: `True`). {{(pid=92056) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 12:40:36.492729 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.fernet_utils [None req-8c9368ae-92c1-43e9-9a42-b1db2f39a1a5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92056) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 12:40:36.555801 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-056d1d5b-2925-4677-90c0-35b3cfde6eb0 None admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:40:36.591881 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 93/183] 10.222.0.30 () {58 vars in 1023 bytes} [Mon May 23 12:40:36 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 131 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) May 23 12:40:36.593136 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c9368ae-92c1-43e9-9a42-b1db2f39a1a5 None None] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:40:36.614903 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75971cf8-38d2-4bdc-ad41-3af1549d2b17 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:40:36.615806 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75971cf8-38d2-4bdc-ad41-3af1549d2b17 None admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:40:36.616224 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75971cf8-38d2-4bdc-ad41-3af1549d2b17 None admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:40:36.616635 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75971cf8-38d2-4bdc-ad41-3af1549d2b17 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:40:36.617603 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75971cf8-38d2-4bdc-ad41-3af1549d2b17 None admin] RBAC: Adding query filter params (role.id=80fb22507c9a4ff09d44d8f12f6fff1a, scope.project.id=40ed642151e34cbfa138265d51948402, user.id=b3478ea957234bb99fde27e6c3d0dfd3) {{(pid=92057) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} May 23 12:40:36.618637 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75971cf8-38d2-4bdc-ad41-3af1549d2b17 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:40:36.620445 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75971cf8-38d2-4bdc-ad41-3af1549d2b17 None admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:40:36.627028 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 91/184] 10.222.0.30 () {56 vars in 1374 bytes} [Mon May 23 12:40:36 2022] GET /identity/v3/role_assignments?user.id=b3478ea957234bb99fde27e6c3d0dfd3&scope.project.id=40ed642151e34cbfa138265d51948402&role.id=80fb22507c9a4ff09d44d8f12f6fff1a => generated 608 bytes in 224 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) May 23 12:40:36.628656 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5406bb4-f221-4e92-9a6e-94ea47539b7d None None] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:40:36.629119 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5406bb4-f221-4e92-9a6e-94ea47539b7d None None] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:40:36.629565 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5406bb4-f221-4e92-9a6e-94ea47539b7d None None] PATH_INFO: `/` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:40:36.630856 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 92/185] 10.222.0.30 () {54 vars in 926 bytes} [Mon May 23 12:40:36 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) May 23 12:40:36.631901 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5406bb4-f221-4e92-9a6e-94ea47539b7d None None] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:40:36.734231 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c9368ae-92c1-43e9-9a42-b1db2f39a1a5 None None] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:40:36.746122 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=100 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:40:36.774946 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c0b26c6-0389-40c4-9a05-2f1d1b555004 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:40:36.776093 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c0b26c6-0389-40c4-9a05-2f1d1b555004 None admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:40:36.777408 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c0b26c6-0389-40c4-9a05-2f1d1b555004 None admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:40:36.778040 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c0b26c6-0389-40c4-9a05-2f1d1b555004 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:40:36.780416 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=100 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:40:36.785976 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5406bb4-f221-4e92-9a6e-94ea47539b7d None None] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:40:36.793985 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c0b26c6-0389-40c4-9a05-2f1d1b555004 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} May 23 12:40:36.802554 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application [None req-8c0b26c6-0389-40c4-9a05-2f1d1b555004 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. May 23 12:40:36.802554 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application Traceback (most recent call last): May 23 12:40:36.802554 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request May 23 12:40:36.802554 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application rv = self.dispatch_request() May 23 12:40:36.802554 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request May 23 12:40:36.802554 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) May 23 12:40:36.802554 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper May 23 12:40:36.802554 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) May 23 12:40:36.802554 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view May 23 12:40:36.802554 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) May 23 12:40:36.802554 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request May 23 12:40:36.802554 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) May 23 12:40:36.802554 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get May 23 12:40:36.802554 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self._get_role(role_id) May 23 12:40:36.802554 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role May 23 12:40:36.802554 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') May 23 12:40:36.803562 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call May 23 12:40:36.803562 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( May 23 12:40:36.803562 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data May 23 12:40:36.803562 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) May 23 12:40:36.803562 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped May 23 12:40:36.803562 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) May 23 12:40:36.803562 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun May 23 12:40:36.803562 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) May 23 12:40:36.803562 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func May 23 12:40:36.803562 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self.get_or_create( May 23 12:40:36.803562 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create May 23 12:40:36.803562 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application with Lock( May 23 12:40:36.803562 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ May 23 12:40:36.803562 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self._enter() May 23 12:40:36.803562 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter May 23 12:40:36.803562 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) May 23 12:40:36.804378 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create May 23 12:40:36.804378 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self.creator() May 23 12:40:36.804378 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value May 23 12:40:36.804378 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application created_value = creator( May 23 12:40:36.804378 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role May 23 12:40:36.804378 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) May 23 12:40:36.804378 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role May 23 12:40:36.804378 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() May 23 12:40:36.804378 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role May 23 12:40:36.804378 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) May 23 12:40:36.804378 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. May 23 12:40:36.804378 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application May 23 12:40:36.806660 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 94/186] 10.222.0.30 () {56 vars in 1096 bytes} [Mon May 23 12:40:36 2022] GET /identity/v3/roles/admin => generated 83 bytes in 214 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) May 23 12:40:36.809463 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0db3e7e0-d853-4f34-97b2-70aec0caf77f None None] REQUEST_METHOD: `POST` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:40:36.809953 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0db3e7e0-d853-4f34-97b2-70aec0caf77f None None] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:40:36.810392 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0db3e7e0-d853-4f34-97b2-70aec0caf77f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:40:36.830725 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.auth.core [None req-0db3e7e0-d853-4f34-97b2-70aec0caf77f None None] MFA Rules not processed for user `0f7119f8fa5c4445977671140ded91a4`. Rule list: `[]` (Enabled: `True`). {{(pid=92056) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 12:40:36.832246 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.fernet_utils [None req-0db3e7e0-d853-4f34-97b2-70aec0caf77f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92056) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 12:40:36.840023 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dad5bbbb-b2a2-4da8-b31c-56680b6731f7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:40:36.841024 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dad5bbbb-b2a2-4da8-b31c-56680b6731f7 None admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:40:36.841312 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dad5bbbb-b2a2-4da8-b31c-56680b6731f7 None admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:40:36.841588 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dad5bbbb-b2a2-4da8-b31c-56680b6731f7 None admin] PATH_INFO: `/v3/domains/default` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:40:36.843568 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dad5bbbb-b2a2-4da8-b31c-56680b6731f7 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:40:36.845040 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dad5bbbb-b2a2-4da8-b31c-56680b6731f7 None admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:40:36.848378 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 93/187] 10.222.0.30 () {56 vars in 1108 bytes} [Mon May 23 12:40:36 2022] GET /identity/v3/domains/default => generated 199 bytes in 217 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) May 23 12:40:36.849234 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dad5bbbb-b2a2-4da8-b31c-56680b6731f7 None admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:40:36.919441 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 95/188] 10.222.0.30 () {58 vars in 1023 bytes} [Mon May 23 12:40:36 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 112 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) May 23 12:40:36.920284 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0db3e7e0-d853-4f34-97b2-70aec0caf77f None None] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:40:36.974915 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dad5bbbb-b2a2-4da8-b31c-56680b6731f7 None admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:40:37.008616 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=100 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:40:37.015526 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7487198-0f03-4800-b1b1-b375efbd032e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:40:37.016197 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7487198-0f03-4800-b1b1-b375efbd032e None admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:40:37.016478 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7487198-0f03-4800-b1b1-b375efbd032e None admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:40:37.016751 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7487198-0f03-4800-b1b1-b375efbd032e None admin] PATH_INFO: `/v3/roles` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:40:37.017493 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7487198-0f03-4800-b1b1-b375efbd032e None admin] RBAC: Adding query filter params (name=admin) {{(pid=92057) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} May 23 12:40:37.017798 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7487198-0f03-4800-b1b1-b375efbd032e None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:40:37.019219 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7487198-0f03-4800-b1b1-b375efbd032e None admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:40:37.027540 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 94/189] 10.222.0.30 () {56 vars in 1099 bytes} [Mon May 23 12:40:36 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 179 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) May 23 12:40:37.033058 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7487198-0f03-4800-b1b1-b375efbd032e None admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:40:37.034382 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0db3e7e0-d853-4f34-97b2-70aec0caf77f None None] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:40:37.066202 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=100 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:40:37.069159 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6fbab0a-32c7-4237-9fdb-fb469cf4cc76 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:40:37.069815 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6fbab0a-32c7-4237-9fdb-fb469cf4cc76 None admin] REQUEST_METHOD: `POST` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:40:37.070125 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6fbab0a-32c7-4237-9fdb-fb469cf4cc76 None admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:40:37.070415 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6fbab0a-32c7-4237-9fdb-fb469cf4cc76 None admin] PATH_INFO: `/v3/users` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:40:37.071102 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6fbab0a-32c7-4237-9fdb-fb469cf4cc76 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:40:37.072562 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6fbab0a-32c7-4237-9fdb-fb469cf4cc76 None admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:40:37.134511 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7487198-0f03-4800-b1b1-b375efbd032e None admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:40:37.167849 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3d2213b-3e79-4091-adb7-839f23cdb0af None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:40:37.168677 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3d2213b-3e79-4091-adb7-839f23cdb0af None admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:40:37.168962 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3d2213b-3e79-4091-adb7-839f23cdb0af None admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:40:37.169233 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3d2213b-3e79-4091-adb7-839f23cdb0af None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:40:37.179066 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a3d2213b-3e79-4091-adb7-839f23cdb0af None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} May 23 12:40:37.187612 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application [None req-a3d2213b-3e79-4091-adb7-839f23cdb0af None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. May 23 12:40:37.187612 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application Traceback (most recent call last): May 23 12:40:37.187612 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request May 23 12:40:37.187612 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application rv = self.dispatch_request() May 23 12:40:37.187612 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request May 23 12:40:37.187612 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) May 23 12:40:37.187612 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper May 23 12:40:37.187612 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) May 23 12:40:37.187612 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view May 23 12:40:37.187612 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) May 23 12:40:37.187612 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request May 23 12:40:37.187612 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) May 23 12:40:37.187612 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get May 23 12:40:37.187612 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self._get_role(role_id) May 23 12:40:37.187612 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role May 23 12:40:37.187612 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') May 23 12:40:37.188716 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call May 23 12:40:37.188716 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( May 23 12:40:37.188716 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data May 23 12:40:37.188716 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) May 23 12:40:37.188716 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped May 23 12:40:37.188716 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) May 23 12:40:37.188716 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun May 23 12:40:37.188716 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) May 23 12:40:37.188716 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func May 23 12:40:37.188716 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self.get_or_create( May 23 12:40:37.188716 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create May 23 12:40:37.188716 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application with Lock( May 23 12:40:37.188716 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ May 23 12:40:37.188716 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self._enter() May 23 12:40:37.188716 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter May 23 12:40:37.188716 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) May 23 12:40:37.189570 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create May 23 12:40:37.189570 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self.creator() May 23 12:40:37.189570 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value May 23 12:40:37.189570 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application created_value = creator( May 23 12:40:37.189570 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role May 23 12:40:37.189570 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) May 23 12:40:37.189570 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role May 23 12:40:37.189570 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() May 23 12:40:37.189570 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role May 23 12:40:37.189570 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) May 23 12:40:37.189570 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. May 23 12:40:37.189570 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application May 23 12:40:37.189570 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 95/190] 10.222.0.30 () {56 vars in 1114 bytes} [Mon May 23 12:40:37 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 161 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) May 23 12:40:37.189570 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3d2213b-3e79-4091-adb7-839f23cdb0af None admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:40:37.241768 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 96/191] 10.222.0.30 () {60 vars in 1132 bytes} [Mon May 23 12:40:36 2022] POST /identity/v3/users => generated 297 bytes in 314 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) May 23 12:40:37.241768 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6fbab0a-32c7-4237-9fdb-fb469cf4cc76 None admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:40:37.292141 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=100 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:40:37.304969 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3d2213b-3e79-4091-adb7-839f23cdb0af None admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:40:37.353267 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb3dc815-f0d4-4d93-8165-40c4f00c3d94 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:40:37.356717 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb3dc815-f0d4-4d93-8165-40c4f00c3d94 None admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:40:37.356945 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb3dc815-f0d4-4d93-8165-40c4f00c3d94 None admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:40:37.357222 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb3dc815-f0d4-4d93-8165-40c4f00c3d94 None admin] PATH_INFO: `/v3/users/237cf4210ddf4163a2353c03fdf231a9` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:40:37.375679 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb3dc815-f0d4-4d93-8165-40c4f00c3d94 None admin] RBAC: Authorizing `identity:get_user(user_id=237cf4210ddf4163a2353c03fdf231a9)` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:40:37.376933 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb3dc815-f0d4-4d93-8165-40c4f00c3d94 None admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:40:37.378815 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 96/192] 10.222.0.30 () {56 vars in 1177 bytes} [Mon May 23 12:40:37 2022] GET /identity/v3/users/237cf4210ddf4163a2353c03fdf231a9 => generated 283 bytes in 189 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) May 23 12:40:37.384297 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb3dc815-f0d4-4d93-8165-40c4f00c3d94 None admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:40:37.388290 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6fbab0a-32c7-4237-9fdb-fb469cf4cc76 None admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:40:37.437350 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:40:37.444404 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d9a747d-ab72-473b-b4df-5d68e7277f4c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:40:37.445082 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d9a747d-ab72-473b-b4df-5d68e7277f4c None admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:40:37.445366 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d9a747d-ab72-473b-b4df-5d68e7277f4c None admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:40:37.445643 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d9a747d-ab72-473b-b4df-5d68e7277f4c None admin] PATH_INFO: `/v3/roles` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:40:37.446885 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d9a747d-ab72-473b-b4df-5d68e7277f4c None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=92056) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} May 23 12:40:37.447256 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d9a747d-ab72-473b-b4df-5d68e7277f4c None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:40:37.453053 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d9a747d-ab72-473b-b4df-5d68e7277f4c None admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:40:37.465576 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 97/193] 10.222.0.30 () {56 vars in 1111 bytes} [Mon May 23 12:40:37 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 230 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) May 23 12:40:37.466894 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d9a747d-ab72-473b-b4df-5d68e7277f4c None admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:40:37.518742 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb3dc815-f0d4-4d93-8165-40c4f00c3d94 None admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:40:37.564688 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1bf49bd5-e457-4161-a1f7-e3a949838a46 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:40:37.565429 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bf49bd5-e457-4161-a1f7-e3a949838a46 None admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:40:37.565719 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bf49bd5-e457-4161-a1f7-e3a949838a46 None admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:40:37.566034 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bf49bd5-e457-4161-a1f7-e3a949838a46 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:40:37.584315 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=100 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:40:37.590186 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d9a747d-ab72-473b-b4df-5d68e7277f4c None admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:40:37.635200 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6721df7c-9611-4ad9-bbd2-c0a2d81a14f9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:40:37.636250 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6721df7c-9611-4ad9-bbd2-c0a2d81a14f9 None admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:40:37.636643 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6721df7c-9611-4ad9-bbd2-c0a2d81a14f9 None admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:40:37.636930 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6721df7c-9611-4ad9-bbd2-c0a2d81a14f9 None admin] PATH_INFO: `/v3/users/237cf4210ddf4163a2353c03fdf231a9` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:40:37.638753 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6721df7c-9611-4ad9-bbd2-c0a2d81a14f9 None admin] RBAC: Authorizing `identity:get_user(user_id=237cf4210ddf4163a2353c03fdf231a9)` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:40:37.640216 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6721df7c-9611-4ad9-bbd2-c0a2d81a14f9 None admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:40:37.641891 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 98/194] 10.222.0.30 () {56 vars in 1177 bytes} [Mon May 23 12:40:37 2022] GET /identity/v3/users/237cf4210ddf4163a2353c03fdf231a9 => generated 283 bytes in 175 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) May 23 12:40:37.647941 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6721df7c-9611-4ad9-bbd2-c0a2d81a14f9 None admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:40:37.661238 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1bf49bd5-e457-4161-a1f7-e3a949838a46 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:40:37.663199 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1bf49bd5-e457-4161-a1f7-e3a949838a46 None admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:40:37.729155 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=100 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:40:37.775752 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6721df7c-9611-4ad9-bbd2-c0a2d81a14f9 None admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:40:37.796826 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 97/195] 10.222.0.30 () {58 vars in 1282 bytes} [Mon May 23 12:40:37 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 413 msecs (HTTP/1.1 200) 6 headers in 358 bytes (2 switches on core 0) May 23 12:40:37.803599 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1bf49bd5-e457-4161-a1f7-e3a949838a46 None admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:40:37.825542 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0c8a43e-7ffd-48f9-a5af-4dd1cb24ef35 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:40:37.826484 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0c8a43e-7ffd-48f9-a5af-4dd1cb24ef35 None admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:40:37.826904 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0c8a43e-7ffd-48f9-a5af-4dd1cb24ef35 None admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:40:37.828553 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0c8a43e-7ffd-48f9-a5af-4dd1cb24ef35 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:40:37.856464 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=100 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:40:37.913087 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0c8a43e-7ffd-48f9-a5af-4dd1cb24ef35 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:40:37.914808 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0c8a43e-7ffd-48f9-a5af-4dd1cb24ef35 None admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:40:37.984015 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=100 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:40:38.003462 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1bf49bd5-e457-4161-a1f7-e3a949838a46 None admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:40:38.029378 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 99/196] 10.222.0.30 () {58 vars in 1282 bytes} [Mon May 23 12:40:37 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 382 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) May 23 12:40:38.035736 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0c8a43e-7ffd-48f9-a5af-4dd1cb24ef35 None admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:40:38.096044 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7ba317f-1270-43fe-83fd-acbe3f3d6db4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:40:38.096838 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7ba317f-1270-43fe-83fd-acbe3f3d6db4 None admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:40:38.097121 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7ba317f-1270-43fe-83fd-acbe3f3d6db4 None admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:40:38.097395 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7ba317f-1270-43fe-83fd-acbe3f3d6db4 None admin] PATH_INFO: `/v3/projects/b39742b1d1b34dfe82a9ce4da39b7582` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:40:38.100468 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7ba317f-1270-43fe-83fd-acbe3f3d6db4 None admin] RBAC: Authorizing `identity:get_project(project_id=b39742b1d1b34dfe82a9ce4da39b7582)` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:40:38.102002 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7ba317f-1270-43fe-83fd-acbe3f3d6db4 None admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:40:38.103772 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 98/197] 10.222.0.30 () {56 vars in 1186 bytes} [Mon May 23 12:40:37 2022] GET /identity/v3/projects/b39742b1d1b34dfe82a9ce4da39b7582 => generated 302 bytes in 301 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) May 23 12:40:38.112605 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7ba317f-1270-43fe-83fd-acbe3f3d6db4 None admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:40:38.217673 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0c8a43e-7ffd-48f9-a5af-4dd1cb24ef35 None admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:40:38.294632 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be7ede46-d7f4-41e8-b78c-353406c970cf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:40:38.295671 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be7ede46-d7f4-41e8-b78c-353406c970cf None admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:40:38.296066 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be7ede46-d7f4-41e8-b78c-353406c970cf None admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:40:38.296574 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be7ede46-d7f4-41e8-b78c-353406c970cf None admin] PATH_INFO: `/v3/projects/b39742b1d1b34dfe82a9ce4da39b7582` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:40:38.297797 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be7ede46-d7f4-41e8-b78c-353406c970cf None admin] RBAC: Authorizing `identity:get_project(project_id=b39742b1d1b34dfe82a9ce4da39b7582)` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:40:38.299481 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be7ede46-d7f4-41e8-b78c-353406c970cf None admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:40:38.301326 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 100/198] 10.222.0.30 () {56 vars in 1186 bytes} [Mon May 23 12:40:38 2022] GET /identity/v3/projects/b39742b1d1b34dfe82a9ce4da39b7582 => generated 302 bytes in 266 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) May 23 12:40:38.314087 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7ba317f-1270-43fe-83fd-acbe3f3d6db4 None admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:40:38.322814 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be7ede46-d7f4-41e8-b78c-353406c970cf None admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:40:38.330124 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=100 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:40:38.357117 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9a6d046-9455-424b-bf4f-ce917f7f7a02 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:40:38.358868 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9a6d046-9455-424b-bf4f-ce917f7f7a02 None admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:40:38.361951 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9a6d046-9455-424b-bf4f-ce917f7f7a02 None admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:40:38.362374 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9a6d046-9455-424b-bf4f-ce917f7f7a02 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:40:38.363373 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9a6d046-9455-424b-bf4f-ce917f7f7a02 None admin] RBAC: Adding query filter params (role.id=bc3c3c9d654940cbaa5177a6be23d1bc, scope.project.id=b39742b1d1b34dfe82a9ce4da39b7582, user.id=237cf4210ddf4163a2353c03fdf231a9) {{(pid=92057) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} May 23 12:40:38.363879 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9a6d046-9455-424b-bf4f-ce917f7f7a02 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:40:38.365468 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9a6d046-9455-424b-bf4f-ce917f7f7a02 None admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:40:38.381971 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=100 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:40:38.383736 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 99/199] 10.222.0.30 () {56 vars in 1374 bytes} [Mon May 23 12:40:38 2022] GET /identity/v3/role_assignments?user.id=237cf4210ddf4163a2353c03fdf231a9&scope.project.id=b39742b1d1b34dfe82a9ce4da39b7582&role.id=bc3c3c9d654940cbaa5177a6be23d1bc => generated 259 bytes in 272 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) May 23 12:40:38.402454 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-936784ed-63b6-497a-9711-171d3aeb7031 None None] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:40:38.403483 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-936784ed-63b6-497a-9711-171d3aeb7031 None None] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:40:38.403935 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-936784ed-63b6-497a-9711-171d3aeb7031 None None] PATH_INFO: `/` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:40:38.406272 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 100/200] 10.222.0.30 () {54 vars in 926 bytes} [Mon May 23 12:40:38 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) May 23 12:40:38.413071 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2f7895c-150e-44eb-925d-afb5da376a18 None None] REQUEST_METHOD: `POST` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:40:38.413354 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2f7895c-150e-44eb-925d-afb5da376a18 None None] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:40:38.413634 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2f7895c-150e-44eb-925d-afb5da376a18 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:40:38.441073 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.auth.core [None req-d2f7895c-150e-44eb-925d-afb5da376a18 None None] MFA Rules not processed for user `0f7119f8fa5c4445977671140ded91a4`. Rule list: `[]` (Enabled: `True`). {{(pid=92057) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 12:40:38.442316 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.fernet_utils [None req-d2f7895c-150e-44eb-925d-afb5da376a18 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92057) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 12:40:38.517285 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be7ede46-d7f4-41e8-b78c-353406c970cf None admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:40:38.580128 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 101/201] 10.222.0.30 () {58 vars in 1023 bytes} [Mon May 23 12:40:38 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 168 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) May 23 12:40:38.590075 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06ead866-7ffd-4dc0-9b68-f7306e355821 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:40:38.590744 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06ead866-7ffd-4dc0-9b68-f7306e355821 None admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:40:38.591023 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06ead866-7ffd-4dc0-9b68-f7306e355821 None admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:40:38.591296 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06ead866-7ffd-4dc0-9b68-f7306e355821 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:40:38.592135 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06ead866-7ffd-4dc0-9b68-f7306e355821 None admin] RBAC: Adding query filter params (role.id=e3b90948645643e888620ab1e5c774bf, scope.project.id=b39742b1d1b34dfe82a9ce4da39b7582, user.id=237cf4210ddf4163a2353c03fdf231a9) {{(pid=92056) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} May 23 12:40:38.592446 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06ead866-7ffd-4dc0-9b68-f7306e355821 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:40:38.593935 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06ead866-7ffd-4dc0-9b68-f7306e355821 None admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:40:38.608167 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 101/202] 10.222.0.30 () {56 vars in 1374 bytes} [Mon May 23 12:40:38 2022] GET /identity/v3/role_assignments?user.id=237cf4210ddf4163a2353c03fdf231a9&scope.project.id=b39742b1d1b34dfe82a9ce4da39b7582&role.id=e3b90948645643e888620ab1e5c774bf => generated 259 bytes in 286 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) May 23 12:40:38.618552 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2f7895c-150e-44eb-925d-afb5da376a18 None None] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:40:38.753730 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2f7895c-150e-44eb-925d-afb5da376a18 None None] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:40:38.772938 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=100 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:40:38.812976 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61be1e60-8ff9-4742-9209-fa31cd6b5ec2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:40:38.813807 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61be1e60-8ff9-4742-9209-fa31cd6b5ec2 None admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:40:38.818457 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61be1e60-8ff9-4742-9209-fa31cd6b5ec2 None admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:40:38.818751 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61be1e60-8ff9-4742-9209-fa31cd6b5ec2 None admin] PATH_INFO: `/v3/roles/reader` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:40:38.838544 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61be1e60-8ff9-4742-9209-fa31cd6b5ec2 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} May 23 12:40:38.846471 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application [None req-61be1e60-8ff9-4742-9209-fa31cd6b5ec2 None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. May 23 12:40:38.846471 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application Traceback (most recent call last): May 23 12:40:38.846471 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request May 23 12:40:38.846471 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application rv = self.dispatch_request() May 23 12:40:38.846471 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request May 23 12:40:38.846471 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) May 23 12:40:38.846471 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper May 23 12:40:38.846471 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) May 23 12:40:38.846471 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view May 23 12:40:38.846471 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) May 23 12:40:38.846471 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request May 23 12:40:38.846471 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) May 23 12:40:38.846471 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get May 23 12:40:38.846471 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self._get_role(role_id) May 23 12:40:38.846471 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role May 23 12:40:38.846471 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') May 23 12:40:38.847357 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call May 23 12:40:38.847357 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( May 23 12:40:38.847357 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data May 23 12:40:38.847357 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) May 23 12:40:38.847357 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped May 23 12:40:38.847357 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) May 23 12:40:38.847357 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun May 23 12:40:38.847357 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) May 23 12:40:38.847357 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func May 23 12:40:38.847357 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self.get_or_create( May 23 12:40:38.847357 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create May 23 12:40:38.847357 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application with Lock( May 23 12:40:38.847357 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ May 23 12:40:38.847357 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self._enter() May 23 12:40:38.847357 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter May 23 12:40:38.847357 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) May 23 12:40:38.848171 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create May 23 12:40:38.848171 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self.creator() May 23 12:40:38.848171 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value May 23 12:40:38.848171 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application created_value = creator( May 23 12:40:38.848171 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role May 23 12:40:38.848171 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) May 23 12:40:38.848171 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role May 23 12:40:38.848171 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() May 23 12:40:38.848171 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role May 23 12:40:38.848171 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) May 23 12:40:38.848171 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. May 23 12:40:38.848171 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application May 23 12:40:38.848171 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 102/203] 10.222.0.30 () {56 vars in 1099 bytes} [Mon May 23 12:40:38 2022] GET /identity/v3/roles/reader => generated 84 bytes in 230 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) May 23 12:40:38.856061 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06ead866-7ffd-4dc0-9b68-f7306e355821 None admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:40:39.003036 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=100 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:40:39.010098 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06ead866-7ffd-4dc0-9b68-f7306e355821 None admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:40:39.067410 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68ba7987-dc55-4675-b152-a97b0cf23315 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:40:39.068094 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68ba7987-dc55-4675-b152-a97b0cf23315 None admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:40:39.068400 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68ba7987-dc55-4675-b152-a97b0cf23315 None admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:40:39.068681 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68ba7987-dc55-4675-b152-a97b0cf23315 None admin] PATH_INFO: `/v3/roles` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:40:39.069444 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68ba7987-dc55-4675-b152-a97b0cf23315 None admin] RBAC: Adding query filter params (name=reader) {{(pid=92056) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} May 23 12:40:39.069780 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68ba7987-dc55-4675-b152-a97b0cf23315 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:40:39.071196 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68ba7987-dc55-4675-b152-a97b0cf23315 None admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:40:39.084838 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 102/204] 10.222.0.30 () {56 vars in 1101 bytes} [Mon May 23 12:40:38 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 230 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) May 23 12:40:39.091266 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61be1e60-8ff9-4742-9209-fa31cd6b5ec2 None admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:40:39.210620 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61be1e60-8ff9-4742-9209-fa31cd6b5ec2 None admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:40:39.253455 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a9732e5-6ad1-43c2-9143-4bfc2eb92c0b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:40:39.254276 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a9732e5-6ad1-43c2-9143-4bfc2eb92c0b None admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:40:39.254561 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a9732e5-6ad1-43c2-9143-4bfc2eb92c0b None admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:40:39.254843 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a9732e5-6ad1-43c2-9143-4bfc2eb92c0b None admin] PATH_INFO: `/v3/users/5c3e3c32e6c8497c8e3d98e09b8de7be` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:40:39.256049 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a9732e5-6ad1-43c2-9143-4bfc2eb92c0b None admin] RBAC: Authorizing `identity:get_user(user_id=5c3e3c32e6c8497c8e3d98e09b8de7be)` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:40:39.257524 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a9732e5-6ad1-43c2-9143-4bfc2eb92c0b None admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:40:39.259519 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 103/205] 10.222.0.30 () {56 vars in 1177 bytes} [Mon May 23 12:40:39 2022] GET /identity/v3/users/5c3e3c32e6c8497c8e3d98e09b8de7be => generated 289 bytes in 169 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) May 23 12:40:39.265627 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68ba7987-dc55-4675-b152-a97b0cf23315 None admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:40:39.405893 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68ba7987-dc55-4675-b152-a97b0cf23315 None admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:40:39.451488 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fdfa78e4-d2d8-471f-93e0-ba5dd55ef3e4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:40:39.452146 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdfa78e4-d2d8-471f-93e0-ba5dd55ef3e4 None admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:40:39.452436 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdfa78e4-d2d8-471f-93e0-ba5dd55ef3e4 None admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:40:39.452712 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdfa78e4-d2d8-471f-93e0-ba5dd55ef3e4 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:40:39.486251 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=100 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:40:39.536101 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fdfa78e4-d2d8-471f-93e0-ba5dd55ef3e4 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:40:39.537532 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fdfa78e4-d2d8-471f-93e0-ba5dd55ef3e4 None admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:40:39.680664 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 103/206] 10.222.0.30 () {58 vars in 1282 bytes} [Mon May 23 12:40:39 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 415 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) May 23 12:40:39.687454 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a9732e5-6ad1-43c2-9143-4bfc2eb92c0b None admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:40:39.739013 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=100 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:40:39.820947 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a9732e5-6ad1-43c2-9143-4bfc2eb92c0b None admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:40:39.869384 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa195229-433d-45bc-8ca9-f1fe4dca1ee3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:40:39.871125 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa195229-433d-45bc-8ca9-f1fe4dca1ee3 None admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:40:39.871514 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa195229-433d-45bc-8ca9-f1fe4dca1ee3 None admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:40:39.871794 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa195229-433d-45bc-8ca9-f1fe4dca1ee3 None admin] PATH_INFO: `/v3/projects/486401ae2e5f40d5a5f060242896ae53` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:40:39.878643 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa195229-433d-45bc-8ca9-f1fe4dca1ee3 None admin] RBAC: Authorizing `identity:get_project(project_id=486401ae2e5f40d5a5f060242896ae53)` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:40:39.880142 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa195229-433d-45bc-8ca9-f1fe4dca1ee3 None admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:40:39.881904 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 104/207] 10.222.0.30 () {56 vars in 1186 bytes} [Mon May 23 12:40:39 2022] GET /identity/v3/projects/486401ae2e5f40d5a5f060242896ae53 => generated 298 bytes in 195 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) May 23 12:40:39.888138 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fdfa78e4-d2d8-471f-93e0-ba5dd55ef3e4 None admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:40:39.966946 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=100 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:40:40.030388 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fdfa78e4-d2d8-471f-93e0-ba5dd55ef3e4 None admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:40:40.083739 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a07d0c9-a701-4a56-b810-a9856920f713 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:40:40.084715 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a07d0c9-a701-4a56-b810-a9856920f713 None admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:40:40.085209 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a07d0c9-a701-4a56-b810-a9856920f713 None admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:40:40.085688 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a07d0c9-a701-4a56-b810-a9856920f713 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:40:40.086721 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a07d0c9-a701-4a56-b810-a9856920f713 None admin] RBAC: Adding query filter params (role.id=c223286ed21a4654a0f2b4e5551831c7, scope.project.id=486401ae2e5f40d5a5f060242896ae53, user.id=5c3e3c32e6c8497c8e3d98e09b8de7be) {{(pid=92056) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} May 23 12:40:40.087223 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a07d0c9-a701-4a56-b810-a9856920f713 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:40:40.089423 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a07d0c9-a701-4a56-b810-a9856920f713 None admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:40:40.102653 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 104/208] 10.222.0.30 () {56 vars in 1374 bytes} [Mon May 23 12:40:39 2022] GET /identity/v3/role_assignments?user.id=5c3e3c32e6c8497c8e3d98e09b8de7be&scope.project.id=486401ae2e5f40d5a5f060242896ae53&role.id=c223286ed21a4654a0f2b4e5551831c7 => generated 608 bytes in 215 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) May 23 12:40:40.254493 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-018b1866-83af-4597-b67a-b67146b0d3dd None None] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:40:40.254778 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-018b1866-83af-4597-b67a-b67146b0d3dd None None] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:40:40.255305 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-018b1866-83af-4597-b67a-b67146b0d3dd None None] PATH_INFO: `/` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:40:40.256909 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 105/209] 10.222.0.30 () {54 vars in 926 bytes} [Mon May 23 12:40:40 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) May 23 12:40:40.263406 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97bfd2fd-5333-4209-b070-25d2cca9592f None None] REQUEST_METHOD: `POST` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:40:40.264045 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97bfd2fd-5333-4209-b070-25d2cca9592f None None] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:40:40.264526 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97bfd2fd-5333-4209-b070-25d2cca9592f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:40:40.283145 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.auth.core [None req-97bfd2fd-5333-4209-b070-25d2cca9592f None None] MFA Rules not processed for user `0f7119f8fa5c4445977671140ded91a4`. Rule list: `[]` (Enabled: `True`). {{(pid=92056) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 12:40:40.284696 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.fernet_utils [None req-97bfd2fd-5333-4209-b070-25d2cca9592f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92056) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 12:40:40.362466 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 105/210] 10.222.0.30 () {58 vars in 1023 bytes} [Mon May 23 12:40:40 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 100 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) May 23 12:40:40.406254 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-018b1866-83af-4597-b67a-b67146b0d3dd None None] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:40:40.473087 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-037c05d1-d337-42b9-81ac-fe040ae1d344 None None] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:40:40.473586 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-037c05d1-d337-42b9-81ac-fe040ae1d344 None None] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:40:40.473800 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-037c05d1-d337-42b9-81ac-fe040ae1d344 None None] PATH_INFO: `/` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:40:40.475264 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 106/211] 10.222.0.30 () {54 vars in 926 bytes} [Mon May 23 12:40:40 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) May 23 12:40:40.483565 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bc367bc-8e79-4ade-88e5-c1a30b97ebc2 None None] REQUEST_METHOD: `POST` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:40:40.484222 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bc367bc-8e79-4ade-88e5-c1a30b97ebc2 None None] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:40:40.484716 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bc367bc-8e79-4ade-88e5-c1a30b97ebc2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:40:40.504187 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.auth.core [None req-0bc367bc-8e79-4ade-88e5-c1a30b97ebc2 None None] MFA Rules not processed for user `0f7119f8fa5c4445977671140ded91a4`. Rule list: `[]` (Enabled: `True`). {{(pid=92056) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 12:40:40.505747 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.fernet_utils [None req-0bc367bc-8e79-4ade-88e5-c1a30b97ebc2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92056) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 12:40:40.550511 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-018b1866-83af-4597-b67a-b67146b0d3dd None None] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:40:40.553250 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=100 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:40:40.563274 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=100 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:40:40.585335 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 107/212] 10.222.0.30 () {58 vars in 1023 bytes} [Mon May 23 12:40:40 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 103 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) May 23 12:40:40.593589 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-416cbb9f-ffc4-45db-8d75-ea9efe36707d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:40:40.594605 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-416cbb9f-ffc4-45db-8d75-ea9efe36707d None admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:40:40.595053 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-416cbb9f-ffc4-45db-8d75-ea9efe36707d None admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:40:40.595469 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-416cbb9f-ffc4-45db-8d75-ea9efe36707d None admin] PATH_INFO: `/v3/domains/default` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:40:40.596770 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-416cbb9f-ffc4-45db-8d75-ea9efe36707d None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:40:40.598392 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-416cbb9f-ffc4-45db-8d75-ea9efe36707d None admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:40:40.606203 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 106/213] 10.222.0.30 () {56 vars in 1108 bytes} [Mon May 23 12:40:40 2022] GET /identity/v3/domains/default => generated 199 bytes in 195 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) May 23 12:40:40.606868 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0bc367bc-8e79-4ade-88e5-c1a30b97ebc2 None None] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:40:40.617785 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-416cbb9f-ffc4-45db-8d75-ea9efe36707d None admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:40:40.749819 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0bc367bc-8e79-4ade-88e5-c1a30b97ebc2 None None] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:40:40.754971 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-416cbb9f-ffc4-45db-8d75-ea9efe36707d None admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:40:40.795721 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ebb2d675-b79b-40e8-bfeb-5a704da35868 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:40:40.796638 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebb2d675-b79b-40e8-bfeb-5a704da35868 None admin] REQUEST_METHOD: `POST` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:40:40.797289 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebb2d675-b79b-40e8-bfeb-5a704da35868 None admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:40:40.798121 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebb2d675-b79b-40e8-bfeb-5a704da35868 None admin] PATH_INFO: `/v3/users` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:40:40.798976 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ebb2d675-b79b-40e8-bfeb-5a704da35868 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:40:40.800891 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ebb2d675-b79b-40e8-bfeb-5a704da35868 None admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:40:40.807034 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed5a1d79-be8c-4b72-a92b-cf3e213ca345 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:40:40.809847 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed5a1d79-be8c-4b72-a92b-cf3e213ca345 None admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:40:40.810364 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed5a1d79-be8c-4b72-a92b-cf3e213ca345 None admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:40:40.810788 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed5a1d79-be8c-4b72-a92b-cf3e213ca345 None admin] PATH_INFO: `/v3/roles/member` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:40:40.822771 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed5a1d79-be8c-4b72-a92b-cf3e213ca345 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} May 23 12:40:40.831811 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application [None req-ed5a1d79-be8c-4b72-a92b-cf3e213ca345 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. May 23 12:40:40.831811 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application Traceback (most recent call last): May 23 12:40:40.831811 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request May 23 12:40:40.831811 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application rv = self.dispatch_request() May 23 12:40:40.831811 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request May 23 12:40:40.831811 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) May 23 12:40:40.831811 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper May 23 12:40:40.831811 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) May 23 12:40:40.831811 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view May 23 12:40:40.831811 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) May 23 12:40:40.831811 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request May 23 12:40:40.831811 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) May 23 12:40:40.831811 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get May 23 12:40:40.831811 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self._get_role(role_id) May 23 12:40:40.831811 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role May 23 12:40:40.831811 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') May 23 12:40:40.832921 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call May 23 12:40:40.832921 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( May 23 12:40:40.832921 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data May 23 12:40:40.832921 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) May 23 12:40:40.832921 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped May 23 12:40:40.832921 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) May 23 12:40:40.832921 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun May 23 12:40:40.832921 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) May 23 12:40:40.832921 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func May 23 12:40:40.832921 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self.get_or_create( May 23 12:40:40.832921 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create May 23 12:40:40.832921 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application with Lock( May 23 12:40:40.832921 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ May 23 12:40:40.832921 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self._enter() May 23 12:40:40.832921 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter May 23 12:40:40.832921 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) May 23 12:40:40.833941 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create May 23 12:40:40.833941 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self.creator() May 23 12:40:40.833941 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value May 23 12:40:40.833941 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application created_value = creator( May 23 12:40:40.833941 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role May 23 12:40:40.833941 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) May 23 12:40:40.833941 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role May 23 12:40:40.833941 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() May 23 12:40:40.833941 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role May 23 12:40:40.833941 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) May 23 12:40:40.833941 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. May 23 12:40:40.833941 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application May 23 12:40:40.836155 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 107/214] 10.222.0.30 () {56 vars in 1099 bytes} [Mon May 23 12:40:40 2022] GET /identity/v3/roles/member => generated 84 bytes in 223 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) May 23 12:40:40.841906 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed5a1d79-be8c-4b72-a92b-cf3e213ca345 None admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:40:40.880115 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=100 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:40:40.923371 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 108/215] 10.222.0.30 () {60 vars in 1132 bytes} [Mon May 23 12:40:40 2022] POST /identity/v3/users => generated 297 bytes in 317 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) May 23 12:40:40.957572 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed5a1d79-be8c-4b72-a92b-cf3e213ca345 None admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:40:41.000149 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94b5f77b-4b63-4cdd-8f8e-99ce915a9ab0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:40:41.001180 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94b5f77b-4b63-4cdd-8f8e-99ce915a9ab0 None admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:40:41.001650 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94b5f77b-4b63-4cdd-8f8e-99ce915a9ab0 None admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:40:41.002112 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94b5f77b-4b63-4cdd-8f8e-99ce915a9ab0 None admin] PATH_INFO: `/v3/roles` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:40:41.003044 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94b5f77b-4b63-4cdd-8f8e-99ce915a9ab0 None admin] RBAC: Adding query filter params (name=member) {{(pid=92057) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} May 23 12:40:41.003514 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94b5f77b-4b63-4cdd-8f8e-99ce915a9ab0 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:40:41.005206 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94b5f77b-4b63-4cdd-8f8e-99ce915a9ab0 None admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:40:41.016618 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 108/216] 10.222.0.30 () {56 vars in 1101 bytes} [Mon May 23 12:40:40 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 175 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) May 23 12:40:41.022603 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ebb2d675-b79b-40e8-bfeb-5a704da35868 None admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:40:41.066045 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:40:41.145248 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ebb2d675-b79b-40e8-bfeb-5a704da35868 None admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:40:41.170181 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c4bda58-985e-4dbf-9ad0-5316443a567a None None] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:40:41.170474 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c4bda58-985e-4dbf-9ad0-5316443a567a None None] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:40:41.170755 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c4bda58-985e-4dbf-9ad0-5316443a567a None None] PATH_INFO: `/` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:40:41.172129 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 109/217] 10.222.0.30 () {54 vars in 926 bytes} [Mon May 23 12:40:41 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) May 23 12:40:41.178999 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a15c841c-f9b7-41ef-8954-b6035568aca3 None None] REQUEST_METHOD: `POST` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:40:41.179294 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a15c841c-f9b7-41ef-8954-b6035568aca3 None None] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:40:41.179565 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a15c841c-f9b7-41ef-8954-b6035568aca3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:40:41.180948 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04a755a9-6e56-4c65-9cf9-0619f74af926 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:40:41.181858 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04a755a9-6e56-4c65-9cf9-0619f74af926 None admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:40:41.182137 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04a755a9-6e56-4c65-9cf9-0619f74af926 None admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:40:41.182727 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04a755a9-6e56-4c65-9cf9-0619f74af926 None admin] PATH_INFO: `/v3/users/84f86330620a493684d2835681c966eb` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:40:41.197655 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.auth.core [None req-a15c841c-f9b7-41ef-8954-b6035568aca3 None None] MFA Rules not processed for user `0f7119f8fa5c4445977671140ded91a4`. Rule list: `[]` (Enabled: `True`). {{(pid=92057) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 12:40:41.199570 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.fernet_utils [None req-a15c841c-f9b7-41ef-8954-b6035568aca3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92057) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 12:40:41.204302 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04a755a9-6e56-4c65-9cf9-0619f74af926 None admin] RBAC: Authorizing `identity:get_user(user_id=84f86330620a493684d2835681c966eb)` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:40:41.205785 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04a755a9-6e56-4c65-9cf9-0619f74af926 None admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:40:41.207612 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 109/218] 10.222.0.30 () {56 vars in 1177 bytes} [Mon May 23 12:40:41 2022] GET /identity/v3/users/84f86330620a493684d2835681c966eb => generated 297 bytes in 186 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) May 23 12:40:41.214118 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04a755a9-6e56-4c65-9cf9-0619f74af926 None admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:40:41.294967 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 110/219] 10.222.0.30 () {58 vars in 1023 bytes} [Mon May 23 12:40:41 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 117 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) May 23 12:40:41.297139 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9932d945-1abc-486d-91d3-fb6949483dba None None] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:40:41.297609 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9932d945-1abc-486d-91d3-fb6949483dba None None] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:40:41.298106 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9932d945-1abc-486d-91d3-fb6949483dba None None] PATH_INFO: `/` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:40:41.299836 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 111/220] 10.222.0.30 () {54 vars in 926 bytes} [Mon May 23 12:40:41 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) May 23 12:40:41.306885 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8629e18e-07c2-4840-bbf8-f3136fd10603 None None] REQUEST_METHOD: `POST` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:40:41.307571 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8629e18e-07c2-4840-bbf8-f3136fd10603 None None] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:40:41.308029 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8629e18e-07c2-4840-bbf8-f3136fd10603 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:40:41.330108 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.auth.core [None req-8629e18e-07c2-4840-bbf8-f3136fd10603 None None] MFA Rules not processed for user `0f7119f8fa5c4445977671140ded91a4`. Rule list: `[]` (Enabled: `True`). {{(pid=92057) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 12:40:41.331920 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=100 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:40:41.332305 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.fernet_utils [None req-8629e18e-07c2-4840-bbf8-f3136fd10603 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92057) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 12:40:41.341548 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04a755a9-6e56-4c65-9cf9-0619f74af926 None admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:40:41.355696 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=100 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:40:41.386773 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8872d4c0-4fd0-46a0-a752-6a1d6f5ff1ed None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:40:41.387717 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8872d4c0-4fd0-46a0-a752-6a1d6f5ff1ed None admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:40:41.388471 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8872d4c0-4fd0-46a0-a752-6a1d6f5ff1ed None admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:40:41.389091 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8872d4c0-4fd0-46a0-a752-6a1d6f5ff1ed None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:40:41.416083 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 112/221] 10.222.0.30 () {58 vars in 1023 bytes} [Mon May 23 12:40:41 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 111 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) May 23 12:40:41.417186 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8629e18e-07c2-4840-bbf8-f3136fd10603 None None] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:40:41.501869 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8872d4c0-4fd0-46a0-a752-6a1d6f5ff1ed None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:40:41.503947 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8872d4c0-4fd0-46a0-a752-6a1d6f5ff1ed None admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:40:41.559502 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8629e18e-07c2-4840-bbf8-f3136fd10603 None None] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:40:41.599708 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-036a96f0-5f58-472b-a5ca-8969bd5e7c06 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:40:41.600487 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-036a96f0-5f58-472b-a5ca-8969bd5e7c06 None admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:40:41.600775 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-036a96f0-5f58-472b-a5ca-8969bd5e7c06 None admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:40:41.601049 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-036a96f0-5f58-472b-a5ca-8969bd5e7c06 None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:40:41.606256 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=100 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:40:41.613022 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 110/222] 10.222.0.30 () {58 vars in 1282 bytes} [Mon May 23 12:40:41 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 400 msecs (HTTP/1.1 200) 6 headers in 358 bytes (2 switches on core 0) May 23 12:40:41.613851 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8872d4c0-4fd0-46a0-a752-6a1d6f5ff1ed None admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:40:41.615333 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-036a96f0-5f58-472b-a5ca-8969bd5e7c06 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} May 23 12:40:41.624957 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application [None req-036a96f0-5f58-472b-a5ca-8969bd5e7c06 None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. May 23 12:40:41.624957 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application Traceback (most recent call last): May 23 12:40:41.624957 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request May 23 12:40:41.624957 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application rv = self.dispatch_request() May 23 12:40:41.624957 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request May 23 12:40:41.624957 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) May 23 12:40:41.624957 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper May 23 12:40:41.624957 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) May 23 12:40:41.624957 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view May 23 12:40:41.624957 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) May 23 12:40:41.624957 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request May 23 12:40:41.624957 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) May 23 12:40:41.624957 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get May 23 12:40:41.624957 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self._get_role(role_id) May 23 12:40:41.624957 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role May 23 12:40:41.624957 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') May 23 12:40:41.625961 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call May 23 12:40:41.625961 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( May 23 12:40:41.625961 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data May 23 12:40:41.625961 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) May 23 12:40:41.625961 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped May 23 12:40:41.625961 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) May 23 12:40:41.625961 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun May 23 12:40:41.625961 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) May 23 12:40:41.625961 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func May 23 12:40:41.625961 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self.get_or_create( May 23 12:40:41.625961 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create May 23 12:40:41.625961 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application with Lock( May 23 12:40:41.625961 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ May 23 12:40:41.625961 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self._enter() May 23 12:40:41.625961 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter May 23 12:40:41.625961 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) May 23 12:40:41.626774 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create May 23 12:40:41.626774 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self.creator() May 23 12:40:41.626774 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value May 23 12:40:41.626774 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application created_value = creator( May 23 12:40:41.626774 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role May 23 12:40:41.626774 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) May 23 12:40:41.626774 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role May 23 12:40:41.626774 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() May 23 12:40:41.626774 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role May 23 12:40:41.626774 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) May 23 12:40:41.626774 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. May 23 12:40:41.626774 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application May 23 12:40:41.629202 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 113/223] 10.222.0.30 () {56 vars in 1114 bytes} [Mon May 23 12:40:41 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 212 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) May 23 12:40:41.630388 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-036a96f0-5f58-472b-a5ca-8969bd5e7c06 None admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:40:41.649784 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=100 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:40:41.721303 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8872d4c0-4fd0-46a0-a752-6a1d6f5ff1ed None admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:40:41.752900 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-036a96f0-5f58-472b-a5ca-8969bd5e7c06 None admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:40:41.756482 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1dcc4b2b-c28f-4d89-8029-2a05e0d70cfe None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:40:41.756614 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1dcc4b2b-c28f-4d89-8029-2a05e0d70cfe None admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:40:41.756862 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1dcc4b2b-c28f-4d89-8029-2a05e0d70cfe None admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:40:41.757134 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1dcc4b2b-c28f-4d89-8029-2a05e0d70cfe None admin] PATH_INFO: `/v3/roles/admin` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:40:41.768050 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1dcc4b2b-c28f-4d89-8029-2a05e0d70cfe None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} May 23 12:40:41.775700 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application [None req-1dcc4b2b-c28f-4d89-8029-2a05e0d70cfe None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. May 23 12:40:41.775700 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application Traceback (most recent call last): May 23 12:40:41.775700 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request May 23 12:40:41.775700 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application rv = self.dispatch_request() May 23 12:40:41.775700 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request May 23 12:40:41.775700 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) May 23 12:40:41.775700 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper May 23 12:40:41.775700 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) May 23 12:40:41.775700 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view May 23 12:40:41.775700 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) May 23 12:40:41.775700 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request May 23 12:40:41.775700 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) May 23 12:40:41.775700 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get May 23 12:40:41.775700 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self._get_role(role_id) May 23 12:40:41.775700 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role May 23 12:40:41.775700 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') May 23 12:40:41.777805 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call May 23 12:40:41.777805 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( May 23 12:40:41.777805 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data May 23 12:40:41.777805 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) May 23 12:40:41.777805 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped May 23 12:40:41.777805 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) May 23 12:40:41.777805 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun May 23 12:40:41.777805 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) May 23 12:40:41.777805 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func May 23 12:40:41.777805 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self.get_or_create( May 23 12:40:41.777805 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create May 23 12:40:41.777805 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application with Lock( May 23 12:40:41.777805 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ May 23 12:40:41.777805 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self._enter() May 23 12:40:41.777805 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter May 23 12:40:41.777805 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) May 23 12:40:41.778676 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create May 23 12:40:41.778676 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self.creator() May 23 12:40:41.778676 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value May 23 12:40:41.778676 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application created_value = creator( May 23 12:40:41.778676 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role May 23 12:40:41.778676 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) May 23 12:40:41.778676 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role May 23 12:40:41.778676 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() May 23 12:40:41.778676 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role May 23 12:40:41.778676 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) May 23 12:40:41.778676 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. May 23 12:40:41.778676 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application May 23 12:40:41.778676 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 111/224] 10.222.0.30 () {56 vars in 1096 bytes} [Mon May 23 12:40:41 2022] GET /identity/v3/roles/admin => generated 83 bytes in 164 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) May 23 12:40:41.778676 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1dcc4b2b-c28f-4d89-8029-2a05e0d70cfe None admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:40:41.802507 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6e2d06d-a50c-4c5e-9c2f-3cae90357ea6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:40:41.803442 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6e2d06d-a50c-4c5e-9c2f-3cae90357ea6 None admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:40:41.803802 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6e2d06d-a50c-4c5e-9c2f-3cae90357ea6 None admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:40:41.804332 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6e2d06d-a50c-4c5e-9c2f-3cae90357ea6 None admin] PATH_INFO: `/v3/projects/b39742b1d1b34dfe82a9ce4da39b7582` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:40:41.805555 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6e2d06d-a50c-4c5e-9c2f-3cae90357ea6 None admin] RBAC: Authorizing `identity:get_project(project_id=b39742b1d1b34dfe82a9ce4da39b7582)` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:40:41.808165 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6e2d06d-a50c-4c5e-9c2f-3cae90357ea6 None admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:40:41.810216 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 114/225] 10.222.0.30 () {56 vars in 1186 bytes} [Mon May 23 12:40:41 2022] GET /identity/v3/projects/b39742b1d1b34dfe82a9ce4da39b7582 => generated 302 bytes in 180 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) May 23 12:40:41.810829 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6e2d06d-a50c-4c5e-9c2f-3cae90357ea6 None admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:40:41.878316 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=100 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:40:41.918261 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1dcc4b2b-c28f-4d89-8029-2a05e0d70cfe None admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:40:41.919424 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=100 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:40:41.939471 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6e2d06d-a50c-4c5e-9c2f-3cae90357ea6 None admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:40:41.952939 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be9eed1d-8119-4fc2-bb6a-ad431b2ee5c3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:40:41.953458 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be9eed1d-8119-4fc2-bb6a-ad431b2ee5c3 None admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:40:41.953746 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be9eed1d-8119-4fc2-bb6a-ad431b2ee5c3 None admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:40:41.954064 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be9eed1d-8119-4fc2-bb6a-ad431b2ee5c3 None admin] PATH_INFO: `/v3/roles` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:40:41.954828 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be9eed1d-8119-4fc2-bb6a-ad431b2ee5c3 None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=92056) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} May 23 12:40:41.955131 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be9eed1d-8119-4fc2-bb6a-ad431b2ee5c3 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:40:41.956537 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be9eed1d-8119-4fc2-bb6a-ad431b2ee5c3 None admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:40:41.965777 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 112/226] 10.222.0.30 () {56 vars in 1111 bytes} [Mon May 23 12:40:41 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 188 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) May 23 12:40:41.966637 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be9eed1d-8119-4fc2-bb6a-ad431b2ee5c3 None admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:40:41.982887 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f5fca73-f983-4af0-9237-61e998f48ee8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:40:41.983934 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f5fca73-f983-4af0-9237-61e998f48ee8 None admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:40:41.984468 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f5fca73-f983-4af0-9237-61e998f48ee8 None admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:40:41.984952 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f5fca73-f983-4af0-9237-61e998f48ee8 None admin] PATH_INFO: `/v3/roles` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:40:41.985982 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f5fca73-f983-4af0-9237-61e998f48ee8 None admin] RBAC: Adding query filter params (name=admin) {{(pid=92057) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} May 23 12:40:41.986514 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f5fca73-f983-4af0-9237-61e998f48ee8 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:40:41.988143 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f5fca73-f983-4af0-9237-61e998f48ee8 None admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:40:41.998504 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 115/227] 10.222.0.30 () {56 vars in 1099 bytes} [Mon May 23 12:40:41 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 188 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) May 23 12:40:42.004117 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f5fca73-f983-4af0-9237-61e998f48ee8 None admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:40:42.078826 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be9eed1d-8119-4fc2-bb6a-ad431b2ee5c3 None admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:40:42.095669 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=100 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:40:42.113013 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7da364b-8453-4282-9d00-a0bd70ce9560 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:40:42.113675 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7da364b-8453-4282-9d00-a0bd70ce9560 None admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:40:42.113995 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7da364b-8453-4282-9d00-a0bd70ce9560 None admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:40:42.114266 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7da364b-8453-4282-9d00-a0bd70ce9560 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:40:42.115095 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7da364b-8453-4282-9d00-a0bd70ce9560 None admin] RBAC: Adding query filter params (role.id=80fb22507c9a4ff09d44d8f12f6fff1a, scope.project.id=b39742b1d1b34dfe82a9ce4da39b7582, user.id=84f86330620a493684d2835681c966eb) {{(pid=92056) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} May 23 12:40:42.115406 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7da364b-8453-4282-9d00-a0bd70ce9560 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:40:42.116849 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7da364b-8453-4282-9d00-a0bd70ce9560 None admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:40:42.123151 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 113/228] 10.222.0.30 () {56 vars in 1374 bytes} [Mon May 23 12:40:41 2022] GET /identity/v3/role_assignments?user.id=84f86330620a493684d2835681c966eb&scope.project.id=b39742b1d1b34dfe82a9ce4da39b7582&role.id=80fb22507c9a4ff09d44d8f12f6fff1a => generated 259 bytes in 157 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) May 23 12:40:42.123819 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7da364b-8453-4282-9d00-a0bd70ce9560 None admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:40:42.128457 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f5fca73-f983-4af0-9237-61e998f48ee8 None admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:40:42.172922 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5297f858-7c6b-4d39-b35a-3c1298d19f0f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:40:42.174269 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5297f858-7c6b-4d39-b35a-3c1298d19f0f None admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:40:42.174814 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5297f858-7c6b-4d39-b35a-3c1298d19f0f None admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:40:42.175318 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5297f858-7c6b-4d39-b35a-3c1298d19f0f None admin] PATH_INFO: `/v3/users/237cf4210ddf4163a2353c03fdf231a9` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:40:42.176997 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5297f858-7c6b-4d39-b35a-3c1298d19f0f None admin] RBAC: Authorizing `identity:get_user(user_id=237cf4210ddf4163a2353c03fdf231a9)` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:40:42.178928 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5297f858-7c6b-4d39-b35a-3c1298d19f0f None admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:40:42.181044 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 116/229] 10.222.0.30 () {56 vars in 1177 bytes} [Mon May 23 12:40:42 2022] GET /identity/v3/users/237cf4210ddf4163a2353c03fdf231a9 => generated 283 bytes in 177 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) May 23 12:40:42.191063 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5297f858-7c6b-4d39-b35a-3c1298d19f0f None admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:40:42.220909 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=100 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:40:42.236456 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7da364b-8453-4282-9d00-a0bd70ce9560 None admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:40:42.274933 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d6bcedc-f088-4141-bb83-78fb6edf4d56 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:40:42.276946 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d6bcedc-f088-4141-bb83-78fb6edf4d56 None admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:40:42.276946 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d6bcedc-f088-4141-bb83-78fb6edf4d56 None admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:40:42.276946 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d6bcedc-f088-4141-bb83-78fb6edf4d56 None admin] PATH_INFO: `/v3/users/237cf4210ddf4163a2353c03fdf231a9` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:40:42.277936 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d6bcedc-f088-4141-bb83-78fb6edf4d56 None admin] RBAC: Authorizing `identity:get_user(user_id=237cf4210ddf4163a2353c03fdf231a9)` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:40:42.279779 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d6bcedc-f088-4141-bb83-78fb6edf4d56 None admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:40:42.285909 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 114/230] 10.222.0.30 () {56 vars in 1177 bytes} [Mon May 23 12:40:42 2022] GET /identity/v3/users/237cf4210ddf4163a2353c03fdf231a9 => generated 283 bytes in 158 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) May 23 12:40:42.298500 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d6bcedc-f088-4141-bb83-78fb6edf4d56 None admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:40:42.362647 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=100 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:40:42.376834 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5297f858-7c6b-4d39-b35a-3c1298d19f0f None admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:40:42.424675 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d6bcedc-f088-4141-bb83-78fb6edf4d56 None admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:40:42.454084 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d3dba76-2c5a-451e-947e-2572bc53687d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:40:42.454962 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d3dba76-2c5a-451e-947e-2572bc53687d None admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:40:42.455242 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d3dba76-2c5a-451e-947e-2572bc53687d None admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:40:42.455518 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d3dba76-2c5a-451e-947e-2572bc53687d None admin] PATH_INFO: `/v3/projects/b39742b1d1b34dfe82a9ce4da39b7582` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:40:42.457732 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d3dba76-2c5a-451e-947e-2572bc53687d None admin] RBAC: Authorizing `identity:get_project(project_id=b39742b1d1b34dfe82a9ce4da39b7582)` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:40:42.459222 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d3dba76-2c5a-451e-947e-2572bc53687d None admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:40:42.468131 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 117/231] 10.222.0.30 () {56 vars in 1186 bytes} [Mon May 23 12:40:42 2022] GET /identity/v3/projects/b39742b1d1b34dfe82a9ce4da39b7582 => generated 302 bytes in 276 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) May 23 12:40:42.472458 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d3dba76-2c5a-451e-947e-2572bc53687d None admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:40:42.495649 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6c96575-5077-4cbb-a339-dc17bdf16254 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:40:42.496425 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6c96575-5077-4cbb-a339-dc17bdf16254 None admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:40:42.496705 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6c96575-5077-4cbb-a339-dc17bdf16254 None admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:40:42.496980 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6c96575-5077-4cbb-a339-dc17bdf16254 None admin] PATH_INFO: `/v3/projects/b39742b1d1b34dfe82a9ce4da39b7582` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:40:42.499444 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6c96575-5077-4cbb-a339-dc17bdf16254 None admin] RBAC: Authorizing `identity:get_project(project_id=b39742b1d1b34dfe82a9ce4da39b7582)` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:40:42.500910 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6c96575-5077-4cbb-a339-dc17bdf16254 None admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:40:42.506054 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 115/232] 10.222.0.30 () {56 vars in 1186 bytes} [Mon May 23 12:40:42 2022] GET /identity/v3/projects/b39742b1d1b34dfe82a9ce4da39b7582 => generated 302 bytes in 208 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) May 23 12:40:42.509004 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6c96575-5077-4cbb-a339-dc17bdf16254 None admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:40:42.582149 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=100 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:40:42.603754 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d3dba76-2c5a-451e-947e-2572bc53687d None admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:40:42.651972 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-268ffd6c-8df2-4914-8761-6137fb94e64a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:40:42.652541 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-268ffd6c-8df2-4914-8761-6137fb94e64a None admin] REQUEST_METHOD: `PUT` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:40:42.652820 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-268ffd6c-8df2-4914-8761-6137fb94e64a None admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:40:42.653088 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-268ffd6c-8df2-4914-8761-6137fb94e64a None admin] PATH_INFO: `/v3/projects/b39742b1d1b34dfe82a9ce4da39b7582/users/237cf4210ddf4163a2353c03fdf231a9/roles/e3b90948645643e888620ab1e5c774bf` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:40:42.660562 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-268ffd6c-8df2-4914-8761-6137fb94e64a None admin] RBAC: Authorizing `identity:create_grant(project_id=b39742b1d1b34dfe82a9ce4da39b7582, user_id=237cf4210ddf4163a2353c03fdf231a9, role_id=e3b90948645643e888620ab1e5c774bf)` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:40:42.662082 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-268ffd6c-8df2-4914-8761-6137fb94e64a None admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:40:42.691000 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6c96575-5077-4cbb-a339-dc17bdf16254 None admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:40:42.708037 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=100 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:40:42.725930 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20c1004a-fb34-4e57-bc73-a129d8549e5e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:40:42.726753 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20c1004a-fb34-4e57-bc73-a129d8549e5e None admin] REQUEST_METHOD: `PUT` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:40:42.727066 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20c1004a-fb34-4e57-bc73-a129d8549e5e None admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:40:42.727340 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20c1004a-fb34-4e57-bc73-a129d8549e5e None admin] PATH_INFO: `/v3/projects/b39742b1d1b34dfe82a9ce4da39b7582/users/237cf4210ddf4163a2353c03fdf231a9/roles/bc3c3c9d654940cbaa5177a6be23d1bc` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:40:42.730075 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20c1004a-fb34-4e57-bc73-a129d8549e5e None admin] RBAC: Authorizing `identity:create_grant(project_id=b39742b1d1b34dfe82a9ce4da39b7582, user_id=237cf4210ddf4163a2353c03fdf231a9, role_id=bc3c3c9d654940cbaa5177a6be23d1bc)` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:40:42.731515 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20c1004a-fb34-4e57-bc73-a129d8549e5e None admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:40:42.747994 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 118/233] 10.222.0.30 () {58 vars in 1439 bytes} [Mon May 23 12:40:42 2022] PUT /identity/v3/projects/b39742b1d1b34dfe82a9ce4da39b7582/users/237cf4210ddf4163a2353c03fdf231a9/roles/e3b90948645643e888620ab1e5c774bf => generated 0 bytes in 276 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) May 23 12:40:42.832293 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 116/234] 10.222.0.30 () {58 vars in 1439 bytes} [Mon May 23 12:40:42 2022] PUT /identity/v3/projects/b39742b1d1b34dfe82a9ce4da39b7582/users/237cf4210ddf4163a2353c03fdf231a9/roles/bc3c3c9d654940cbaa5177a6be23d1bc => generated 0 bytes in 324 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) May 23 12:40:43.561837 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3883bf4f-b13e-4937-8091-c73f13c10755 None None] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:40:43.562381 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3883bf4f-b13e-4937-8091-c73f13c10755 None None] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:40:43.562458 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3883bf4f-b13e-4937-8091-c73f13c10755 None None] PATH_INFO: `/` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:40:43.563950 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 119/235] 10.222.0.30 () {54 vars in 926 bytes} [Mon May 23 12:40:43 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) May 23 12:40:43.569357 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4eb0c55c-695b-4aa2-b2dc-636d6083fda4 None None] REQUEST_METHOD: `POST` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:40:43.569744 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4eb0c55c-695b-4aa2-b2dc-636d6083fda4 None None] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:40:43.569928 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4eb0c55c-695b-4aa2-b2dc-636d6083fda4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:40:43.594343 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.auth.core [None req-4eb0c55c-695b-4aa2-b2dc-636d6083fda4 None None] MFA Rules not processed for user `0f7119f8fa5c4445977671140ded91a4`. Rule list: `[]` (Enabled: `True`). {{(pid=92056) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 12:40:43.595548 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.fernet_utils [None req-4eb0c55c-695b-4aa2-b2dc-636d6083fda4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92056) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 12:40:43.689903 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 117/236] 10.222.0.30 () {58 vars in 1023 bytes} [Mon May 23 12:40:43 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 116 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) May 23 12:40:43.731466 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3883bf4f-b13e-4937-8091-c73f13c10755 None None] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:40:43.819517 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4dc77310-4818-41ec-ba68-4c646bd38745 None None] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:40:43.819755 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4dc77310-4818-41ec-ba68-4c646bd38745 None None] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:40:43.819998 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4dc77310-4818-41ec-ba68-4c646bd38745 None None] PATH_INFO: `/` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:40:43.821368 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 118/237] 10.222.0.30 () {54 vars in 926 bytes} [Mon May 23 12:40:43 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) May 23 12:40:43.826675 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27875710-4307-482b-9f20-2a5f394eea11 None None] REQUEST_METHOD: `POST` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:40:43.826985 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27875710-4307-482b-9f20-2a5f394eea11 None None] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:40:43.827344 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27875710-4307-482b-9f20-2a5f394eea11 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:40:43.848566 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.auth.core [None req-27875710-4307-482b-9f20-2a5f394eea11 None None] MFA Rules not processed for user `0f7119f8fa5c4445977671140ded91a4`. Rule list: `[]` (Enabled: `True`). {{(pid=92056) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 12:40:43.849650 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.fernet_utils [None req-27875710-4307-482b-9f20-2a5f394eea11 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92056) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 12:40:43.878956 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3883bf4f-b13e-4937-8091-c73f13c10755 None None] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:40:43.944470 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b396c98-cf1e-4333-ab8b-58f333c9cc72 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:40:43.945541 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b396c98-cf1e-4333-ab8b-58f333c9cc72 None admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:40:43.946068 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b396c98-cf1e-4333-ab8b-58f333c9cc72 None admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:40:43.946486 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b396c98-cf1e-4333-ab8b-58f333c9cc72 None admin] PATH_INFO: `/v3/roles/reader` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:40:43.951904 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 119/238] 10.222.0.30 () {58 vars in 1023 bytes} [Mon May 23 12:40:43 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 126 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) May 23 12:40:43.960004 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b396c98-cf1e-4333-ab8b-58f333c9cc72 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} May 23 12:40:43.974091 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:40:43.975609 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application [None req-0b396c98-cf1e-4333-ab8b-58f333c9cc72 None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. May 23 12:40:43.975609 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application Traceback (most recent call last): May 23 12:40:43.975609 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request May 23 12:40:43.975609 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application rv = self.dispatch_request() May 23 12:40:43.975609 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request May 23 12:40:43.975609 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) May 23 12:40:43.975609 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper May 23 12:40:43.975609 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) May 23 12:40:43.975609 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view May 23 12:40:43.975609 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) May 23 12:40:43.975609 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request May 23 12:40:43.975609 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) May 23 12:40:43.975609 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get May 23 12:40:43.975609 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self._get_role(role_id) May 23 12:40:43.975609 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role May 23 12:40:43.975609 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') May 23 12:40:43.976713 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call May 23 12:40:43.976713 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( May 23 12:40:43.976713 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data May 23 12:40:43.976713 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) May 23 12:40:43.976713 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped May 23 12:40:43.976713 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) May 23 12:40:43.976713 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun May 23 12:40:43.976713 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) May 23 12:40:43.976713 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func May 23 12:40:43.976713 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self.get_or_create( May 23 12:40:43.976713 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create May 23 12:40:43.976713 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application with Lock( May 23 12:40:43.976713 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ May 23 12:40:43.976713 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self._enter() May 23 12:40:43.976713 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter May 23 12:40:43.976713 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) May 23 12:40:43.977713 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create May 23 12:40:43.977713 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self.creator() May 23 12:40:43.977713 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value May 23 12:40:43.977713 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application created_value = creator( May 23 12:40:43.977713 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role May 23 12:40:43.977713 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) May 23 12:40:43.977713 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role May 23 12:40:43.977713 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() May 23 12:40:43.977713 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role May 23 12:40:43.977713 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) May 23 12:40:43.977713 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. May 23 12:40:43.977713 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application May 23 12:40:43.980799 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 120/239] 10.222.0.30 () {56 vars in 1099 bytes} [Mon May 23 12:40:43 2022] GET /identity/v3/roles/reader => generated 84 bytes in 250 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) May 23 12:40:43.986682 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27875710-4307-482b-9f20-2a5f394eea11 None None] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:40:44.014049 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b396c98-cf1e-4333-ab8b-58f333c9cc72 None admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:40:44.046263 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:40:44.157710 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27875710-4307-482b-9f20-2a5f394eea11 None None] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:40:44.159992 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b396c98-cf1e-4333-ab8b-58f333c9cc72 None admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:40:44.193059 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c057fc2b-84b7-4306-a9ca-39188000f932 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:40:44.193730 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c057fc2b-84b7-4306-a9ca-39188000f932 None admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:40:44.194047 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c057fc2b-84b7-4306-a9ca-39188000f932 None admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:40:44.194325 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c057fc2b-84b7-4306-a9ca-39188000f932 None admin] PATH_INFO: `/v3/roles` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:40:44.195074 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c057fc2b-84b7-4306-a9ca-39188000f932 None admin] RBAC: Adding query filter params (name=reader) {{(pid=92056) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} May 23 12:40:44.195382 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c057fc2b-84b7-4306-a9ca-39188000f932 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:40:44.196820 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c057fc2b-84b7-4306-a9ca-39188000f932 None admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:40:44.206276 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 120/240] 10.222.0.30 () {56 vars in 1101 bytes} [Mon May 23 12:40:43 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 220 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) May 23 12:40:44.212899 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c057fc2b-84b7-4306-a9ca-39188000f932 None admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:40:44.237549 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ca5bcb7-d72c-4629-ace0-9add82171ab9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:40:44.238429 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ca5bcb7-d72c-4629-ace0-9add82171ab9 None admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:40:44.238715 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ca5bcb7-d72c-4629-ace0-9add82171ab9 None admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:40:44.239049 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ca5bcb7-d72c-4629-ace0-9add82171ab9 None admin] PATH_INFO: `/v3/domains/default` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:40:44.241400 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ca5bcb7-d72c-4629-ace0-9add82171ab9 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:40:44.243035 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ca5bcb7-d72c-4629-ace0-9add82171ab9 None admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:40:44.249942 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 121/241] 10.222.0.30 () {56 vars in 1108 bytes} [Mon May 23 12:40:44 2022] GET /identity/v3/domains/default => generated 199 bytes in 236 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) May 23 12:40:44.255618 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ca5bcb7-d72c-4629-ace0-9add82171ab9 None admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:40:44.360203 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=100 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:40:44.364139 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c057fc2b-84b7-4306-a9ca-39188000f932 None admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:40:44.364816 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=100 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:40:44.402689 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0eda6831-e55b-4440-b15d-b136856161e9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:40:44.403579 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0eda6831-e55b-4440-b15d-b136856161e9 None admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:40:44.403867 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0eda6831-e55b-4440-b15d-b136856161e9 None admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:40:44.404145 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0eda6831-e55b-4440-b15d-b136856161e9 None admin] PATH_INFO: `/v3/users/3f1c3ab2da314bff8c13f587a8d55a52` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:40:44.419994 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0eda6831-e55b-4440-b15d-b136856161e9 None admin] RBAC: Authorizing `identity:get_user(user_id=3f1c3ab2da314bff8c13f587a8d55a52)` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:40:44.421401 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0eda6831-e55b-4440-b15d-b136856161e9 None admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:40:44.423236 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 121/242] 10.222.0.30 () {56 vars in 1177 bytes} [Mon May 23 12:40:44 2022] GET /identity/v3/users/3f1c3ab2da314bff8c13f587a8d55a52 => generated 297 bytes in 211 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) May 23 12:40:44.430781 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0eda6831-e55b-4440-b15d-b136856161e9 None admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:40:44.439606 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ca5bcb7-d72c-4629-ace0-9add82171ab9 None admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:40:44.530579 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad550bce-b2b0-4f30-98b3-1af4c4b3ed87 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:40:44.531636 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad550bce-b2b0-4f30-98b3-1af4c4b3ed87 None admin] REQUEST_METHOD: `POST` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:40:44.532144 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad550bce-b2b0-4f30-98b3-1af4c4b3ed87 None admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:40:44.532635 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad550bce-b2b0-4f30-98b3-1af4c4b3ed87 None admin] PATH_INFO: `/v3/users` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:40:44.533549 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad550bce-b2b0-4f30-98b3-1af4c4b3ed87 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:40:44.535312 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad550bce-b2b0-4f30-98b3-1af4c4b3ed87 None admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:40:44.541678 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0eda6831-e55b-4440-b15d-b136856161e9 None admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:40:44.585193 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96cad9ef-64c7-4ace-939c-8327c6b6da9e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:40:44.585971 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96cad9ef-64c7-4ace-939c-8327c6b6da9e None admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:40:44.586165 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96cad9ef-64c7-4ace-939c-8327c6b6da9e None admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:40:44.586438 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96cad9ef-64c7-4ace-939c-8327c6b6da9e None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:40:44.614232 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=100 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:40:44.644828 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 122/243] 10.222.0.30 () {60 vars in 1132 bytes} [Mon May 23 12:40:44 2022] POST /identity/v3/users => generated 293 bytes in 389 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) May 23 12:40:44.667970 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96cad9ef-64c7-4ace-939c-8327c6b6da9e None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:40:44.669449 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96cad9ef-64c7-4ace-939c-8327c6b6da9e None admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:40:44.786919 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 122/244] 10.222.0.30 () {58 vars in 1282 bytes} [Mon May 23 12:40:44 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 357 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) May 23 12:40:44.793067 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad550bce-b2b0-4f30-98b3-1af4c4b3ed87 None admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:40:44.855512 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25cff545-d2dc-4be2-8c2d-f93a1789589a None None] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:40:44.855826 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25cff545-d2dc-4be2-8c2d-f93a1789589a None None] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:40:44.856066 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25cff545-d2dc-4be2-8c2d-f93a1789589a None None] PATH_INFO: `/` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:40:44.857646 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 123/245] 10.222.0.30 () {54 vars in 926 bytes} [Mon May 23 12:40:44 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) May 23 12:40:44.866522 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd0db0e7-1883-40e0-aa3e-953149686223 None None] REQUEST_METHOD: `POST` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:40:44.866856 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd0db0e7-1883-40e0-aa3e-953149686223 None None] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:40:44.867072 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd0db0e7-1883-40e0-aa3e-953149686223 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:40:44.898320 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.auth.core [None req-dd0db0e7-1883-40e0-aa3e-953149686223 None None] MFA Rules not processed for user `0f7119f8fa5c4445977671140ded91a4`. Rule list: `[]` (Enabled: `True`). {{(pid=92056) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 12:40:44.899528 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.fernet_utils [None req-dd0db0e7-1883-40e0-aa3e-953149686223 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92056) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 12:40:44.970426 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=100 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:40:44.986080 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad550bce-b2b0-4f30-98b3-1af4c4b3ed87 None admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:40:44.994734 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 124/246] 10.222.0.30 () {58 vars in 1023 bytes} [Mon May 23 12:40:44 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 123 msecs (HTTP/1.1 201) 6 headers in 363 bytes (3 switches on core 0) May 23 12:40:45.012033 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd0db0e7-1883-40e0-aa3e-953149686223 None None] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:40:45.013505 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:40:45.027849 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce7813c7-6239-452c-bc61-1ae3bf3d4973 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:40:45.029815 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce7813c7-6239-452c-bc61-1ae3bf3d4973 None admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:40:45.030278 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce7813c7-6239-452c-bc61-1ae3bf3d4973 None admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:40:45.030951 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce7813c7-6239-452c-bc61-1ae3bf3d4973 None admin] PATH_INFO: `/v3/projects/b39742b1d1b34dfe82a9ce4da39b7582` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:40:45.032497 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce7813c7-6239-452c-bc61-1ae3bf3d4973 None admin] RBAC: Authorizing `identity:get_project(project_id=b39742b1d1b34dfe82a9ce4da39b7582)` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:40:45.034548 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce7813c7-6239-452c-bc61-1ae3bf3d4973 None admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:40:45.037370 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 123/247] 10.222.0.30 () {56 vars in 1186 bytes} [Mon May 23 12:40:44 2022] GET /identity/v3/projects/b39742b1d1b34dfe82a9ce4da39b7582 => generated 302 bytes in 245 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) May 23 12:40:45.043580 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce7813c7-6239-452c-bc61-1ae3bf3d4973 None admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:40:45.126099 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd0db0e7-1883-40e0-aa3e-953149686223 None None] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:40:45.151316 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce7813c7-6239-452c-bc61-1ae3bf3d4973 None admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:40:45.165144 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f8f40b6-d2a4-4e7c-ae66-228b69c3afb2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:40:45.165980 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f8f40b6-d2a4-4e7c-ae66-228b69c3afb2 None admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:40:45.166294 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f8f40b6-d2a4-4e7c-ae66-228b69c3afb2 None admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:40:45.167114 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f8f40b6-d2a4-4e7c-ae66-228b69c3afb2 None admin] PATH_INFO: `/v3/roles/member` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:40:45.180500 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f8f40b6-d2a4-4e7c-ae66-228b69c3afb2 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} May 23 12:40:45.191214 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application [None req-0f8f40b6-d2a4-4e7c-ae66-228b69c3afb2 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. May 23 12:40:45.191214 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application Traceback (most recent call last): May 23 12:40:45.191214 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request May 23 12:40:45.191214 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application rv = self.dispatch_request() May 23 12:40:45.191214 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request May 23 12:40:45.191214 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) May 23 12:40:45.191214 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper May 23 12:40:45.191214 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) May 23 12:40:45.191214 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view May 23 12:40:45.191214 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) May 23 12:40:45.191214 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request May 23 12:40:45.191214 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) May 23 12:40:45.191214 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get May 23 12:40:45.191214 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self._get_role(role_id) May 23 12:40:45.191214 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role May 23 12:40:45.191214 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') May 23 12:40:45.192352 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call May 23 12:40:45.192352 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( May 23 12:40:45.192352 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data May 23 12:40:45.192352 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) May 23 12:40:45.192352 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped May 23 12:40:45.192352 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) May 23 12:40:45.192352 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun May 23 12:40:45.192352 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) May 23 12:40:45.192352 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func May 23 12:40:45.192352 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self.get_or_create( May 23 12:40:45.192352 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create May 23 12:40:45.192352 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application with Lock( May 23 12:40:45.192352 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ May 23 12:40:45.192352 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self._enter() May 23 12:40:45.192352 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter May 23 12:40:45.192352 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) May 23 12:40:45.193218 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create May 23 12:40:45.193218 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self.creator() May 23 12:40:45.193218 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value May 23 12:40:45.193218 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application created_value = creator( May 23 12:40:45.193218 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role May 23 12:40:45.193218 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) May 23 12:40:45.193218 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role May 23 12:40:45.193218 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() May 23 12:40:45.193218 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role May 23 12:40:45.193218 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) May 23 12:40:45.193218 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. May 23 12:40:45.193218 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application May 23 12:40:45.195160 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 125/248] 10.222.0.30 () {56 vars in 1099 bytes} [Mon May 23 12:40:45 2022] GET /identity/v3/roles/member => generated 84 bytes in 184 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) May 23 12:40:45.201291 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4d1e181-ac54-4a00-966f-1d7d3543718e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:40:45.202436 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4d1e181-ac54-4a00-966f-1d7d3543718e None admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:40:45.202851 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4d1e181-ac54-4a00-966f-1d7d3543718e None admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:40:45.203249 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4d1e181-ac54-4a00-966f-1d7d3543718e None admin] PATH_INFO: `/v3/role_assignments` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:40:45.204207 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4d1e181-ac54-4a00-966f-1d7d3543718e None admin] RBAC: Adding query filter params (role.id=c223286ed21a4654a0f2b4e5551831c7, scope.project.id=b39742b1d1b34dfe82a9ce4da39b7582, user.id=3f1c3ab2da314bff8c13f587a8d55a52) {{(pid=92057) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} May 23 12:40:45.204658 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4d1e181-ac54-4a00-966f-1d7d3543718e None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:40:45.205202 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f8f40b6-d2a4-4e7c-ae66-228b69c3afb2 None admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:40:45.206472 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4d1e181-ac54-4a00-966f-1d7d3543718e None admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:40:45.214969 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 124/249] 10.222.0.30 () {56 vars in 1374 bytes} [Mon May 23 12:40:45 2022] GET /identity/v3/role_assignments?user.id=3f1c3ab2da314bff8c13f587a8d55a52&scope.project.id=b39742b1d1b34dfe82a9ce4da39b7582&role.id=c223286ed21a4654a0f2b4e5551831c7 => generated 259 bytes in 172 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) May 23 12:40:45.268862 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=100 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:40:45.333179 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f8f40b6-d2a4-4e7c-ae66-228b69c3afb2 None admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:40:45.377091 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44aeff6d-a201-4bee-b144-3103c41509cb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:40:45.377774 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44aeff6d-a201-4bee-b144-3103c41509cb None admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:40:45.378135 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44aeff6d-a201-4bee-b144-3103c41509cb None admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:40:45.378419 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44aeff6d-a201-4bee-b144-3103c41509cb None admin] PATH_INFO: `/v3/roles` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:40:45.379187 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44aeff6d-a201-4bee-b144-3103c41509cb None admin] RBAC: Adding query filter params (name=member) {{(pid=92056) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} May 23 12:40:45.379496 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44aeff6d-a201-4bee-b144-3103c41509cb None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:40:45.380938 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44aeff6d-a201-4bee-b144-3103c41509cb None admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:40:45.392402 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 126/250] 10.222.0.30 () {56 vars in 1101 bytes} [Mon May 23 12:40:45 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 188 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) May 23 12:40:45.398751 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4d1e181-ac54-4a00-966f-1d7d3543718e None admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:40:45.474004 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=100 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:40:45.519087 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4d1e181-ac54-4a00-966f-1d7d3543718e None admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:40:45.555513 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b1acbd9-5913-4b32-a9f1-5add3b3f1895 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:40:45.556387 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b1acbd9-5913-4b32-a9f1-5add3b3f1895 None admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:40:45.556672 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b1acbd9-5913-4b32-a9f1-5add3b3f1895 None admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:40:45.557049 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b1acbd9-5913-4b32-a9f1-5add3b3f1895 None admin] PATH_INFO: `/v3/users/84f86330620a493684d2835681c966eb` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:40:45.558527 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b1acbd9-5913-4b32-a9f1-5add3b3f1895 None admin] RBAC: Authorizing `identity:get_user(user_id=84f86330620a493684d2835681c966eb)` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:40:45.560048 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b1acbd9-5913-4b32-a9f1-5add3b3f1895 None admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:40:45.564535 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acdfc6fe-042e-4f0d-a5d5-fa7582ac8683 None None] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:40:45.565070 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acdfc6fe-042e-4f0d-a5d5-fa7582ac8683 None None] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:40:45.565551 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acdfc6fe-042e-4f0d-a5d5-fa7582ac8683 None None] PATH_INFO: `/` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:40:45.565979 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 125/251] 10.222.0.30 () {56 vars in 1177 bytes} [Mon May 23 12:40:45 2022] GET /identity/v3/users/84f86330620a493684d2835681c966eb => generated 297 bytes in 164 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) May 23 12:40:45.567390 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b1acbd9-5913-4b32-a9f1-5add3b3f1895 None admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:40:45.567815 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 127/252] 10.222.0.30 () {54 vars in 926 bytes} [Mon May 23 12:40:45 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) May 23 12:40:45.579560 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9b72e73-3104-4688-a916-3ca72b322e9d None None] REQUEST_METHOD: `POST` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:40:45.580155 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9b72e73-3104-4688-a916-3ca72b322e9d None None] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:40:45.580720 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9b72e73-3104-4688-a916-3ca72b322e9d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:40:45.618283 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.auth.core [None req-c9b72e73-3104-4688-a916-3ca72b322e9d None None] MFA Rules not processed for user `0f7119f8fa5c4445977671140ded91a4`. Rule list: `[]` (Enabled: `True`). {{(pid=92056) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 12:40:45.620475 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.fernet_utils [None req-c9b72e73-3104-4688-a916-3ca72b322e9d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92056) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 12:40:45.675823 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b1acbd9-5913-4b32-a9f1-5add3b3f1895 None admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:40:45.710331 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=100 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:40:45.721731 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-885d6af0-8451-4ef4-860c-9a172354b42a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:40:45.729230 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-885d6af0-8451-4ef4-860c-9a172354b42a None admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:40:45.729230 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-885d6af0-8451-4ef4-860c-9a172354b42a None admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:40:45.729230 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-885d6af0-8451-4ef4-860c-9a172354b42a None admin] PATH_INFO: `/v3/projects/b39742b1d1b34dfe82a9ce4da39b7582` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:40:45.729230 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-885d6af0-8451-4ef4-860c-9a172354b42a None admin] RBAC: Authorizing `identity:get_project(project_id=b39742b1d1b34dfe82a9ce4da39b7582)` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:40:45.729230 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-885d6af0-8451-4ef4-860c-9a172354b42a None admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:40:45.729230 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 126/253] 10.222.0.30 () {56 vars in 1186 bytes} [Mon May 23 12:40:45 2022] GET /identity/v3/projects/b39742b1d1b34dfe82a9ce4da39b7582 => generated 302 bytes in 161 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) May 23 12:40:45.734252 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-885d6af0-8451-4ef4-860c-9a172354b42a None admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:40:45.735594 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 128/254] 10.222.0.30 () {58 vars in 1023 bytes} [Mon May 23 12:40:45 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 157 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) May 23 12:40:45.771297 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9b72e73-3104-4688-a916-3ca72b322e9d None None] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:40:45.853007 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-885d6af0-8451-4ef4-860c-9a172354b42a None admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:40:45.866286 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=100 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:40:45.917056 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dbeaa099-6d65-4acb-9557-749b926ff168 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:40:45.918156 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbeaa099-6d65-4acb-9557-749b926ff168 None admin] REQUEST_METHOD: `PUT` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:40:45.918436 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbeaa099-6d65-4acb-9557-749b926ff168 None admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:40:45.918712 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbeaa099-6d65-4acb-9557-749b926ff168 None admin] PATH_INFO: `/v3/projects/b39742b1d1b34dfe82a9ce4da39b7582/users/84f86330620a493684d2835681c966eb/roles/80fb22507c9a4ff09d44d8f12f6fff1a` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:40:45.925155 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dbeaa099-6d65-4acb-9557-749b926ff168 None admin] RBAC: Authorizing `identity:create_grant(project_id=b39742b1d1b34dfe82a9ce4da39b7582, user_id=84f86330620a493684d2835681c966eb, role_id=80fb22507c9a4ff09d44d8f12f6fff1a)` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:40:45.926650 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dbeaa099-6d65-4acb-9557-749b926ff168 None admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:40:45.972222 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9b72e73-3104-4688-a916-3ca72b322e9d None None] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:40:46.016414 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 127/255] 10.222.0.30 () {58 vars in 1439 bytes} [Mon May 23 12:40:45 2022] PUT /identity/v3/projects/b39742b1d1b34dfe82a9ce4da39b7582/users/84f86330620a493684d2835681c966eb/roles/80fb22507c9a4ff09d44d8f12f6fff1a => generated 0 bytes in 283 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) May 23 12:40:46.016937 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19109852-0797-4e65-baba-365ddf08491e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:40:46.017740 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ffd6e7c-d416-49fc-b9a9-1c4146b96fde None None] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:40:46.017848 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19109852-0797-4e65-baba-365ddf08491e None admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:40:46.018054 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19109852-0797-4e65-baba-365ddf08491e None admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:40:46.018150 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ffd6e7c-d416-49fc-b9a9-1c4146b96fde None None] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:40:46.018326 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ffd6e7c-d416-49fc-b9a9-1c4146b96fde None None] PATH_INFO: `/` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:40:46.018560 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19109852-0797-4e65-baba-365ddf08491e None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:40:46.019597 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 128/256] 10.222.0.30 () {54 vars in 926 bytes} [Mon May 23 12:40:46 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) May 23 12:40:46.026458 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd8cbeb2-34da-4072-b9f0-c51cdf925c61 None None] REQUEST_METHOD: `POST` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:40:46.026787 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd8cbeb2-34da-4072-b9f0-c51cdf925c61 None None] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:40:46.027076 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd8cbeb2-34da-4072-b9f0-c51cdf925c61 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:40:46.031681 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19109852-0797-4e65-baba-365ddf08491e None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} May 23 12:40:46.040104 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application [None req-19109852-0797-4e65-baba-365ddf08491e None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. May 23 12:40:46.040104 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application Traceback (most recent call last): May 23 12:40:46.040104 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request May 23 12:40:46.040104 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application rv = self.dispatch_request() May 23 12:40:46.040104 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request May 23 12:40:46.040104 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) May 23 12:40:46.040104 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper May 23 12:40:46.040104 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) May 23 12:40:46.040104 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view May 23 12:40:46.040104 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) May 23 12:40:46.040104 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request May 23 12:40:46.040104 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) May 23 12:40:46.040104 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get May 23 12:40:46.040104 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self._get_role(role_id) May 23 12:40:46.040104 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role May 23 12:40:46.040104 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') May 23 12:40:46.041136 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call May 23 12:40:46.041136 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( May 23 12:40:46.041136 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data May 23 12:40:46.041136 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) May 23 12:40:46.041136 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped May 23 12:40:46.041136 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) May 23 12:40:46.041136 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun May 23 12:40:46.041136 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) May 23 12:40:46.041136 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func May 23 12:40:46.041136 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self.get_or_create( May 23 12:40:46.041136 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create May 23 12:40:46.041136 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application with Lock( May 23 12:40:46.041136 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ May 23 12:40:46.041136 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self._enter() May 23 12:40:46.041136 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter May 23 12:40:46.041136 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) May 23 12:40:46.043276 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create May 23 12:40:46.043276 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self.creator() May 23 12:40:46.043276 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value May 23 12:40:46.043276 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application created_value = creator( May 23 12:40:46.043276 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role May 23 12:40:46.043276 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) May 23 12:40:46.043276 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role May 23 12:40:46.043276 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() May 23 12:40:46.043276 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role May 23 12:40:46.043276 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) May 23 12:40:46.043276 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. May 23 12:40:46.043276 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application May 23 12:40:46.043276 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 129/257] 10.222.0.30 () {56 vars in 1114 bytes} [Mon May 23 12:40:45 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 272 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) May 23 12:40:46.049384 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19109852-0797-4e65-baba-365ddf08491e None admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:40:46.071146 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.auth.core [None req-cd8cbeb2-34da-4072-b9f0-c51cdf925c61 None None] MFA Rules not processed for user `0f7119f8fa5c4445977671140ded91a4`. Rule list: `[]` (Enabled: `True`). {{(pid=92057) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 12:40:46.072359 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.fernet_utils [None req-cd8cbeb2-34da-4072-b9f0-c51cdf925c61 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92057) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 12:40:46.167902 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=100 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:40:46.174695 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19109852-0797-4e65-baba-365ddf08491e None admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:40:46.216133 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-554c86d0-f02e-4ee0-9e1d-658fc8752bc3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:40:46.216817 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-554c86d0-f02e-4ee0-9e1d-658fc8752bc3 None admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:40:46.217150 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-554c86d0-f02e-4ee0-9e1d-658fc8752bc3 None admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:40:46.217433 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-554c86d0-f02e-4ee0-9e1d-658fc8752bc3 None admin] PATH_INFO: `/v3/roles` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:40:46.218381 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-554c86d0-f02e-4ee0-9e1d-658fc8752bc3 None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=92056) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} May 23 12:40:46.218699 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-554c86d0-f02e-4ee0-9e1d-658fc8752bc3 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:40:46.220126 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-554c86d0-f02e-4ee0-9e1d-658fc8752bc3 None admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:40:46.227480 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:40:46.233281 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 130/258] 10.222.0.30 () {56 vars in 1111 bytes} [Mon May 23 12:40:46 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 185 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) May 23 12:40:46.239115 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-554c86d0-f02e-4ee0-9e1d-658fc8752bc3 None admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:40:46.323851 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 129/259] 10.222.0.30 () {58 vars in 1023 bytes} [Mon May 23 12:40:46 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 299 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) May 23 12:40:46.356813 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd8cbeb2-34da-4072-b9f0-c51cdf925c61 None None] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:40:46.363753 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-554c86d0-f02e-4ee0-9e1d-658fc8752bc3 None admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:40:46.401895 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79855122-497c-4ae6-8f8a-8338f1fc6e82 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:40:46.403420 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79855122-497c-4ae6-8f8a-8338f1fc6e82 None admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:40:46.403846 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79855122-497c-4ae6-8f8a-8338f1fc6e82 None admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:40:46.404568 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79855122-497c-4ae6-8f8a-8338f1fc6e82 None admin] PATH_INFO: `/v3/users/237cf4210ddf4163a2353c03fdf231a9` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:40:46.406132 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79855122-497c-4ae6-8f8a-8338f1fc6e82 None admin] RBAC: Authorizing `identity:get_user(user_id=237cf4210ddf4163a2353c03fdf231a9)` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:40:46.407935 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79855122-497c-4ae6-8f8a-8338f1fc6e82 None admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:40:46.410044 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 131/260] 10.222.0.30 () {56 vars in 1177 bytes} [Mon May 23 12:40:46 2022] GET /identity/v3/users/237cf4210ddf4163a2353c03fdf231a9 => generated 283 bytes in 171 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) May 23 12:40:46.416043 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79855122-497c-4ae6-8f8a-8338f1fc6e82 None admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:40:46.462332 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=100 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:40:46.524949 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79855122-497c-4ae6-8f8a-8338f1fc6e82 None admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:40:46.564714 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da4d6c77-8cf6-4e3a-8ad8-5c4b29615d12 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:40:46.565649 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da4d6c77-8cf6-4e3a-8ad8-5c4b29615d12 None admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:40:46.566188 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da4d6c77-8cf6-4e3a-8ad8-5c4b29615d12 None admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:40:46.566724 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da4d6c77-8cf6-4e3a-8ad8-5c4b29615d12 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:40:46.643649 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da4d6c77-8cf6-4e3a-8ad8-5c4b29615d12 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:40:46.645787 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da4d6c77-8cf6-4e3a-8ad8-5c4b29615d12 None admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:40:46.682793 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=100 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:40:46.691982 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd8cbeb2-34da-4072-b9f0-c51cdf925c61 None None] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:40:46.772902 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-acc4ca8e-38bc-46f3-87d3-765a04fe6af0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:40:46.773716 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acc4ca8e-38bc-46f3-87d3-765a04fe6af0 None admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:40:46.774030 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acc4ca8e-38bc-46f3-87d3-765a04fe6af0 None admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:40:46.774306 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acc4ca8e-38bc-46f3-87d3-765a04fe6af0 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:40:46.798972 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-acc4ca8e-38bc-46f3-87d3-765a04fe6af0 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} May 23 12:40:46.806749 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application [None req-acc4ca8e-38bc-46f3-87d3-765a04fe6af0 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. May 23 12:40:46.806749 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application Traceback (most recent call last): May 23 12:40:46.806749 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request May 23 12:40:46.806749 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application rv = self.dispatch_request() May 23 12:40:46.806749 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request May 23 12:40:46.806749 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) May 23 12:40:46.806749 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper May 23 12:40:46.806749 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) May 23 12:40:46.806749 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view May 23 12:40:46.806749 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) May 23 12:40:46.806749 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request May 23 12:40:46.806749 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) May 23 12:40:46.806749 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get May 23 12:40:46.806749 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self._get_role(role_id) May 23 12:40:46.806749 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role May 23 12:40:46.806749 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') May 23 12:40:46.808395 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call May 23 12:40:46.808395 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( May 23 12:40:46.808395 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data May 23 12:40:46.808395 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) May 23 12:40:46.808395 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped May 23 12:40:46.808395 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) May 23 12:40:46.808395 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun May 23 12:40:46.808395 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) May 23 12:40:46.808395 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func May 23 12:40:46.808395 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self.get_or_create( May 23 12:40:46.808395 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create May 23 12:40:46.808395 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application with Lock( May 23 12:40:46.808395 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ May 23 12:40:46.808395 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self._enter() May 23 12:40:46.808395 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter May 23 12:40:46.808395 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) May 23 12:40:46.809220 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create May 23 12:40:46.809220 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self.creator() May 23 12:40:46.809220 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value May 23 12:40:46.809220 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application created_value = creator( May 23 12:40:46.809220 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role May 23 12:40:46.809220 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) May 23 12:40:46.809220 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role May 23 12:40:46.809220 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() May 23 12:40:46.809220 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role May 23 12:40:46.809220 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) May 23 12:40:46.809220 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. May 23 12:40:46.809220 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application May 23 12:40:46.816267 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 132/261] 10.222.0.30 () {58 vars in 1282 bytes} [Mon May 23 12:40:46 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 401 msecs (HTTP/1.1 200) 6 headers in 358 bytes (2 switches on core 0) May 23 12:40:46.817505 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 130/262] 10.222.0.30 () {56 vars in 1096 bytes} [Mon May 23 12:40:46 2022] GET /identity/v3/roles/admin => generated 83 bytes in 461 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) May 23 12:40:46.823398 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da4d6c77-8cf6-4e3a-8ad8-5c4b29615d12 None admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:40:46.826345 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-acc4ca8e-38bc-46f3-87d3-765a04fe6af0 None admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:40:46.918531 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=100 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:40:46.962020 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da4d6c77-8cf6-4e3a-8ad8-5c4b29615d12 None admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:40:46.991032 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=100 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:40:46.993372 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9b66c4e-a36e-4748-8f4c-67e2f4101afc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:40:46.994773 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9b66c4e-a36e-4748-8f4c-67e2f4101afc None admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:40:46.995071 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9b66c4e-a36e-4748-8f4c-67e2f4101afc None admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:40:46.995387 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9b66c4e-a36e-4748-8f4c-67e2f4101afc None admin] PATH_INFO: `/v3/projects/b39742b1d1b34dfe82a9ce4da39b7582` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:40:46.996791 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9b66c4e-a36e-4748-8f4c-67e2f4101afc None admin] RBAC: Authorizing `identity:get_project(project_id=b39742b1d1b34dfe82a9ce4da39b7582)` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:40:46.998360 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9b66c4e-a36e-4748-8f4c-67e2f4101afc None admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:40:47.000604 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 133/263] 10.222.0.30 () {56 vars in 1186 bytes} [Mon May 23 12:40:46 2022] GET /identity/v3/projects/b39742b1d1b34dfe82a9ce4da39b7582 => generated 302 bytes in 178 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) May 23 12:40:47.006222 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9b66c4e-a36e-4748-8f4c-67e2f4101afc None admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:40:47.008034 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-acc4ca8e-38bc-46f3-87d3-765a04fe6af0 None admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:40:47.051906 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-522a6079-e782-4f1d-a5aa-3c60f4ecc0f0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:40:47.052585 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-522a6079-e782-4f1d-a5aa-3c60f4ecc0f0 None admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:40:47.052869 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-522a6079-e782-4f1d-a5aa-3c60f4ecc0f0 None admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:40:47.053140 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-522a6079-e782-4f1d-a5aa-3c60f4ecc0f0 None admin] PATH_INFO: `/v3/roles` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:40:47.053903 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-522a6079-e782-4f1d-a5aa-3c60f4ecc0f0 None admin] RBAC: Adding query filter params (name=admin) {{(pid=92057) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} May 23 12:40:47.054214 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-522a6079-e782-4f1d-a5aa-3c60f4ecc0f0 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:40:47.055613 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-522a6079-e782-4f1d-a5aa-3c60f4ecc0f0 None admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:40:47.074331 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 131/264] 10.222.0.30 () {56 vars in 1099 bytes} [Mon May 23 12:40:46 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 248 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) May 23 12:40:47.083088 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-522a6079-e782-4f1d-a5aa-3c60f4ecc0f0 None admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:40:47.119177 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9b66c4e-a36e-4748-8f4c-67e2f4101afc None admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:40:47.155540 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50a7690c-35ab-4075-8af4-623dcfe30daf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:40:47.156202 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50a7690c-35ab-4075-8af4-623dcfe30daf None admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:40:47.156489 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50a7690c-35ab-4075-8af4-623dcfe30daf None admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:40:47.156814 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50a7690c-35ab-4075-8af4-623dcfe30daf None admin] PATH_INFO: `/v3/role_assignments` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:40:47.157649 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-50a7690c-35ab-4075-8af4-623dcfe30daf None admin] RBAC: Adding query filter params (role.id=e3b90948645643e888620ab1e5c774bf, scope.project.id=b39742b1d1b34dfe82a9ce4da39b7582, user.id=237cf4210ddf4163a2353c03fdf231a9) {{(pid=92056) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} May 23 12:40:47.157993 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-50a7690c-35ab-4075-8af4-623dcfe30daf None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:40:47.159577 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-50a7690c-35ab-4075-8af4-623dcfe30daf None admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:40:47.166973 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 134/265] 10.222.0.30 () {56 vars in 1374 bytes} [Mon May 23 12:40:47 2022] GET /identity/v3/role_assignments?user.id=237cf4210ddf4163a2353c03fdf231a9&scope.project.id=b39742b1d1b34dfe82a9ce4da39b7582&role.id=e3b90948645643e888620ab1e5c774bf => generated 608 bytes in 161 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) May 23 12:40:47.257566 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-522a6079-e782-4f1d-a5aa-3c60f4ecc0f0 None admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:40:47.262983 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=100 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:40:47.306862 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9863b3f-c1f0-4a89-8fad-346ca8bbc69e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:40:47.307971 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9863b3f-c1f0-4a89-8fad-346ca8bbc69e None admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:40:47.308266 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9863b3f-c1f0-4a89-8fad-346ca8bbc69e None admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:40:47.308543 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9863b3f-c1f0-4a89-8fad-346ca8bbc69e None admin] PATH_INFO: `/v3/users/237cf4210ddf4163a2353c03fdf231a9` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:40:47.310079 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9863b3f-c1f0-4a89-8fad-346ca8bbc69e None admin] RBAC: Authorizing `identity:get_user(user_id=237cf4210ddf4163a2353c03fdf231a9)` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:40:47.311646 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9863b3f-c1f0-4a89-8fad-346ca8bbc69e None admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:40:47.313726 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 132/266] 10.222.0.30 () {56 vars in 1177 bytes} [Mon May 23 12:40:47 2022] GET /identity/v3/users/237cf4210ddf4163a2353c03fdf231a9 => generated 283 bytes in 231 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) May 23 12:40:47.319241 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50a7690c-35ab-4075-8af4-623dcfe30daf None admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:40:47.386453 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=100 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:40:47.430913 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50a7690c-35ab-4075-8af4-623dcfe30daf None admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:40:47.467008 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8ce0c07-d1b6-4565-9b9a-2fa41eb7652f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:40:47.467670 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8ce0c07-d1b6-4565-9b9a-2fa41eb7652f None admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:40:47.467952 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8ce0c07-d1b6-4565-9b9a-2fa41eb7652f None admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:40:47.468228 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8ce0c07-d1b6-4565-9b9a-2fa41eb7652f None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:40:47.550631 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8ce0c07-d1b6-4565-9b9a-2fa41eb7652f None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:40:47.552518 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8ce0c07-d1b6-4565-9b9a-2fa41eb7652f None admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:40:47.613322 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=100 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:40:47.658422 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 135/267] 10.222.0.30 () {58 vars in 1282 bytes} [Mon May 23 12:40:47 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 340 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) May 23 12:40:47.663862 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9863b3f-c1f0-4a89-8fad-346ca8bbc69e None admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:40:47.778281 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9863b3f-c1f0-4a89-8fad-346ca8bbc69e None admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:40:47.813639 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c901fb7-2b03-437d-9b0e-923d3c3e1f49 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:40:47.814528 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c901fb7-2b03-437d-9b0e-923d3c3e1f49 None admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:40:47.814813 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c901fb7-2b03-437d-9b0e-923d3c3e1f49 None admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:40:47.815089 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c901fb7-2b03-437d-9b0e-923d3c3e1f49 None admin] PATH_INFO: `/v3/projects/b39742b1d1b34dfe82a9ce4da39b7582` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:40:47.816299 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c901fb7-2b03-437d-9b0e-923d3c3e1f49 None admin] RBAC: Authorizing `identity:get_project(project_id=b39742b1d1b34dfe82a9ce4da39b7582)` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:40:47.817758 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c901fb7-2b03-437d-9b0e-923d3c3e1f49 None admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:40:47.819798 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 133/268] 10.222.0.30 () {56 vars in 1186 bytes} [Mon May 23 12:40:47 2022] GET /identity/v3/projects/b39742b1d1b34dfe82a9ce4da39b7582 => generated 302 bytes in 156 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) May 23 12:40:47.826731 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8ce0c07-d1b6-4565-9b9a-2fa41eb7652f None admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:40:47.934542 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8ce0c07-d1b6-4565-9b9a-2fa41eb7652f None admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:40:47.970586 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f017e03-2009-4283-bd92-a0475a061e6e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:40:47.971749 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f017e03-2009-4283-bd92-a0475a061e6e None admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:40:47.972033 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f017e03-2009-4283-bd92-a0475a061e6e None admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:40:47.972308 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f017e03-2009-4283-bd92-a0475a061e6e None admin] PATH_INFO: `/v3/role_assignments` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:40:47.973175 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f017e03-2009-4283-bd92-a0475a061e6e None admin] RBAC: Adding query filter params (role.id=bc3c3c9d654940cbaa5177a6be23d1bc, scope.project.id=b39742b1d1b34dfe82a9ce4da39b7582, user.id=237cf4210ddf4163a2353c03fdf231a9) {{(pid=92056) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} May 23 12:40:47.973489 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f017e03-2009-4283-bd92-a0475a061e6e None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:40:47.975118 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f017e03-2009-4283-bd92-a0475a061e6e None admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:40:47.982546 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 136/269] 10.222.0.30 () {56 vars in 1374 bytes} [Mon May 23 12:40:47 2022] GET /identity/v3/role_assignments?user.id=237cf4210ddf4163a2353c03fdf231a9&scope.project.id=b39742b1d1b34dfe82a9ce4da39b7582&role.id=bc3c3c9d654940cbaa5177a6be23d1bc => generated 608 bytes in 157 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) May 23 12:40:48.502465 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b2006cf-7eeb-473d-bc2f-851ae021724d None None] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:40:48.502837 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b2006cf-7eeb-473d-bc2f-851ae021724d None None] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:40:48.503034 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b2006cf-7eeb-473d-bc2f-851ae021724d None None] PATH_INFO: `/` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:40:48.504701 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 134/270] 10.222.0.30 () {54 vars in 926 bytes} [Mon May 23 12:40:48 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) May 23 12:40:48.510198 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14957d2d-dbd4-4bba-b87d-dc106141b207 None None] REQUEST_METHOD: `POST` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:40:48.510495 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14957d2d-dbd4-4bba-b87d-dc106141b207 None None] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:40:48.510896 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14957d2d-dbd4-4bba-b87d-dc106141b207 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:40:48.530644 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.auth.core [None req-14957d2d-dbd4-4bba-b87d-dc106141b207 None None] MFA Rules not processed for user `0f7119f8fa5c4445977671140ded91a4`. Rule list: `[]` (Enabled: `True`). {{(pid=92056) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 12:40:48.532196 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.fernet_utils [None req-14957d2d-dbd4-4bba-b87d-dc106141b207 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92056) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 12:40:48.534906 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2143418-dfd2-445c-b394-1a32ca914d4d None None] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:40:48.535262 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2143418-dfd2-445c-b394-1a32ca914d4d None None] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:40:48.535611 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2143418-dfd2-445c-b394-1a32ca914d4d None None] PATH_INFO: `/` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:40:48.537986 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 135/271] 10.222.0.30 () {54 vars in 926 bytes} [Mon May 23 12:40:48 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) May 23 12:40:48.543686 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11fb8be5-8522-4f84-8d3c-97b445455ef5 None None] REQUEST_METHOD: `POST` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:40:48.543686 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11fb8be5-8522-4f84-8d3c-97b445455ef5 None None] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:40:48.543903 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11fb8be5-8522-4f84-8d3c-97b445455ef5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:40:48.555018 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=100 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:40:48.561265 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.auth.core [None req-11fb8be5-8522-4f84-8d3c-97b445455ef5 None None] MFA Rules not processed for user `0f7119f8fa5c4445977671140ded91a4`. Rule list: `[]` (Enabled: `True`). {{(pid=92057) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 12:40:48.562605 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.fernet_utils [None req-11fb8be5-8522-4f84-8d3c-97b445455ef5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92057) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 12:40:48.599581 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=100 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:40:48.614103 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 137/272] 10.222.0.30 () {58 vars in 1023 bytes} [Mon May 23 12:40:48 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 102 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) May 23 12:40:48.614103 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e182ae7c-aa1d-4e8b-a533-a81a3e65d110 None None] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:40:48.614103 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e182ae7c-aa1d-4e8b-a533-a81a3e65d110 None None] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:40:48.614103 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e182ae7c-aa1d-4e8b-a533-a81a3e65d110 None None] PATH_INFO: `/` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:40:48.614872 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 138/273] 10.222.0.30 () {54 vars in 926 bytes} [Mon May 23 12:40:48 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) May 23 12:40:48.627048 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ca32d97-0042-4b1f-b825-9ed2be18dae6 None None] REQUEST_METHOD: `POST` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:40:48.627627 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ca32d97-0042-4b1f-b825-9ed2be18dae6 None None] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:40:48.628126 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ca32d97-0042-4b1f-b825-9ed2be18dae6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:40:48.640260 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 136/274] 10.222.0.30 () {58 vars in 1023 bytes} [Mon May 23 12:40:48 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 97 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) May 23 12:40:48.641068 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11fb8be5-8522-4f84-8d3c-97b445455ef5 None None] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:40:48.646387 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.auth.core [None req-0ca32d97-0042-4b1f-b825-9ed2be18dae6 None None] MFA Rules not processed for user `0f7119f8fa5c4445977671140ded91a4`. Rule list: `[]` (Enabled: `True`). {{(pid=92056) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 12:40:48.647782 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.fernet_utils [None req-0ca32d97-0042-4b1f-b825-9ed2be18dae6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92056) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 12:40:48.718426 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 139/275] 10.222.0.30 () {58 vars in 1023 bytes} [Mon May 23 12:40:48 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 92 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) May 23 12:40:48.728026 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ca32d97-0042-4b1f-b825-9ed2be18dae6 None None] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:40:48.762903 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11fb8be5-8522-4f84-8d3c-97b445455ef5 None None] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:40:48.802224 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-822dcb9a-eeed-450a-8067-c423162be757 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:40:48.803152 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-822dcb9a-eeed-450a-8067-c423162be757 None admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:40:48.803403 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-822dcb9a-eeed-450a-8067-c423162be757 None admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:40:48.803679 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-822dcb9a-eeed-450a-8067-c423162be757 None admin] PATH_INFO: `/v3/roles/reader` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:40:48.815249 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-822dcb9a-eeed-450a-8067-c423162be757 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} May 23 12:40:48.824422 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application [None req-822dcb9a-eeed-450a-8067-c423162be757 None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. May 23 12:40:48.824422 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application Traceback (most recent call last): May 23 12:40:48.824422 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request May 23 12:40:48.824422 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application rv = self.dispatch_request() May 23 12:40:48.824422 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request May 23 12:40:48.824422 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) May 23 12:40:48.824422 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper May 23 12:40:48.824422 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) May 23 12:40:48.824422 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view May 23 12:40:48.824422 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) May 23 12:40:48.824422 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request May 23 12:40:48.824422 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) May 23 12:40:48.824422 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get May 23 12:40:48.824422 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self._get_role(role_id) May 23 12:40:48.824422 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role May 23 12:40:48.824422 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') May 23 12:40:48.825441 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call May 23 12:40:48.825441 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( May 23 12:40:48.825441 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data May 23 12:40:48.825441 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) May 23 12:40:48.825441 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped May 23 12:40:48.825441 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) May 23 12:40:48.825441 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun May 23 12:40:48.825441 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) May 23 12:40:48.825441 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func May 23 12:40:48.825441 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self.get_or_create( May 23 12:40:48.825441 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create May 23 12:40:48.825441 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application with Lock( May 23 12:40:48.825441 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ May 23 12:40:48.825441 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self._enter() May 23 12:40:48.825441 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter May 23 12:40:48.825441 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) May 23 12:40:48.826332 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create May 23 12:40:48.826332 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self.creator() May 23 12:40:48.826332 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value May 23 12:40:48.826332 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application created_value = creator( May 23 12:40:48.826332 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role May 23 12:40:48.826332 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) May 23 12:40:48.826332 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role May 23 12:40:48.826332 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() May 23 12:40:48.826332 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role May 23 12:40:48.826332 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) May 23 12:40:48.826332 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. May 23 12:40:48.826332 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application May 23 12:40:48.826950 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=100 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:40:48.827021 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 137/276] 10.222.0.30 () {56 vars in 1099 bytes} [Mon May 23 12:40:48 2022] GET /identity/v3/roles/reader => generated 84 bytes in 186 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) May 23 12:40:48.827804 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-822dcb9a-eeed-450a-8067-c423162be757 None admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:40:48.848917 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ca32d97-0042-4b1f-b825-9ed2be18dae6 None None] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:40:48.861126 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=100 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:40:48.891376 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57880a8a-5bd4-418c-ab07-9ef493362d52 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:40:48.892169 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57880a8a-5bd4-418c-ab07-9ef493362d52 None admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:40:48.892451 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57880a8a-5bd4-418c-ab07-9ef493362d52 None admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:40:48.892725 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57880a8a-5bd4-418c-ab07-9ef493362d52 None admin] PATH_INFO: `/v3/roles/member` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:40:48.903231 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57880a8a-5bd4-418c-ab07-9ef493362d52 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} May 23 12:40:48.911286 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application [None req-57880a8a-5bd4-418c-ab07-9ef493362d52 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. May 23 12:40:48.911286 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application Traceback (most recent call last): May 23 12:40:48.911286 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request May 23 12:40:48.911286 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application rv = self.dispatch_request() May 23 12:40:48.911286 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request May 23 12:40:48.911286 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) May 23 12:40:48.911286 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper May 23 12:40:48.911286 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) May 23 12:40:48.911286 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view May 23 12:40:48.911286 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) May 23 12:40:48.911286 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request May 23 12:40:48.911286 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) May 23 12:40:48.911286 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get May 23 12:40:48.911286 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self._get_role(role_id) May 23 12:40:48.911286 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role May 23 12:40:48.911286 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') May 23 12:40:48.912216 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call May 23 12:40:48.912216 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( May 23 12:40:48.912216 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data May 23 12:40:48.912216 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) May 23 12:40:48.912216 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped May 23 12:40:48.912216 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) May 23 12:40:48.912216 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun May 23 12:40:48.912216 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) May 23 12:40:48.912216 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func May 23 12:40:48.912216 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self.get_or_create( May 23 12:40:48.912216 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create May 23 12:40:48.912216 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application with Lock( May 23 12:40:48.912216 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ May 23 12:40:48.912216 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self._enter() May 23 12:40:48.912216 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter May 23 12:40:48.912216 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) May 23 12:40:48.913046 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create May 23 12:40:48.913046 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self.creator() May 23 12:40:48.913046 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value May 23 12:40:48.913046 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application created_value = creator( May 23 12:40:48.913046 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role May 23 12:40:48.913046 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) May 23 12:40:48.913046 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role May 23 12:40:48.913046 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() May 23 12:40:48.913046 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role May 23 12:40:48.913046 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) May 23 12:40:48.913046 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. May 23 12:40:48.913046 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application May 23 12:40:48.913046 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 140/277] 10.222.0.30 () {56 vars in 1099 bytes} [Mon May 23 12:40:48 2022] GET /identity/v3/roles/member => generated 84 bytes in 186 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) May 23 12:40:48.913713 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57880a8a-5bd4-418c-ab07-9ef493362d52 None admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:40:48.943164 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-822dcb9a-eeed-450a-8067-c423162be757 None admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:40:48.987418 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6bec803d-dcd7-411c-8f3f-ebdd015f57a5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:40:48.988210 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bec803d-dcd7-411c-8f3f-ebdd015f57a5 None admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:40:48.988548 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bec803d-dcd7-411c-8f3f-ebdd015f57a5 None admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:40:48.988814 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bec803d-dcd7-411c-8f3f-ebdd015f57a5 None admin] PATH_INFO: `/v3/domains/default` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:40:48.989956 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6bec803d-dcd7-411c-8f3f-ebdd015f57a5 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:40:48.991473 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6bec803d-dcd7-411c-8f3f-ebdd015f57a5 None admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:40:48.993299 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 138/278] 10.222.0.30 () {56 vars in 1108 bytes} [Mon May 23 12:40:48 2022] GET /identity/v3/domains/default => generated 199 bytes in 166 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) May 23 12:40:48.995454 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6bec803d-dcd7-411c-8f3f-ebdd015f57a5 None admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:40:49.027156 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57880a8a-5bd4-418c-ab07-9ef493362d52 None admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:40:49.066037 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd920f16-aeb2-4401-86bd-effa72dbf331 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:40:49.066866 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd920f16-aeb2-4401-86bd-effa72dbf331 None admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:40:49.067160 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd920f16-aeb2-4401-86bd-effa72dbf331 None admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:40:49.067439 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd920f16-aeb2-4401-86bd-effa72dbf331 None admin] PATH_INFO: `/v3/roles` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:40:49.068196 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd920f16-aeb2-4401-86bd-effa72dbf331 None admin] RBAC: Adding query filter params (name=reader) {{(pid=92056) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} May 23 12:40:49.068508 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd920f16-aeb2-4401-86bd-effa72dbf331 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:40:49.069921 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd920f16-aeb2-4401-86bd-effa72dbf331 None admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:40:49.083143 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 141/279] 10.222.0.30 () {56 vars in 1101 bytes} [Mon May 23 12:40:48 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 166 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) May 23 12:40:49.083143 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd920f16-aeb2-4401-86bd-effa72dbf331 None admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:40:49.094275 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=100 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:40:49.100435 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=100 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:40:49.105173 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6bec803d-dcd7-411c-8f3f-ebdd015f57a5 None admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:40:49.139549 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-801afd9c-7e3a-4a82-97b8-97fa9d4290aa None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:40:49.140215 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-801afd9c-7e3a-4a82-97b8-97fa9d4290aa None admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:40:49.140503 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-801afd9c-7e3a-4a82-97b8-97fa9d4290aa None admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:40:49.140781 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-801afd9c-7e3a-4a82-97b8-97fa9d4290aa None admin] PATH_INFO: `/v3/roles` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:40:49.141524 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-801afd9c-7e3a-4a82-97b8-97fa9d4290aa None admin] RBAC: Adding query filter params (name=member) {{(pid=92057) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} May 23 12:40:49.141832 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-801afd9c-7e3a-4a82-97b8-97fa9d4290aa None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:40:49.143243 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-801afd9c-7e3a-4a82-97b8-97fa9d4290aa None admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:40:49.151842 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 139/280] 10.222.0.30 () {56 vars in 1101 bytes} [Mon May 23 12:40:48 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 158 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) May 23 12:40:49.152642 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-801afd9c-7e3a-4a82-97b8-97fa9d4290aa None admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:40:49.186074 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd920f16-aeb2-4401-86bd-effa72dbf331 None admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:40:49.219178 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f78f48a7-7c60-4460-90ff-421d8afd06e5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:40:49.219843 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f78f48a7-7c60-4460-90ff-421d8afd06e5 None admin] REQUEST_METHOD: `POST` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:40:49.220128 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f78f48a7-7c60-4460-90ff-421d8afd06e5 None admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:40:49.220406 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f78f48a7-7c60-4460-90ff-421d8afd06e5 None admin] PATH_INFO: `/v3/users` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:40:49.221113 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f78f48a7-7c60-4460-90ff-421d8afd06e5 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:40:49.222601 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f78f48a7-7c60-4460-90ff-421d8afd06e5 None admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:40:49.251655 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-801afd9c-7e3a-4a82-97b8-97fa9d4290aa None admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:40:49.284371 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0c33ba2-0397-42fa-830e-0374929223e9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:40:49.285174 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0c33ba2-0397-42fa-830e-0374929223e9 None admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:40:49.285457 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0c33ba2-0397-42fa-830e-0374929223e9 None admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:40:49.285761 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0c33ba2-0397-42fa-830e-0374929223e9 None admin] PATH_INFO: `/v3/users/3f1c3ab2da314bff8c13f587a8d55a52` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:40:49.287008 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0c33ba2-0397-42fa-830e-0374929223e9 None admin] RBAC: Authorizing `identity:get_user(user_id=3f1c3ab2da314bff8c13f587a8d55a52)` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:40:49.288443 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0c33ba2-0397-42fa-830e-0374929223e9 None admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:40:49.290259 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 140/281] 10.222.0.30 () {56 vars in 1177 bytes} [Mon May 23 12:40:49 2022] GET /identity/v3/users/3f1c3ab2da314bff8c13f587a8d55a52 => generated 297 bytes in 138 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) May 23 12:40:49.292357 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d95b28c-b92a-40ff-8888-01335e500e2c None None] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:40:49.292643 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d95b28c-b92a-40ff-8888-01335e500e2c None None] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:40:49.292919 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d95b28c-b92a-40ff-8888-01335e500e2c None None] PATH_INFO: `/` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:40:49.294166 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 141/282] 10.222.0.30 () {54 vars in 926 bytes} [Mon May 23 12:40:49 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) May 23 12:40:49.294859 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d95b28c-b92a-40ff-8888-01335e500e2c None None] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:40:49.322577 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=100 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:40:49.333333 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 142/283] 10.222.0.30 () {60 vars in 1132 bytes} [Mon May 23 12:40:49 2022] POST /identity/v3/users => generated 293 bytes in 254 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) May 23 12:40:49.334199 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f78f48a7-7c60-4460-90ff-421d8afd06e5 None admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:40:49.406937 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:40:49.425045 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d95b28c-b92a-40ff-8888-01335e500e2c None None] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:40:49.449425 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f78f48a7-7c60-4460-90ff-421d8afd06e5 None admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:40:49.464529 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6fc1a024-528d-4d05-8a53-42983a29d7e2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:40:49.465422 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fc1a024-528d-4d05-8a53-42983a29d7e2 None admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:40:49.465807 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fc1a024-528d-4d05-8a53-42983a29d7e2 None admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:40:49.466021 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fc1a024-528d-4d05-8a53-42983a29d7e2 None admin] PATH_INFO: `/v3/users/3d469b7f6b4142bb9ca60e30e9b2a1d5` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:40:49.481980 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6fc1a024-528d-4d05-8a53-42983a29d7e2 None admin] RBAC: Authorizing `identity:get_user(user_id=3d469b7f6b4142bb9ca60e30e9b2a1d5)` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:40:49.483582 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6fc1a024-528d-4d05-8a53-42983a29d7e2 None admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:40:49.490016 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 142/284] 10.222.0.30 () {56 vars in 1177 bytes} [Mon May 23 12:40:49 2022] GET /identity/v3/users/3d469b7f6b4142bb9ca60e30e9b2a1d5 => generated 293 bytes in 191 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) May 23 12:40:49.490016 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a4d94a7-09fb-45c1-bbb3-ba2ffd77007a None None] REQUEST_METHOD: `POST` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:40:49.490016 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a4d94a7-09fb-45c1-bbb3-ba2ffd77007a None None] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:40:49.490016 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a4d94a7-09fb-45c1-bbb3-ba2ffd77007a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:40:49.493279 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5999476f-513b-49a6-a257-9fa2c8bb96b4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:40:49.494393 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5999476f-513b-49a6-a257-9fa2c8bb96b4 None admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:40:49.494752 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5999476f-513b-49a6-a257-9fa2c8bb96b4 None admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:40:49.495057 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5999476f-513b-49a6-a257-9fa2c8bb96b4 None admin] PATH_INFO: `/v3/projects/b39742b1d1b34dfe82a9ce4da39b7582` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:40:49.496504 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5999476f-513b-49a6-a257-9fa2c8bb96b4 None admin] RBAC: Authorizing `identity:get_project(project_id=b39742b1d1b34dfe82a9ce4da39b7582)` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:40:49.498034 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5999476f-513b-49a6-a257-9fa2c8bb96b4 None admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:40:49.499865 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 143/285] 10.222.0.30 () {56 vars in 1186 bytes} [Mon May 23 12:40:49 2022] GET /identity/v3/projects/b39742b1d1b34dfe82a9ce4da39b7582 => generated 302 bytes in 166 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) May 23 12:40:49.500664 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5999476f-513b-49a6-a257-9fa2c8bb96b4 None admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:40:49.512101 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.auth.core [None req-9a4d94a7-09fb-45c1-bbb3-ba2ffd77007a None None] MFA Rules not processed for user `0f7119f8fa5c4445977671140ded91a4`. Rule list: `[]` (Enabled: `True`). {{(pid=92057) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 12:40:49.513315 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.fernet_utils [None req-9a4d94a7-09fb-45c1-bbb3-ba2ffd77007a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92057) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 12:40:49.605512 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 143/286] 10.222.0.30 () {58 vars in 1023 bytes} [Mon May 23 12:40:49 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 119 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) May 23 12:40:49.606350 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a4d94a7-09fb-45c1-bbb3-ba2ffd77007a None None] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:40:49.628090 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=100 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:40:49.635498 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5999476f-513b-49a6-a257-9fa2c8bb96b4 None admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:40:49.675425 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=100 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:40:49.688046 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-effd2f47-cfd5-4a24-a52d-1ca1790ea08f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:40:49.688771 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-effd2f47-cfd5-4a24-a52d-1ca1790ea08f None admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:40:49.689054 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-effd2f47-cfd5-4a24-a52d-1ca1790ea08f None admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:40:49.689848 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-effd2f47-cfd5-4a24-a52d-1ca1790ea08f None admin] PATH_INFO: `/v3/role_assignments` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:40:49.690260 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-effd2f47-cfd5-4a24-a52d-1ca1790ea08f None admin] RBAC: Adding query filter params (role.id=80fb22507c9a4ff09d44d8f12f6fff1a, user.id=3d469b7f6b4142bb9ca60e30e9b2a1d5, scope.system=all) {{(pid=92056) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} May 23 12:40:49.692715 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-effd2f47-cfd5-4a24-a52d-1ca1790ea08f None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:40:49.692715 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-effd2f47-cfd5-4a24-a52d-1ca1790ea08f None admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:40:49.699366 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 144/287] 10.222.0.30 () {56 vars in 1308 bytes} [Mon May 23 12:40:49 2022] GET /identity/v3/role_assignments?user.id=3d469b7f6b4142bb9ca60e30e9b2a1d5&scope.system=all&role.id=80fb22507c9a4ff09d44d8f12f6fff1a => generated 226 bytes in 198 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) May 23 12:40:49.699567 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-effd2f47-cfd5-4a24-a52d-1ca1790ea08f None admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:40:49.762949 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a4d94a7-09fb-45c1-bbb3-ba2ffd77007a None None] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:40:49.817334 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc30fc9d-c180-46b0-a692-01866f4d3282 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:40:49.819280 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc30fc9d-c180-46b0-a692-01866f4d3282 None admin] REQUEST_METHOD: `PUT` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:40:49.821404 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc30fc9d-c180-46b0-a692-01866f4d3282 None admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:40:49.821698 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc30fc9d-c180-46b0-a692-01866f4d3282 None admin] PATH_INFO: `/v3/projects/b39742b1d1b34dfe82a9ce4da39b7582/users/3f1c3ab2da314bff8c13f587a8d55a52/roles/c223286ed21a4654a0f2b4e5551831c7` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:40:49.830255 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc30fc9d-c180-46b0-a692-01866f4d3282 None admin] RBAC: Authorizing `identity:create_grant(project_id=b39742b1d1b34dfe82a9ce4da39b7582, user_id=3f1c3ab2da314bff8c13f587a8d55a52, role_id=c223286ed21a4654a0f2b4e5551831c7)` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:40:49.831740 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc30fc9d-c180-46b0-a692-01866f4d3282 None admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:40:49.855330 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-effd2f47-cfd5-4a24-a52d-1ca1790ea08f None admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:40:49.895925 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf3a9c7f-fdbc-488f-92db-4668c3b208b2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:40:49.896728 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf3a9c7f-fdbc-488f-92db-4668c3b208b2 None admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:40:49.897013 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf3a9c7f-fdbc-488f-92db-4668c3b208b2 None admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:40:49.897290 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf3a9c7f-fdbc-488f-92db-4668c3b208b2 None admin] PATH_INFO: `/v3/roles/member` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:40:49.910748 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf3a9c7f-fdbc-488f-92db-4668c3b208b2 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} May 23 12:40:49.915380 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 144/288] 10.222.0.30 () {58 vars in 1439 bytes} [Mon May 23 12:40:49 2022] PUT /identity/v3/projects/b39742b1d1b34dfe82a9ce4da39b7582/users/3f1c3ab2da314bff8c13f587a8d55a52/roles/c223286ed21a4654a0f2b4e5551831c7 => generated 0 bytes in 309 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) May 23 12:40:49.920617 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application [None req-cf3a9c7f-fdbc-488f-92db-4668c3b208b2 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. May 23 12:40:49.920617 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application Traceback (most recent call last): May 23 12:40:49.920617 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request May 23 12:40:49.920617 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application rv = self.dispatch_request() May 23 12:40:49.920617 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request May 23 12:40:49.920617 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) May 23 12:40:49.920617 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper May 23 12:40:49.920617 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) May 23 12:40:49.920617 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view May 23 12:40:49.920617 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) May 23 12:40:49.920617 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request May 23 12:40:49.920617 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) May 23 12:40:49.920617 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get May 23 12:40:49.920617 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self._get_role(role_id) May 23 12:40:49.920617 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role May 23 12:40:49.920617 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') May 23 12:40:49.921650 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call May 23 12:40:49.921650 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( May 23 12:40:49.921650 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data May 23 12:40:49.921650 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) May 23 12:40:49.921650 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped May 23 12:40:49.921650 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) May 23 12:40:49.921650 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun May 23 12:40:49.921650 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) May 23 12:40:49.921650 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func May 23 12:40:49.921650 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self.get_or_create( May 23 12:40:49.921650 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create May 23 12:40:49.921650 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application with Lock( May 23 12:40:49.921650 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ May 23 12:40:49.921650 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self._enter() May 23 12:40:49.921650 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter May 23 12:40:49.921650 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) May 23 12:40:49.922518 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create May 23 12:40:49.922518 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self.creator() May 23 12:40:49.922518 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value May 23 12:40:49.922518 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application created_value = creator( May 23 12:40:49.922518 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role May 23 12:40:49.922518 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) May 23 12:40:49.922518 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role May 23 12:40:49.922518 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() May 23 12:40:49.922518 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role May 23 12:40:49.922518 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) May 23 12:40:49.922518 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. May 23 12:40:49.922518 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application May 23 12:40:49.925301 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 145/289] 10.222.0.30 () {56 vars in 1099 bytes} [Mon May 23 12:40:49 2022] GET /identity/v3/roles/member => generated 84 bytes in 226 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) May 23 12:40:49.931108 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc30fc9d-c180-46b0-a692-01866f4d3282 None admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:40:50.034886 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:40:50.062046 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc30fc9d-c180-46b0-a692-01866f4d3282 None admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:40:50.101453 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36772b24-37ac-461d-becc-3f11e7942769 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:40:50.102532 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36772b24-37ac-461d-becc-3f11e7942769 None admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:40:50.103021 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36772b24-37ac-461d-becc-3f11e7942769 None admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:40:50.104053 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36772b24-37ac-461d-becc-3f11e7942769 None admin] PATH_INFO: `/v3/roles` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:40:50.105011 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36772b24-37ac-461d-becc-3f11e7942769 None admin] RBAC: Adding query filter params (name=member) {{(pid=92057) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} May 23 12:40:50.105516 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36772b24-37ac-461d-becc-3f11e7942769 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:40:50.107176 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36772b24-37ac-461d-becc-3f11e7942769 None admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:40:50.117625 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 145/290] 10.222.0.30 () {56 vars in 1101 bytes} [Mon May 23 12:40:49 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 187 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) May 23 12:40:50.126723 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf3a9c7f-fdbc-488f-92db-4668c3b208b2 None admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:40:50.202267 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=100 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:40:50.250878 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf3a9c7f-fdbc-488f-92db-4668c3b208b2 None admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:40:50.288259 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f6b00f5-1307-4968-9815-35caa8867016 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:40:50.289501 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f6b00f5-1307-4968-9815-35caa8867016 None admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:40:50.290026 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f6b00f5-1307-4968-9815-35caa8867016 None admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:40:50.290505 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f6b00f5-1307-4968-9815-35caa8867016 None admin] PATH_INFO: `/v3/users/84f86330620a493684d2835681c966eb` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:40:50.292606 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f6b00f5-1307-4968-9815-35caa8867016 None admin] RBAC: Authorizing `identity:get_user(user_id=84f86330620a493684d2835681c966eb)` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:40:50.294317 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f6b00f5-1307-4968-9815-35caa8867016 None admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:40:50.296594 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 146/291] 10.222.0.30 () {56 vars in 1177 bytes} [Mon May 23 12:40:50 2022] GET /identity/v3/users/84f86330620a493684d2835681c966eb => generated 297 bytes in 170 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) May 23 12:40:50.302983 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36772b24-37ac-461d-becc-3f11e7942769 None admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:40:50.438692 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36772b24-37ac-461d-becc-3f11e7942769 None admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:40:50.492081 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45c2d3a2-0bde-4077-91e6-fa5a695ac7fe None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:40:50.492659 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45c2d3a2-0bde-4077-91e6-fa5a695ac7fe None admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:40:50.492944 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45c2d3a2-0bde-4077-91e6-fa5a695ac7fe None admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:40:50.493219 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45c2d3a2-0bde-4077-91e6-fa5a695ac7fe None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:40:50.508308 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=100 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:40:50.589330 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45c2d3a2-0bde-4077-91e6-fa5a695ac7fe None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:40:50.590847 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45c2d3a2-0bde-4077-91e6-fa5a695ac7fe None admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:40:50.736680 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 146/292] 10.222.0.30 () {58 vars in 1282 bytes} [Mon May 23 12:40:50 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 434 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) May 23 12:40:50.742287 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f6b00f5-1307-4968-9815-35caa8867016 None admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:40:50.900692 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f6b00f5-1307-4968-9815-35caa8867016 None admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:40:50.930288 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=100 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:40:50.944675 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1cff95a6-4b40-4770-9818-68e9a216d2f3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:40:50.947902 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1cff95a6-4b40-4770-9818-68e9a216d2f3 None admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:40:50.947902 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1cff95a6-4b40-4770-9818-68e9a216d2f3 None admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:40:50.947902 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1cff95a6-4b40-4770-9818-68e9a216d2f3 None admin] PATH_INFO: `/v3/projects/b39742b1d1b34dfe82a9ce4da39b7582` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:40:50.947902 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1cff95a6-4b40-4770-9818-68e9a216d2f3 None admin] RBAC: Authorizing `identity:get_project(project_id=b39742b1d1b34dfe82a9ce4da39b7582)` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:40:50.949010 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1cff95a6-4b40-4770-9818-68e9a216d2f3 None admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:40:50.956292 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 147/293] 10.222.0.30 () {56 vars in 1186 bytes} [Mon May 23 12:40:50 2022] GET /identity/v3/projects/b39742b1d1b34dfe82a9ce4da39b7582 => generated 302 bytes in 214 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) May 23 12:40:50.966973 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45c2d3a2-0bde-4077-91e6-fa5a695ac7fe None admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:40:51.017152 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=100 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:40:51.080583 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45c2d3a2-0bde-4077-91e6-fa5a695ac7fe None admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:40:51.131094 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-891bce64-3cf2-48db-8942-9a9fcb5bf272 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:40:51.132122 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-891bce64-3cf2-48db-8942-9a9fcb5bf272 None admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:40:51.132625 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-891bce64-3cf2-48db-8942-9a9fcb5bf272 None admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:40:51.133097 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-891bce64-3cf2-48db-8942-9a9fcb5bf272 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:40:51.134140 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-891bce64-3cf2-48db-8942-9a9fcb5bf272 None admin] RBAC: Adding query filter params (role.id=80fb22507c9a4ff09d44d8f12f6fff1a, scope.project.id=b39742b1d1b34dfe82a9ce4da39b7582, user.id=84f86330620a493684d2835681c966eb) {{(pid=92057) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} May 23 12:40:51.135197 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-891bce64-3cf2-48db-8942-9a9fcb5bf272 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:40:51.136934 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-891bce64-3cf2-48db-8942-9a9fcb5bf272 None admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:40:51.146332 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 147/294] 10.222.0.30 () {56 vars in 1374 bytes} [Mon May 23 12:40:50 2022] GET /identity/v3/role_assignments?user.id=84f86330620a493684d2835681c966eb&scope.project.id=b39742b1d1b34dfe82a9ce4da39b7582&role.id=80fb22507c9a4ff09d44d8f12f6fff1a => generated 608 bytes in 180 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) May 23 12:40:52.014100 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16947c30-2b61-4658-929b-b56455130881 None None] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:40:52.014100 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16947c30-2b61-4658-929b-b56455130881 None None] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:40:52.014686 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16947c30-2b61-4658-929b-b56455130881 None None] PATH_INFO: `/` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:40:52.016290 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 148/295] 10.222.0.30 () {54 vars in 926 bytes} [Mon May 23 12:40:52 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) May 23 12:40:52.022681 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06f16a52-353f-45e0-8aa5-c8b8c0a22bc2 None None] REQUEST_METHOD: `POST` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:40:52.023207 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06f16a52-353f-45e0-8aa5-c8b8c0a22bc2 None None] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:40:52.023498 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06f16a52-353f-45e0-8aa5-c8b8c0a22bc2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:40:52.026911 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-000d6c1b-dbc7-41bb-8cd4-a3687790de25 None None] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:40:52.027195 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-000d6c1b-dbc7-41bb-8cd4-a3687790de25 None None] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:40:52.027472 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-000d6c1b-dbc7-41bb-8cd4-a3687790de25 None None] PATH_INFO: `/` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:40:52.028953 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 149/296] 10.222.0.30 () {54 vars in 926 bytes} [Mon May 23 12:40:52 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) May 23 12:40:52.034882 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3debc53-1bcb-41e0-af2c-6653edc2dfd6 None None] REQUEST_METHOD: `POST` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:40:52.035246 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3debc53-1bcb-41e0-af2c-6653edc2dfd6 None None] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:40:52.035599 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3debc53-1bcb-41e0-af2c-6653edc2dfd6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:40:52.047832 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.auth.core [None req-06f16a52-353f-45e0-8aa5-c8b8c0a22bc2 None None] MFA Rules not processed for user `0f7119f8fa5c4445977671140ded91a4`. Rule list: `[]` (Enabled: `True`). {{(pid=92057) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 12:40:52.049843 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.fernet_utils [None req-06f16a52-353f-45e0-8aa5-c8b8c0a22bc2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92057) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 12:40:52.054624 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.auth.core [None req-d3debc53-1bcb-41e0-af2c-6653edc2dfd6 None None] MFA Rules not processed for user `0f7119f8fa5c4445977671140ded91a4`. Rule list: `[]` (Enabled: `True`). {{(pid=92056) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 12:40:52.055871 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.fernet_utils [None req-d3debc53-1bcb-41e0-af2c-6653edc2dfd6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92056) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 12:40:52.137920 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 150/297] 10.222.0.30 () {58 vars in 1023 bytes} [Mon May 23 12:40:52 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 104 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) May 23 12:40:52.145936 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 148/298] 10.222.0.30 () {58 vars in 1023 bytes} [Mon May 23 12:40:52 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 124 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) May 23 12:40:52.162353 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3debc53-1bcb-41e0-af2c-6653edc2dfd6 None None] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:40:52.170703 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06f16a52-353f-45e0-8aa5-c8b8c0a22bc2 None None] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:40:52.208041 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=100 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:40:52.289392 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06f16a52-353f-45e0-8aa5-c8b8c0a22bc2 None None] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:40:52.300900 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3debc53-1bcb-41e0-af2c-6653edc2dfd6 None None] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:40:52.327407 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4bf3e73b-e99d-4ec1-97d2-3f9031093ef9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:40:52.328312 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bf3e73b-e99d-4ec1-97d2-3f9031093ef9 None admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:40:52.328663 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bf3e73b-e99d-4ec1-97d2-3f9031093ef9 None admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:40:52.329006 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bf3e73b-e99d-4ec1-97d2-3f9031093ef9 None admin] PATH_INFO: `/v3/domains/default` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:40:52.330290 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4bf3e73b-e99d-4ec1-97d2-3f9031093ef9 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:40:52.331281 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=100 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:40:52.332034 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4bf3e73b-e99d-4ec1-97d2-3f9031093ef9 None admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:40:52.333602 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 149/299] 10.222.0.30 () {56 vars in 1108 bytes} [Mon May 23 12:40:52 2022] GET /identity/v3/domains/default => generated 199 bytes in 163 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) May 23 12:40:52.339439 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4bf3e73b-e99d-4ec1-97d2-3f9031093ef9 None admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:40:52.349650 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7a2dc9c-7735-4d10-9991-88065e7156f1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:40:52.350731 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7a2dc9c-7735-4d10-9991-88065e7156f1 None admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:40:52.351665 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7a2dc9c-7735-4d10-9991-88065e7156f1 None admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:40:52.352129 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7a2dc9c-7735-4d10-9991-88065e7156f1 None admin] PATH_INFO: `/v3/roles/reader` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:40:52.364147 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7a2dc9c-7735-4d10-9991-88065e7156f1 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} May 23 12:40:52.373719 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application [None req-e7a2dc9c-7735-4d10-9991-88065e7156f1 None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. May 23 12:40:52.373719 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application Traceback (most recent call last): May 23 12:40:52.373719 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request May 23 12:40:52.373719 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application rv = self.dispatch_request() May 23 12:40:52.373719 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request May 23 12:40:52.373719 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) May 23 12:40:52.373719 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper May 23 12:40:52.373719 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) May 23 12:40:52.373719 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view May 23 12:40:52.373719 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) May 23 12:40:52.373719 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request May 23 12:40:52.373719 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) May 23 12:40:52.373719 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get May 23 12:40:52.373719 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self._get_role(role_id) May 23 12:40:52.373719 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role May 23 12:40:52.373719 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') May 23 12:40:52.374859 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call May 23 12:40:52.374859 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( May 23 12:40:52.374859 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data May 23 12:40:52.374859 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) May 23 12:40:52.374859 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped May 23 12:40:52.374859 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) May 23 12:40:52.374859 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun May 23 12:40:52.374859 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) May 23 12:40:52.374859 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func May 23 12:40:52.374859 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self.get_or_create( May 23 12:40:52.374859 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create May 23 12:40:52.374859 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application with Lock( May 23 12:40:52.374859 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ May 23 12:40:52.374859 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self._enter() May 23 12:40:52.374859 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter May 23 12:40:52.374859 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) May 23 12:40:52.375894 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create May 23 12:40:52.375894 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self.creator() May 23 12:40:52.375894 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value May 23 12:40:52.375894 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application created_value = creator( May 23 12:40:52.375894 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role May 23 12:40:52.375894 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) May 23 12:40:52.375894 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role May 23 12:40:52.375894 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() May 23 12:40:52.375894 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role May 23 12:40:52.375894 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) May 23 12:40:52.375894 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. May 23 12:40:52.375894 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application May 23 12:40:52.382655 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 151/300] 10.222.0.30 () {56 vars in 1099 bytes} [Mon May 23 12:40:52 2022] GET /identity/v3/roles/reader => generated 84 bytes in 221 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) May 23 12:40:52.383766 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7a2dc9c-7735-4d10-9991-88065e7156f1 None admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:40:52.455918 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=100 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:40:52.459165 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4bf3e73b-e99d-4ec1-97d2-3f9031093ef9 None admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:40:52.494979 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7a2dc9c-7735-4d10-9991-88065e7156f1 None admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:40:52.497457 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6740b213-410e-439f-a10f-9520298a9ba3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:40:52.498165 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6740b213-410e-439f-a10f-9520298a9ba3 None admin] REQUEST_METHOD: `POST` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:40:52.498452 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6740b213-410e-439f-a10f-9520298a9ba3 None admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:40:52.498732 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6740b213-410e-439f-a10f-9520298a9ba3 None admin] PATH_INFO: `/v3/groups` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:40:52.499433 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6740b213-410e-439f-a10f-9520298a9ba3 None admin] RBAC: Authorizing `identity:create_group()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:40:52.500894 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6740b213-410e-439f-a10f-9520298a9ba3 None admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:40:52.528283 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9031d81c-c720-4ccb-99eb-66b347a9b186 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:40:52.529192 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9031d81c-c720-4ccb-99eb-66b347a9b186 None admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:40:52.529482 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9031d81c-c720-4ccb-99eb-66b347a9b186 None admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:40:52.529756 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9031d81c-c720-4ccb-99eb-66b347a9b186 None admin] PATH_INFO: `/v3/roles` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:40:52.530530 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9031d81c-c720-4ccb-99eb-66b347a9b186 None admin] RBAC: Adding query filter params (name=reader) {{(pid=92056) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} May 23 12:40:52.530838 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9031d81c-c720-4ccb-99eb-66b347a9b186 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:40:52.532205 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9031d81c-c720-4ccb-99eb-66b347a9b186 None admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:40:52.544142 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 152/301] 10.222.0.30 () {56 vars in 1101 bytes} [Mon May 23 12:40:52 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 157 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) May 23 12:40:52.544142 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5bc0e683-d670-42ff-b83a-378611de6fdb None None] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:40:52.544142 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5bc0e683-d670-42ff-b83a-378611de6fdb None None] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:40:52.544142 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5bc0e683-d670-42ff-b83a-378611de6fdb None None] PATH_INFO: `/` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:40:52.545140 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 153/302] 10.222.0.30 () {54 vars in 926 bytes} [Mon May 23 12:40:52 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) May 23 12:40:52.546045 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5bc0e683-d670-42ff-b83a-378611de6fdb None None] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:40:52.569674 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 150/303] 10.222.0.30 () {60 vars in 1134 bytes} [Mon May 23 12:40:52 2022] POST /identity/v3/groups => generated 229 bytes in 224 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) May 23 12:40:52.569674 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79d9602b-65ab-4ea1-b2f3-0376223cecd9 None None] REQUEST_METHOD: `POST` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:40:52.569674 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79d9602b-65ab-4ea1-b2f3-0376223cecd9 None None] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:40:52.569674 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79d9602b-65ab-4ea1-b2f3-0376223cecd9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:40:52.590505 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.auth.core [None req-79d9602b-65ab-4ea1-b2f3-0376223cecd9 None None] MFA Rules not processed for user `0f7119f8fa5c4445977671140ded91a4`. Rule list: `[]` (Enabled: `True`). {{(pid=92057) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 12:40:52.592096 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.fernet_utils [None req-79d9602b-65ab-4ea1-b2f3-0376223cecd9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92057) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 12:40:52.610105 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=100 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:40:52.692656 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 151/304] 10.222.0.30 () {58 vars in 1023 bytes} [Mon May 23 12:40:52 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 125 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) May 23 12:40:52.698644 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5bc0e683-d670-42ff-b83a-378611de6fdb None None] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:40:52.714066 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79d9602b-65ab-4ea1-b2f3-0376223cecd9 None None] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:40:52.740721 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8371617c-dbad-434a-bf9f-d0d7f4376a2c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:40:52.741899 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8371617c-dbad-434a-bf9f-d0d7f4376a2c None admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:40:52.742288 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8371617c-dbad-434a-bf9f-d0d7f4376a2c None admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:40:52.742667 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8371617c-dbad-434a-bf9f-d0d7f4376a2c None admin] PATH_INFO: `/v3/users/4c82e0529aea4c23a9556941da8b78db` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:40:52.760666 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8371617c-dbad-434a-bf9f-d0d7f4376a2c None admin] RBAC: Authorizing `identity:get_user(user_id=4c82e0529aea4c23a9556941da8b78db)` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:40:52.763066 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8371617c-dbad-434a-bf9f-d0d7f4376a2c None admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:40:52.764638 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 154/305] 10.222.0.30 () {56 vars in 1177 bytes} [Mon May 23 12:40:52 2022] GET /identity/v3/users/4c82e0529aea4c23a9556941da8b78db => generated 293 bytes in 219 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) May 23 12:40:52.770151 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8371617c-dbad-434a-bf9f-d0d7f4376a2c None admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:40:52.833788 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:40:52.843002 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79d9602b-65ab-4ea1-b2f3-0376223cecd9 None None] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:40:52.877987 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed6a3776-1512-48bb-9407-58bfe142c85e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:40:52.879129 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed6a3776-1512-48bb-9407-58bfe142c85e None admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:40:52.879480 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed6a3776-1512-48bb-9407-58bfe142c85e None admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:40:52.879826 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed6a3776-1512-48bb-9407-58bfe142c85e None admin] PATH_INFO: `/v3/roles/member` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:40:52.881583 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=100 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:40:52.885122 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8371617c-dbad-434a-bf9f-d0d7f4376a2c None admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:40:52.891382 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed6a3776-1512-48bb-9407-58bfe142c85e None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} May 23 12:40:52.899142 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application [None req-ed6a3776-1512-48bb-9407-58bfe142c85e None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. May 23 12:40:52.899142 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application Traceback (most recent call last): May 23 12:40:52.899142 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request May 23 12:40:52.899142 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application rv = self.dispatch_request() May 23 12:40:52.899142 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request May 23 12:40:52.899142 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) May 23 12:40:52.899142 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper May 23 12:40:52.899142 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) May 23 12:40:52.899142 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view May 23 12:40:52.899142 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) May 23 12:40:52.899142 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request May 23 12:40:52.899142 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) May 23 12:40:52.899142 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get May 23 12:40:52.899142 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self._get_role(role_id) May 23 12:40:52.899142 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role May 23 12:40:52.899142 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') May 23 12:40:52.900294 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call May 23 12:40:52.900294 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( May 23 12:40:52.900294 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data May 23 12:40:52.900294 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) May 23 12:40:52.900294 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped May 23 12:40:52.900294 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) May 23 12:40:52.900294 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun May 23 12:40:52.900294 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) May 23 12:40:52.900294 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func May 23 12:40:52.900294 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self.get_or_create( May 23 12:40:52.900294 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create May 23 12:40:52.900294 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application with Lock( May 23 12:40:52.900294 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ May 23 12:40:52.900294 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self._enter() May 23 12:40:52.900294 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter May 23 12:40:52.900294 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) May 23 12:40:52.901166 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create May 23 12:40:52.901166 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self.creator() May 23 12:40:52.901166 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value May 23 12:40:52.901166 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application created_value = creator( May 23 12:40:52.901166 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role May 23 12:40:52.901166 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) May 23 12:40:52.901166 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role May 23 12:40:52.901166 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() May 23 12:40:52.901166 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role May 23 12:40:52.901166 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) May 23 12:40:52.901166 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. May 23 12:40:52.901166 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application May 23 12:40:52.901166 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 152/306] 10.222.0.30 () {56 vars in 1099 bytes} [Mon May 23 12:40:52 2022] GET /identity/v3/roles/member => generated 84 bytes in 187 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) May 23 12:40:52.902115 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-102086ee-e6c5-4a25-b3ab-79c95714e4b6 None None] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:40:52.902405 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-102086ee-e6c5-4a25-b3ab-79c95714e4b6 None None] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:40:52.902683 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-102086ee-e6c5-4a25-b3ab-79c95714e4b6 None None] PATH_INFO: `/` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:40:52.904359 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 153/307] 10.222.0.30 () {54 vars in 926 bytes} [Mon May 23 12:40:52 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) May 23 12:40:52.908893 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-102086ee-e6c5-4a25-b3ab-79c95714e4b6 None None] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:40:52.927309 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4740ecaa-a011-4ebf-90b1-ee4983900143 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:40:52.927954 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4740ecaa-a011-4ebf-90b1-ee4983900143 None admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:40:52.928242 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4740ecaa-a011-4ebf-90b1-ee4983900143 None admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:40:52.928520 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4740ecaa-a011-4ebf-90b1-ee4983900143 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:40:52.929348 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4740ecaa-a011-4ebf-90b1-ee4983900143 None admin] RBAC: Adding query filter params (role.id=c223286ed21a4654a0f2b4e5551831c7, user.id=4c82e0529aea4c23a9556941da8b78db, scope.system=all) {{(pid=92056) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} May 23 12:40:52.929661 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4740ecaa-a011-4ebf-90b1-ee4983900143 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:40:52.931219 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4740ecaa-a011-4ebf-90b1-ee4983900143 None admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:40:52.936960 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 155/308] 10.222.0.30 () {56 vars in 1308 bytes} [Mon May 23 12:40:52 2022] GET /identity/v3/role_assignments?user.id=4c82e0529aea4c23a9556941da8b78db&scope.system=all&role.id=c223286ed21a4654a0f2b4e5551831c7 => generated 226 bytes in 167 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) May 23 12:40:52.938077 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0d2d567-db58-43c9-9e7f-65172157b69b None None] REQUEST_METHOD: `POST` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:40:52.938761 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0d2d567-db58-43c9-9e7f-65172157b69b None None] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:40:52.939396 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0d2d567-db58-43c9-9e7f-65172157b69b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:40:52.958939 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.auth.core [None req-f0d2d567-db58-43c9-9e7f-65172157b69b None None] MFA Rules not processed for user `0f7119f8fa5c4445977671140ded91a4`. Rule list: `[]` (Enabled: `True`). {{(pid=92056) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 12:40:52.960354 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.fernet_utils [None req-f0d2d567-db58-43c9-9e7f-65172157b69b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92056) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 12:40:53.056090 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 156/309] 10.222.0.30 () {58 vars in 1023 bytes} [Mon May 23 12:40:52 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 118 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) May 23 12:40:53.066091 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-102086ee-e6c5-4a25-b3ab-79c95714e4b6 None None] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:40:53.080043 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0d2d567-db58-43c9-9e7f-65172157b69b None None] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:40:53.121209 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3e0e9f7-9f2f-4c13-bbf8-27831de796b9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:40:53.121960 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3e0e9f7-9f2f-4c13-bbf8-27831de796b9 None admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:40:53.122253 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3e0e9f7-9f2f-4c13-bbf8-27831de796b9 None admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:40:53.122535 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3e0e9f7-9f2f-4c13-bbf8-27831de796b9 None admin] PATH_INFO: `/v3/roles` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:40:53.123607 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3e0e9f7-9f2f-4c13-bbf8-27831de796b9 None admin] RBAC: Adding query filter params (name=member) {{(pid=92057) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} May 23 12:40:53.123925 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3e0e9f7-9f2f-4c13-bbf8-27831de796b9 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:40:53.125837 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3e0e9f7-9f2f-4c13-bbf8-27831de796b9 None admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:40:53.141924 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 154/310] 10.222.0.30 () {56 vars in 1101 bytes} [Mon May 23 12:40:52 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 230 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) May 23 12:40:53.144357 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3e0e9f7-9f2f-4c13-bbf8-27831de796b9 None admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:40:53.202252 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=100 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:40:53.209725 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=100 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:40:53.220648 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0d2d567-db58-43c9-9e7f-65172157b69b None None] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:40:53.260588 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3e0e9f7-9f2f-4c13-bbf8-27831de796b9 None admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:40:53.267160 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb63b33b-f068-4631-80ae-84703f64040c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:40:53.268757 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb63b33b-f068-4631-80ae-84703f64040c None admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:40:53.269280 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb63b33b-f068-4631-80ae-84703f64040c None admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:40:53.269822 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb63b33b-f068-4631-80ae-84703f64040c None admin] PATH_INFO: `/v3/roles/reader` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:40:53.292541 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb63b33b-f068-4631-80ae-84703f64040c None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} May 23 12:40:53.303450 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a9d777c-aafe-4a34-81f5-f20ba7609e6e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:40:53.304685 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a9d777c-aafe-4a34-81f5-f20ba7609e6e None admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:40:53.305157 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a9d777c-aafe-4a34-81f5-f20ba7609e6e None admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:40:53.305620 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a9d777c-aafe-4a34-81f5-f20ba7609e6e None admin] PATH_INFO: `/v3/users/3d469b7f6b4142bb9ca60e30e9b2a1d5` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:40:53.306581 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application [None req-fb63b33b-f068-4631-80ae-84703f64040c None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. May 23 12:40:53.306581 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application Traceback (most recent call last): May 23 12:40:53.306581 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request May 23 12:40:53.306581 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application rv = self.dispatch_request() May 23 12:40:53.306581 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request May 23 12:40:53.306581 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) May 23 12:40:53.306581 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper May 23 12:40:53.306581 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) May 23 12:40:53.306581 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view May 23 12:40:53.306581 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) May 23 12:40:53.306581 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request May 23 12:40:53.306581 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) May 23 12:40:53.306581 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get May 23 12:40:53.306581 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self._get_role(role_id) May 23 12:40:53.306581 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role May 23 12:40:53.306581 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') May 23 12:40:53.307930 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call May 23 12:40:53.307930 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( May 23 12:40:53.307930 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data May 23 12:40:53.307930 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) May 23 12:40:53.307930 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped May 23 12:40:53.307930 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) May 23 12:40:53.307930 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun May 23 12:40:53.307930 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) May 23 12:40:53.307930 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func May 23 12:40:53.307930 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self.get_or_create( May 23 12:40:53.307930 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create May 23 12:40:53.307930 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application with Lock( May 23 12:40:53.307930 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ May 23 12:40:53.307930 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self._enter() May 23 12:40:53.307930 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter May 23 12:40:53.307930 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) May 23 12:40:53.308790 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create May 23 12:40:53.308790 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self.creator() May 23 12:40:53.308790 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value May 23 12:40:53.308790 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application created_value = creator( May 23 12:40:53.308790 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role May 23 12:40:53.308790 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) May 23 12:40:53.308790 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role May 23 12:40:53.308790 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() May 23 12:40:53.308790 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role May 23 12:40:53.308790 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) May 23 12:40:53.308790 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. May 23 12:40:53.308790 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application May 23 12:40:53.308790 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 157/311] 10.222.0.30 () {56 vars in 1099 bytes} [Mon May 23 12:40:53 2022] GET /identity/v3/roles/reader => generated 84 bytes in 229 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) May 23 12:40:53.310824 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a9d777c-aafe-4a34-81f5-f20ba7609e6e None admin] RBAC: Authorizing `identity:get_user(user_id=3d469b7f6b4142bb9ca60e30e9b2a1d5)` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:40:53.312311 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a9d777c-aafe-4a34-81f5-f20ba7609e6e None admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:40:53.313354 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb63b33b-f068-4631-80ae-84703f64040c None admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:40:53.314352 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 155/312] 10.222.0.30 () {56 vars in 1177 bytes} [Mon May 23 12:40:53 2022] GET /identity/v3/users/3d469b7f6b4142bb9ca60e30e9b2a1d5 => generated 293 bytes in 171 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) May 23 12:40:53.319332 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a9d777c-aafe-4a34-81f5-f20ba7609e6e None admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:40:53.430302 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb63b33b-f068-4631-80ae-84703f64040c None admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:40:53.435715 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=100 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:40:53.459769 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a9d777c-aafe-4a34-81f5-f20ba7609e6e None admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:40:53.478834 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ebc11d15-463a-4ff3-acd8-01255eb16f80 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:40:53.479969 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebc11d15-463a-4ff3-acd8-01255eb16f80 None admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:40:53.480560 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebc11d15-463a-4ff3-acd8-01255eb16f80 None admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:40:53.481108 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebc11d15-463a-4ff3-acd8-01255eb16f80 None admin] PATH_INFO: `/v3/roles` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:40:53.482132 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ebc11d15-463a-4ff3-acd8-01255eb16f80 None admin] RBAC: Adding query filter params (name=reader) {{(pid=92056) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} May 23 12:40:53.482687 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ebc11d15-463a-4ff3-acd8-01255eb16f80 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:40:53.484473 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ebc11d15-463a-4ff3-acd8-01255eb16f80 None admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:40:53.493952 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 158/313] 10.222.0.30 () {56 vars in 1101 bytes} [Mon May 23 12:40:53 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 181 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) May 23 12:40:53.499069 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ebc11d15-463a-4ff3-acd8-01255eb16f80 None admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:40:53.505929 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-853580ec-71d1-4db4-aaca-29882df71f75 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:40:53.505929 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-853580ec-71d1-4db4-aaca-29882df71f75 None admin] REQUEST_METHOD: `PUT` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:40:53.505929 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-853580ec-71d1-4db4-aaca-29882df71f75 None admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:40:53.506215 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-853580ec-71d1-4db4-aaca-29882df71f75 None admin] PATH_INFO: `/v3/system/users/3d469b7f6b4142bb9ca60e30e9b2a1d5/roles/80fb22507c9a4ff09d44d8f12f6fff1a` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:40:53.507530 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-853580ec-71d1-4db4-aaca-29882df71f75 None admin] RBAC: Authorizing `identity:create_system_grant_for_user(user_id=3d469b7f6b4142bb9ca60e30e9b2a1d5, role_id=80fb22507c9a4ff09d44d8f12f6fff1a)` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:40:53.508913 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-853580ec-71d1-4db4-aaca-29882df71f75 None admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:40:53.519971 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=100 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:40:53.602473 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 156/314] 10.222.0.30 () {58 vars in 1334 bytes} [Mon May 23 12:40:53 2022] PUT /identity/v3/system/users/3d469b7f6b4142bb9ca60e30e9b2a1d5/roles/80fb22507c9a4ff09d44d8f12f6fff1a => generated 0 bytes in 284 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) May 23 12:40:53.614069 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ebc11d15-463a-4ff3-acd8-01255eb16f80 None admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:40:53.659304 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79a3cb0e-bb2e-4a4a-bbb9-6d5d4cea103d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:40:53.660294 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79a3cb0e-bb2e-4a4a-bbb9-6d5d4cea103d None admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:40:53.660582 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79a3cb0e-bb2e-4a4a-bbb9-6d5d4cea103d None admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:40:53.660859 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79a3cb0e-bb2e-4a4a-bbb9-6d5d4cea103d None admin] PATH_INFO: `/v3/users/3f1c3ab2da314bff8c13f587a8d55a52` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:40:53.662196 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79a3cb0e-bb2e-4a4a-bbb9-6d5d4cea103d None admin] RBAC: Authorizing `identity:get_user(user_id=3f1c3ab2da314bff8c13f587a8d55a52)` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:40:53.663710 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79a3cb0e-bb2e-4a4a-bbb9-6d5d4cea103d None admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:40:53.665519 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 159/315] 10.222.0.30 () {56 vars in 1177 bytes} [Mon May 23 12:40:53 2022] GET /identity/v3/users/3f1c3ab2da314bff8c13f587a8d55a52 => generated 297 bytes in 167 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) May 23 12:40:53.672731 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-853580ec-71d1-4db4-aaca-29882df71f75 None admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:40:53.800415 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-853580ec-71d1-4db4-aaca-29882df71f75 None admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:40:53.838818 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eda76dd9-fdde-485b-b27f-0f59a8406993 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:40:53.839767 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eda76dd9-fdde-485b-b27f-0f59a8406993 None admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:40:53.840197 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eda76dd9-fdde-485b-b27f-0f59a8406993 None admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:40:53.840614 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eda76dd9-fdde-485b-b27f-0f59a8406993 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:40:53.857482 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:40:53.912316 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eda76dd9-fdde-485b-b27f-0f59a8406993 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:40:53.913996 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eda76dd9-fdde-485b-b27f-0f59a8406993 None admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:40:54.017186 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 157/316] 10.222.0.30 () {58 vars in 1282 bytes} [Mon May 23 12:40:53 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 345 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) May 23 12:40:54.022639 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79a3cb0e-bb2e-4a4a-bbb9-6d5d4cea103d None admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:40:54.115884 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=100 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:40:54.134049 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79a3cb0e-bb2e-4a4a-bbb9-6d5d4cea103d None admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:40:54.173186 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-505a0cb8-d12f-46c3-a385-3fcd6809e93c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:40:54.174326 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-505a0cb8-d12f-46c3-a385-3fcd6809e93c None admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:40:54.174745 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-505a0cb8-d12f-46c3-a385-3fcd6809e93c None admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:40:54.175154 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-505a0cb8-d12f-46c3-a385-3fcd6809e93c None admin] PATH_INFO: `/v3/projects/b39742b1d1b34dfe82a9ce4da39b7582` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:40:54.176503 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-505a0cb8-d12f-46c3-a385-3fcd6809e93c None admin] RBAC: Authorizing `identity:get_project(project_id=b39742b1d1b34dfe82a9ce4da39b7582)` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:40:54.178129 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-505a0cb8-d12f-46c3-a385-3fcd6809e93c None admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:40:54.180250 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 160/317] 10.222.0.30 () {56 vars in 1186 bytes} [Mon May 23 12:40:54 2022] GET /identity/v3/projects/b39742b1d1b34dfe82a9ce4da39b7582 => generated 302 bytes in 158 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) May 23 12:40:54.190991 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eda76dd9-fdde-485b-b27f-0f59a8406993 None admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:40:54.238429 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=100 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:40:54.296875 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eda76dd9-fdde-485b-b27f-0f59a8406993 None admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:40:54.332997 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca47c59e-95f2-4cc4-b7bd-1ea4bbcd83bf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:40:54.333935 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca47c59e-95f2-4cc4-b7bd-1ea4bbcd83bf None admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:40:54.334371 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca47c59e-95f2-4cc4-b7bd-1ea4bbcd83bf None admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:40:54.334780 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca47c59e-95f2-4cc4-b7bd-1ea4bbcd83bf None admin] PATH_INFO: `/v3/role_assignments` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:40:54.335753 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca47c59e-95f2-4cc4-b7bd-1ea4bbcd83bf None admin] RBAC: Adding query filter params (role.id=c223286ed21a4654a0f2b4e5551831c7, scope.project.id=b39742b1d1b34dfe82a9ce4da39b7582, user.id=3f1c3ab2da314bff8c13f587a8d55a52) {{(pid=92057) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} May 23 12:40:54.336201 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca47c59e-95f2-4cc4-b7bd-1ea4bbcd83bf None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:40:54.338322 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca47c59e-95f2-4cc4-b7bd-1ea4bbcd83bf None admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:40:54.345158 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 158/318] 10.222.0.30 () {56 vars in 1374 bytes} [Mon May 23 12:40:54 2022] GET /identity/v3/role_assignments?user.id=3f1c3ab2da314bff8c13f587a8d55a52&scope.project.id=b39742b1d1b34dfe82a9ce4da39b7582&role.id=c223286ed21a4654a0f2b4e5551831c7 => generated 608 bytes in 155 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) May 23 12:40:54.893055 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7592dfcc-8eb8-40e4-8c0a-ea0560317558 None None] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:40:54.893411 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7592dfcc-8eb8-40e4-8c0a-ea0560317558 None None] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:40:54.893607 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7592dfcc-8eb8-40e4-8c0a-ea0560317558 None None] PATH_INFO: `/` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:40:54.895149 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 161/319] 10.222.0.30 () {54 vars in 926 bytes} [Mon May 23 12:40:54 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) May 23 12:40:54.900927 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12b52d2d-d559-4ba4-9772-e3243ab46cd7 None None] REQUEST_METHOD: `POST` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:40:54.901221 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12b52d2d-d559-4ba4-9772-e3243ab46cd7 None None] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:40:54.901500 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12b52d2d-d559-4ba4-9772-e3243ab46cd7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:40:54.918355 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.auth.core [None req-12b52d2d-d559-4ba4-9772-e3243ab46cd7 None None] MFA Rules not processed for user `0f7119f8fa5c4445977671140ded91a4`. Rule list: `[]` (Enabled: `True`). {{(pid=92057) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 12:40:54.919539 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.fernet_utils [None req-12b52d2d-d559-4ba4-9772-e3243ab46cd7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92057) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 12:40:54.989140 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 159/320] 10.222.0.30 () {58 vars in 1023 bytes} [Mon May 23 12:40:54 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 90 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) May 23 12:40:55.012689 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7592dfcc-8eb8-40e4-8c0a-ea0560317558 None None] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:40:55.119612 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7592dfcc-8eb8-40e4-8c0a-ea0560317558 None None] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:40:55.152954 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3fb163de-02af-49ff-9d34-de931d71b314 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:40:55.153744 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fb163de-02af-49ff-9d34-de931d71b314 None admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:40:55.154313 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fb163de-02af-49ff-9d34-de931d71b314 None admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:40:55.154587 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fb163de-02af-49ff-9d34-de931d71b314 None admin] PATH_INFO: `/v3/domains/default` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:40:55.155737 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3fb163de-02af-49ff-9d34-de931d71b314 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:40:55.157221 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3fb163de-02af-49ff-9d34-de931d71b314 None admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:40:55.159111 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 162/321] 10.222.0.30 () {56 vars in 1108 bytes} [Mon May 23 12:40:55 2022] GET /identity/v3/domains/default => generated 199 bytes in 147 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) May 23 12:40:55.164384 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12b52d2d-d559-4ba4-9772-e3243ab46cd7 None None] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:40:55.203729 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5604eb2-8602-4262-baea-7b67f6366ca6 None None] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:40:55.203913 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5604eb2-8602-4262-baea-7b67f6366ca6 None None] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:40:55.204187 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5604eb2-8602-4262-baea-7b67f6366ca6 None None] PATH_INFO: `/` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:40:55.204766 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=100 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:40:55.206491 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 163/322] 10.222.0.30 () {54 vars in 926 bytes} [Mon May 23 12:40:55 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) May 23 12:40:55.211622 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4cc4c0f-1b45-4b76-8db7-2a7792e591d2 None None] REQUEST_METHOD: `POST` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:40:55.211896 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4cc4c0f-1b45-4b76-8db7-2a7792e591d2 None None] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:40:55.212176 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4cc4c0f-1b45-4b76-8db7-2a7792e591d2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:40:55.231437 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.auth.core [None req-c4cc4c0f-1b45-4b76-8db7-2a7792e591d2 None None] MFA Rules not processed for user `0f7119f8fa5c4445977671140ded91a4`. Rule list: `[]` (Enabled: `True`). {{(pid=92056) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 12:40:55.232630 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.fernet_utils [None req-c4cc4c0f-1b45-4b76-8db7-2a7792e591d2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92056) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 12:40:55.248673 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=100 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:40:55.272238 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12b52d2d-d559-4ba4-9772-e3243ab46cd7 None None] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:40:55.309908 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 164/323] 10.222.0.30 () {58 vars in 1023 bytes} [Mon May 23 12:40:55 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 93 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) May 23 12:40:55.312160 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b7094c0-6623-4e01-80b2-fbaf665ca125 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:40:55.312825 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b7094c0-6623-4e01-80b2-fbaf665ca125 None admin] REQUEST_METHOD: `POST` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:40:55.313105 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b7094c0-6623-4e01-80b2-fbaf665ca125 None admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:40:55.313376 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b7094c0-6623-4e01-80b2-fbaf665ca125 None admin] PATH_INFO: `/v3/groups` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:40:55.314095 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b7094c0-6623-4e01-80b2-fbaf665ca125 None admin] RBAC: Authorizing `identity:create_group()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:40:55.315551 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b7094c0-6623-4e01-80b2-fbaf665ca125 None admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:40:55.327929 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4cc4c0f-1b45-4b76-8db7-2a7792e591d2 None None] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:40:55.413341 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 160/324] 10.222.0.30 () {60 vars in 1134 bytes} [Mon May 23 12:40:55 2022] POST /identity/v3/groups => generated 226 bytes in 249 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) May 23 12:40:55.438326 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4cc4c0f-1b45-4b76-8db7-2a7792e591d2 None None] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:40:55.475644 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34a246d0-0988-4c27-85f2-0e6347585b97 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:40:55.476458 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34a246d0-0988-4c27-85f2-0e6347585b97 None admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:40:55.476739 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34a246d0-0988-4c27-85f2-0e6347585b97 None admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:40:55.477009 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34a246d0-0988-4c27-85f2-0e6347585b97 None admin] PATH_INFO: `/v3/roles/reader` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:40:55.489616 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34a246d0-0988-4c27-85f2-0e6347585b97 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} May 23 12:40:55.498861 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application [None req-34a246d0-0988-4c27-85f2-0e6347585b97 None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. May 23 12:40:55.498861 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application Traceback (most recent call last): May 23 12:40:55.498861 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request May 23 12:40:55.498861 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application rv = self.dispatch_request() May 23 12:40:55.498861 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request May 23 12:40:55.498861 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) May 23 12:40:55.498861 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper May 23 12:40:55.498861 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) May 23 12:40:55.498861 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view May 23 12:40:55.498861 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) May 23 12:40:55.498861 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request May 23 12:40:55.498861 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) May 23 12:40:55.498861 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get May 23 12:40:55.498861 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self._get_role(role_id) May 23 12:40:55.498861 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role May 23 12:40:55.498861 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') May 23 12:40:55.499816 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call May 23 12:40:55.499816 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( May 23 12:40:55.499816 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data May 23 12:40:55.499816 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) May 23 12:40:55.499816 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped May 23 12:40:55.499816 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) May 23 12:40:55.499816 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun May 23 12:40:55.499816 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) May 23 12:40:55.499816 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func May 23 12:40:55.499816 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self.get_or_create( May 23 12:40:55.499816 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create May 23 12:40:55.499816 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application with Lock( May 23 12:40:55.499816 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ May 23 12:40:55.499816 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self._enter() May 23 12:40:55.499816 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter May 23 12:40:55.499816 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) May 23 12:40:55.500708 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create May 23 12:40:55.500708 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self.creator() May 23 12:40:55.500708 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value May 23 12:40:55.500708 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application created_value = creator( May 23 12:40:55.500708 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role May 23 12:40:55.500708 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) May 23 12:40:55.500708 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role May 23 12:40:55.500708 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() May 23 12:40:55.500708 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role May 23 12:40:55.500708 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) May 23 12:40:55.500708 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. May 23 12:40:55.500708 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application May 23 12:40:55.500708 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 165/325] 10.222.0.30 () {56 vars in 1099 bytes} [Mon May 23 12:40:55 2022] GET /identity/v3/roles/reader => generated 84 bytes in 173 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) May 23 12:40:55.506523 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b7094c0-6623-4e01-80b2-fbaf665ca125 None admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:40:55.614079 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:40:55.618421 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b7094c0-6623-4e01-80b2-fbaf665ca125 None admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:40:55.655469 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4a68ce3-7821-4f27-8278-65d5c0d05b56 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:40:55.656199 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4a68ce3-7821-4f27-8278-65d5c0d05b56 None admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:40:55.656487 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4a68ce3-7821-4f27-8278-65d5c0d05b56 None admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:40:55.656760 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4a68ce3-7821-4f27-8278-65d5c0d05b56 None admin] PATH_INFO: `/v3/roles` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:40:55.657501 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4a68ce3-7821-4f27-8278-65d5c0d05b56 None admin] RBAC: Adding query filter params (name=reader) {{(pid=92057) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} May 23 12:40:55.657948 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4a68ce3-7821-4f27-8278-65d5c0d05b56 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:40:55.659409 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4a68ce3-7821-4f27-8278-65d5c0d05b56 None admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:40:55.668633 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 161/326] 10.222.0.30 () {56 vars in 1101 bytes} [Mon May 23 12:40:55 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 163 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) May 23 12:40:55.674161 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34a246d0-0988-4c27-85f2-0e6347585b97 None admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:40:55.696851 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=100 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:40:55.802542 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34a246d0-0988-4c27-85f2-0e6347585b97 None admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:40:55.846725 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c9baba2-8ab3-4ec3-b944-9f72f13a709e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:40:55.847604 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c9baba2-8ab3-4ec3-b944-9f72f13a709e None admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:40:55.847883 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c9baba2-8ab3-4ec3-b944-9f72f13a709e None admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:40:55.848154 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c9baba2-8ab3-4ec3-b944-9f72f13a709e None admin] PATH_INFO: `/v3/users/4c82e0529aea4c23a9556941da8b78db` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:40:55.849398 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c9baba2-8ab3-4ec3-b944-9f72f13a709e None admin] RBAC: Authorizing `identity:get_user(user_id=4c82e0529aea4c23a9556941da8b78db)` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:40:55.850888 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c9baba2-8ab3-4ec3-b944-9f72f13a709e None admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:40:55.852809 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 166/327] 10.222.0.30 () {56 vars in 1177 bytes} [Mon May 23 12:40:55 2022] GET /identity/v3/users/4c82e0529aea4c23a9556941da8b78db => generated 293 bytes in 179 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) May 23 12:40:55.858627 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4a68ce3-7821-4f27-8278-65d5c0d05b56 None admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:40:55.942373 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-889406f7-4aad-48fc-8127-1793adbacfde None None] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:40:55.942710 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-889406f7-4aad-48fc-8127-1793adbacfde None None] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:40:55.942931 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-889406f7-4aad-48fc-8127-1793adbacfde None None] PATH_INFO: `/` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:40:55.944321 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 167/328] 10.222.0.30 () {54 vars in 926 bytes} [Mon May 23 12:40:55 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) May 23 12:40:55.949997 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6e8804f-4eba-488b-b90b-4ceae65425de None None] REQUEST_METHOD: `POST` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:40:55.950311 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6e8804f-4eba-488b-b90b-4ceae65425de None None] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:40:55.950553 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6e8804f-4eba-488b-b90b-4ceae65425de None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:40:55.971246 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.auth.core [None req-c6e8804f-4eba-488b-b90b-4ceae65425de None None] MFA Rules not processed for user `0f7119f8fa5c4445977671140ded91a4`. Rule list: `[]` (Enabled: `True`). {{(pid=92056) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 12:40:55.972448 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.fernet_utils [None req-c6e8804f-4eba-488b-b90b-4ceae65425de None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92056) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 12:40:55.988419 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4a68ce3-7821-4f27-8278-65d5c0d05b56 None admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:40:56.040420 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82932a0b-73c0-4392-8a4b-a2c52c76dba5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:40:56.041897 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82932a0b-73c0-4392-8a4b-a2c52c76dba5 None admin] REQUEST_METHOD: `PUT` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:40:56.042356 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82932a0b-73c0-4392-8a4b-a2c52c76dba5 None admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:40:56.042644 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82932a0b-73c0-4392-8a4b-a2c52c76dba5 None admin] PATH_INFO: `/v3/system/users/4c82e0529aea4c23a9556941da8b78db/roles/c223286ed21a4654a0f2b4e5551831c7` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:40:56.047307 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82932a0b-73c0-4392-8a4b-a2c52c76dba5 None admin] RBAC: Authorizing `identity:create_system_grant_for_user(user_id=4c82e0529aea4c23a9556941da8b78db, role_id=c223286ed21a4654a0f2b4e5551831c7)` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:40:56.048764 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82932a0b-73c0-4392-8a4b-a2c52c76dba5 None admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:40:56.073805 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 168/329] 10.222.0.30 () {58 vars in 1023 bytes} [Mon May 23 12:40:55 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 124 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) May 23 12:40:56.098265 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6e8804f-4eba-488b-b90b-4ceae65425de None None] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:40:56.109613 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=100 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:40:56.197980 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 162/330] 10.222.0.30 () {58 vars in 1334 bytes} [Mon May 23 12:40:55 2022] PUT /identity/v3/system/users/4c82e0529aea4c23a9556941da8b78db/roles/c223286ed21a4654a0f2b4e5551831c7 => generated 0 bytes in 340 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) May 23 12:40:56.243353 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6e8804f-4eba-488b-b90b-4ceae65425de None None] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:40:56.284002 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c1476f4-c06c-49d5-bfa4-fefb0bc579a7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:40:56.284811 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c1476f4-c06c-49d5-bfa4-fefb0bc579a7 None admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:40:56.285094 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c1476f4-c06c-49d5-bfa4-fefb0bc579a7 None admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:40:56.285369 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c1476f4-c06c-49d5-bfa4-fefb0bc579a7 None admin] PATH_INFO: `/v3/roles/member` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:40:56.296912 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c1476f4-c06c-49d5-bfa4-fefb0bc579a7 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} May 23 12:40:56.304911 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application [None req-8c1476f4-c06c-49d5-bfa4-fefb0bc579a7 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. May 23 12:40:56.304911 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application Traceback (most recent call last): May 23 12:40:56.304911 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request May 23 12:40:56.304911 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application rv = self.dispatch_request() May 23 12:40:56.304911 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request May 23 12:40:56.304911 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) May 23 12:40:56.304911 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper May 23 12:40:56.304911 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) May 23 12:40:56.304911 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view May 23 12:40:56.304911 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) May 23 12:40:56.304911 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request May 23 12:40:56.304911 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) May 23 12:40:56.304911 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get May 23 12:40:56.304911 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self._get_role(role_id) May 23 12:40:56.304911 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role May 23 12:40:56.304911 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') May 23 12:40:56.305965 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call May 23 12:40:56.305965 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( May 23 12:40:56.305965 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data May 23 12:40:56.305965 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) May 23 12:40:56.305965 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped May 23 12:40:56.305965 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) May 23 12:40:56.305965 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun May 23 12:40:56.305965 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) May 23 12:40:56.305965 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func May 23 12:40:56.305965 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self.get_or_create( May 23 12:40:56.305965 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create May 23 12:40:56.305965 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application with Lock( May 23 12:40:56.305965 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ May 23 12:40:56.305965 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self._enter() May 23 12:40:56.305965 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter May 23 12:40:56.305965 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) May 23 12:40:56.307027 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create May 23 12:40:56.307027 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self.creator() May 23 12:40:56.307027 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value May 23 12:40:56.307027 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application created_value = creator( May 23 12:40:56.307027 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role May 23 12:40:56.307027 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) May 23 12:40:56.307027 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role May 23 12:40:56.307027 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() May 23 12:40:56.307027 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role May 23 12:40:56.307027 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) May 23 12:40:56.307027 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. May 23 12:40:56.307027 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application May 23 12:40:56.307027 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 169/331] 10.222.0.30 () {56 vars in 1099 bytes} [Mon May 23 12:40:56 2022] GET /identity/v3/roles/member => generated 84 bytes in 209 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) May 23 12:40:56.315061 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82932a0b-73c0-4392-8a4b-a2c52c76dba5 None admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:40:56.363074 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:40:56.469302 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82932a0b-73c0-4392-8a4b-a2c52c76dba5 None admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:40:56.512119 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a518c7e-4865-4f3d-9e53-9cedcbe9f972 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:40:56.512800 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a518c7e-4865-4f3d-9e53-9cedcbe9f972 None admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:40:56.513090 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a518c7e-4865-4f3d-9e53-9cedcbe9f972 None admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:40:56.513368 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a518c7e-4865-4f3d-9e53-9cedcbe9f972 None admin] PATH_INFO: `/v3/roles` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:40:56.514320 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a518c7e-4865-4f3d-9e53-9cedcbe9f972 None admin] RBAC: Adding query filter params (name=member) {{(pid=92057) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} May 23 12:40:56.514723 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a518c7e-4865-4f3d-9e53-9cedcbe9f972 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:40:56.516114 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a518c7e-4865-4f3d-9e53-9cedcbe9f972 None admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:40:56.525554 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 163/332] 10.222.0.30 () {56 vars in 1101 bytes} [Mon May 23 12:40:56 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 211 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) May 23 12:40:56.531405 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c1476f4-c06c-49d5-bfa4-fefb0bc579a7 None admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:40:56.620686 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=100 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:40:56.662264 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c1476f4-c06c-49d5-bfa4-fefb0bc579a7 None admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:40:56.717528 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e6d5d44-b3b0-4915-9582-2c11361d2456 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:40:56.718450 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e6d5d44-b3b0-4915-9582-2c11361d2456 None admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:40:56.718733 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e6d5d44-b3b0-4915-9582-2c11361d2456 None admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:40:56.719009 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e6d5d44-b3b0-4915-9582-2c11361d2456 None admin] PATH_INFO: `/v3/users/3d469b7f6b4142bb9ca60e30e9b2a1d5` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:40:56.720236 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e6d5d44-b3b0-4915-9582-2c11361d2456 None admin] RBAC: Authorizing `identity:get_user(user_id=3d469b7f6b4142bb9ca60e30e9b2a1d5)` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:40:56.721702 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e6d5d44-b3b0-4915-9582-2c11361d2456 None admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:40:56.723699 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 170/333] 10.222.0.30 () {56 vars in 1177 bytes} [Mon May 23 12:40:56 2022] GET /identity/v3/users/3d469b7f6b4142bb9ca60e30e9b2a1d5 => generated 293 bytes in 193 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) May 23 12:40:56.729756 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a518c7e-4865-4f3d-9e53-9cedcbe9f972 None admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:40:56.848716 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=100 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:40:56.867161 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a518c7e-4865-4f3d-9e53-9cedcbe9f972 None admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:40:56.913792 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0037d144-ed47-4944-b943-3dbf82ae7a42 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:40:56.914505 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0037d144-ed47-4944-b943-3dbf82ae7a42 None admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:40:56.914860 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0037d144-ed47-4944-b943-3dbf82ae7a42 None admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:40:56.915176 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0037d144-ed47-4944-b943-3dbf82ae7a42 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:40:56.916655 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0037d144-ed47-4944-b943-3dbf82ae7a42 None admin] RBAC: Adding query filter params (role.id=80fb22507c9a4ff09d44d8f12f6fff1a, user.id=3d469b7f6b4142bb9ca60e30e9b2a1d5, scope.system=all) {{(pid=92057) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} May 23 12:40:56.916978 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0037d144-ed47-4944-b943-3dbf82ae7a42 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:40:56.918577 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0037d144-ed47-4944-b943-3dbf82ae7a42 None admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:40:56.938192 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 164/334] 10.222.0.30 () {56 vars in 1308 bytes} [Mon May 23 12:40:56 2022] GET /identity/v3/role_assignments?user.id=3d469b7f6b4142bb9ca60e30e9b2a1d5&scope.system=all&role.id=80fb22507c9a4ff09d44d8f12f6fff1a => generated 510 bytes in 209 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) May 23 12:40:58.631854 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0d94388-19d4-4fa6-81cf-c0e855fd82e9 None None] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:40:58.632943 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0d94388-19d4-4fa6-81cf-c0e855fd82e9 None None] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:40:58.633438 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0d94388-19d4-4fa6-81cf-c0e855fd82e9 None None] PATH_INFO: `/` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:40:58.636040 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 171/335] 10.222.0.30 () {54 vars in 926 bytes} [Mon May 23 12:40:58 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) May 23 12:40:58.641224 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92f2eae4-1f76-4558-9b94-912b452fe26d None None] REQUEST_METHOD: `POST` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:40:58.642169 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92f2eae4-1f76-4558-9b94-912b452fe26d None None] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:40:58.642658 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92f2eae4-1f76-4558-9b94-912b452fe26d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:40:58.662151 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.auth.core [None req-92f2eae4-1f76-4558-9b94-912b452fe26d None None] MFA Rules not processed for user `0f7119f8fa5c4445977671140ded91a4`. Rule list: `[]` (Enabled: `True`). {{(pid=92057) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 12:40:58.663692 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.fernet_utils [None req-92f2eae4-1f76-4558-9b94-912b452fe26d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92057) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 12:40:58.766972 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 165/336] 10.222.0.30 () {58 vars in 1023 bytes} [Mon May 23 12:40:58 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 127 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) May 23 12:40:58.808928 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0d94388-19d4-4fa6-81cf-c0e855fd82e9 None None] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:40:58.942671 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0d94388-19d4-4fa6-81cf-c0e855fd82e9 None None] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:40:58.984938 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bcc82455-6616-4360-90f1-a5d9cb1311da None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:40:58.986183 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcc82455-6616-4360-90f1-a5d9cb1311da None admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:40:58.986677 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcc82455-6616-4360-90f1-a5d9cb1311da None admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:40:58.987155 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcc82455-6616-4360-90f1-a5d9cb1311da None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:40:58.990468 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=100 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:40:59.007851 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bcc82455-6616-4360-90f1-a5d9cb1311da None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} May 23 12:40:59.021992 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application [None req-bcc82455-6616-4360-90f1-a5d9cb1311da None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. May 23 12:40:59.021992 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application Traceback (most recent call last): May 23 12:40:59.021992 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request May 23 12:40:59.021992 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application rv = self.dispatch_request() May 23 12:40:59.021992 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request May 23 12:40:59.021992 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) May 23 12:40:59.021992 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper May 23 12:40:59.021992 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) May 23 12:40:59.021992 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view May 23 12:40:59.021992 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) May 23 12:40:59.021992 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request May 23 12:40:59.021992 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) May 23 12:40:59.021992 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get May 23 12:40:59.021992 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self._get_role(role_id) May 23 12:40:59.021992 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role May 23 12:40:59.021992 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') May 23 12:40:59.024162 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call May 23 12:40:59.024162 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( May 23 12:40:59.024162 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data May 23 12:40:59.024162 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) May 23 12:40:59.024162 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped May 23 12:40:59.024162 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) May 23 12:40:59.024162 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun May 23 12:40:59.024162 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) May 23 12:40:59.024162 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func May 23 12:40:59.024162 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self.get_or_create( May 23 12:40:59.024162 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create May 23 12:40:59.024162 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application with Lock( May 23 12:40:59.024162 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ May 23 12:40:59.024162 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self._enter() May 23 12:40:59.024162 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter May 23 12:40:59.024162 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) May 23 12:40:59.027115 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create May 23 12:40:59.027115 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self.creator() May 23 12:40:59.027115 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value May 23 12:40:59.027115 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application created_value = creator( May 23 12:40:59.027115 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role May 23 12:40:59.027115 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) May 23 12:40:59.027115 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role May 23 12:40:59.027115 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() May 23 12:40:59.027115 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role May 23 12:40:59.027115 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) May 23 12:40:59.027115 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. May 23 12:40:59.027115 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application May 23 12:40:59.027115 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 172/337] 10.222.0.30 () {56 vars in 1114 bytes} [Mon May 23 12:40:58 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 215 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) May 23 12:40:59.033910 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92f2eae4-1f76-4558-9b94-912b452fe26d None None] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:40:59.073819 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d6b257c-0ec8-446c-823c-d2a460d1e44d None None] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:40:59.074189 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d6b257c-0ec8-446c-823c-d2a460d1e44d None None] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:40:59.074398 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d6b257c-0ec8-446c-823c-d2a460d1e44d None None] PATH_INFO: `/` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:40:59.076054 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 173/338] 10.222.0.30 () {54 vars in 926 bytes} [Mon May 23 12:40:59 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) May 23 12:40:59.081714 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3710112c-2651-40c4-8df7-6a9449d6976b None None] REQUEST_METHOD: `POST` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:40:59.082016 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3710112c-2651-40c4-8df7-6a9449d6976b None None] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:40:59.082288 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3710112c-2651-40c4-8df7-6a9449d6976b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:40:59.117922 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.auth.core [None req-3710112c-2651-40c4-8df7-6a9449d6976b None None] MFA Rules not processed for user `0f7119f8fa5c4445977671140ded91a4`. Rule list: `[]` (Enabled: `True`). {{(pid=92056) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 12:40:59.123767 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.fernet_utils [None req-3710112c-2651-40c4-8df7-6a9449d6976b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92056) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 12:40:59.229199 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=100 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:40:59.281865 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92f2eae4-1f76-4558-9b94-912b452fe26d None None] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:40:59.316462 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 174/339] 10.222.0.30 () {58 vars in 1023 bytes} [Mon May 23 12:40:59 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 235 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) May 23 12:40:59.325847 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99ee6e4c-c793-463f-a779-beb140dd1a65 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:40:59.326608 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99ee6e4c-c793-463f-a779-beb140dd1a65 None admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:40:59.326906 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99ee6e4c-c793-463f-a779-beb140dd1a65 None admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:40:59.327181 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99ee6e4c-c793-463f-a779-beb140dd1a65 None admin] PATH_INFO: `/v3/roles` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:40:59.328003 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99ee6e4c-c793-463f-a779-beb140dd1a65 None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=92057) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} May 23 12:40:59.328310 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99ee6e4c-c793-463f-a779-beb140dd1a65 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:40:59.329730 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99ee6e4c-c793-463f-a779-beb140dd1a65 None admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:40:59.336776 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b55cf28f-0e65-4a17-872e-72741a23d843 None None] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:40:59.337097 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b55cf28f-0e65-4a17-872e-72741a23d843 None None] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:40:59.337954 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b55cf28f-0e65-4a17-872e-72741a23d843 None None] PATH_INFO: `/` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:40:59.339823 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 175/340] 10.222.0.30 () {54 vars in 926 bytes} [Mon May 23 12:40:59 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) May 23 12:40:59.340083 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 166/341] 10.222.0.30 () {56 vars in 1111 bytes} [Mon May 23 12:40:59 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 310 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) May 23 12:40:59.345754 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99ee6e4c-c793-463f-a779-beb140dd1a65 None admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:40:59.346288 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bf9444c-61f4-483c-978b-d992aea6da27 None None] REQUEST_METHOD: `POST` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:40:59.346588 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bf9444c-61f4-483c-978b-d992aea6da27 None None] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:40:59.346870 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bf9444c-61f4-483c-978b-d992aea6da27 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:40:59.373505 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.auth.core [None req-7bf9444c-61f4-483c-978b-d992aea6da27 None None] MFA Rules not processed for user `0f7119f8fa5c4445977671140ded91a4`. Rule list: `[]` (Enabled: `True`). {{(pid=92056) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 12:40:59.375151 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.fernet_utils [None req-7bf9444c-61f4-483c-978b-d992aea6da27 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92056) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 12:40:59.468170 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99ee6e4c-c793-463f-a779-beb140dd1a65 None admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:40:59.507616 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6d4ea8d-2d55-4132-9aca-e0c1cf03d08a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:40:59.508174 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6d4ea8d-2d55-4132-9aca-e0c1cf03d08a None admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:40:59.508459 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6d4ea8d-2d55-4132-9aca-e0c1cf03d08a None admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:40:59.508737 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6d4ea8d-2d55-4132-9aca-e0c1cf03d08a None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:40:59.516566 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 176/342] 10.222.0.30 () {58 vars in 1023 bytes} [Mon May 23 12:40:59 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 171 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) May 23 12:40:59.517420 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7bf9444c-61f4-483c-978b-d992aea6da27 None None] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:40:59.520887 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=100 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:40:59.598266 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6d4ea8d-2d55-4132-9aca-e0c1cf03d08a None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:40:59.599862 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6d4ea8d-2d55-4132-9aca-e0c1cf03d08a None admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:40:59.610653 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=100 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:40:59.658513 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7bf9444c-61f4-483c-978b-d992aea6da27 None None] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:40:59.698067 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d10fb1d0-f969-446d-bf4c-a39657e845f2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:40:59.699013 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d10fb1d0-f969-446d-bf4c-a39657e845f2 None admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:40:59.699524 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d10fb1d0-f969-446d-bf4c-a39657e845f2 None admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:40:59.699993 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d10fb1d0-f969-446d-bf4c-a39657e845f2 None admin] PATH_INFO: `/v3/roles/reader` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:40:59.712238 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d10fb1d0-f969-446d-bf4c-a39657e845f2 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} May 23 12:40:59.720659 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application [None req-d10fb1d0-f969-446d-bf4c-a39657e845f2 None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. May 23 12:40:59.720659 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application Traceback (most recent call last): May 23 12:40:59.720659 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request May 23 12:40:59.720659 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application rv = self.dispatch_request() May 23 12:40:59.720659 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request May 23 12:40:59.720659 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) May 23 12:40:59.720659 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper May 23 12:40:59.720659 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) May 23 12:40:59.720659 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view May 23 12:40:59.720659 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) May 23 12:40:59.720659 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request May 23 12:40:59.720659 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) May 23 12:40:59.720659 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get May 23 12:40:59.720659 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self._get_role(role_id) May 23 12:40:59.720659 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role May 23 12:40:59.720659 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') May 23 12:40:59.721574 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call May 23 12:40:59.721574 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( May 23 12:40:59.721574 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data May 23 12:40:59.721574 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) May 23 12:40:59.721574 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped May 23 12:40:59.721574 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) May 23 12:40:59.721574 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun May 23 12:40:59.721574 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) May 23 12:40:59.721574 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func May 23 12:40:59.721574 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self.get_or_create( May 23 12:40:59.721574 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create May 23 12:40:59.721574 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application with Lock( May 23 12:40:59.721574 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ May 23 12:40:59.721574 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self._enter() May 23 12:40:59.721574 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter May 23 12:40:59.721574 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) May 23 12:40:59.722432 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create May 23 12:40:59.722432 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self.creator() May 23 12:40:59.722432 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value May 23 12:40:59.722432 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application created_value = creator( May 23 12:40:59.722432 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role May 23 12:40:59.722432 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) May 23 12:40:59.722432 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role May 23 12:40:59.722432 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() May 23 12:40:59.722432 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role May 23 12:40:59.722432 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) May 23 12:40:59.722432 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. May 23 12:40:59.722432 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application May 23 12:40:59.725306 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 177/343] 10.222.0.30 () {56 vars in 1099 bytes} [Mon May 23 12:40:59 2022] GET /identity/v3/roles/reader => generated 84 bytes in 208 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) May 23 12:40:59.726806 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92c92cf6-53b9-4806-bc2c-548d5714f96c None None] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:40:59.727366 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92c92cf6-53b9-4806-bc2c-548d5714f96c None None] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:40:59.727923 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92c92cf6-53b9-4806-bc2c-548d5714f96c None None] PATH_INFO: `/` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:40:59.729602 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 167/344] 10.222.0.30 () {58 vars in 1282 bytes} [Mon May 23 12:40:59 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 384 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) May 23 12:40:59.730546 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6d4ea8d-2d55-4132-9aca-e0c1cf03d08a None admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:40:59.730794 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 178/345] 10.222.0.30 () {54 vars in 926 bytes} [Mon May 23 12:40:59 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) May 23 12:40:59.736263 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92c92cf6-53b9-4806-bc2c-548d5714f96c None None] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:40:59.769824 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=100 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:40:59.847966 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6d4ea8d-2d55-4132-9aca-e0c1cf03d08a None admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:40:59.855365 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92c92cf6-53b9-4806-bc2c-548d5714f96c None None] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:40:59.874398 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=100 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:40:59.888106 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a85caf5-eeb8-41dc-8c76-fb91a3fa479a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:40:59.888988 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a85caf5-eeb8-41dc-8c76-fb91a3fa479a None admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:40:59.889304 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a85caf5-eeb8-41dc-8c76-fb91a3fa479a None admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:40:59.889548 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a85caf5-eeb8-41dc-8c76-fb91a3fa479a None admin] PATH_INFO: `/v3/roles/member` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:40:59.896918 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-308bd214-1a83-4e7e-828f-8575445a48fd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:40:59.898329 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-308bd214-1a83-4e7e-828f-8575445a48fd None admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:40:59.898847 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-308bd214-1a83-4e7e-828f-8575445a48fd None admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:40:59.899379 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-308bd214-1a83-4e7e-828f-8575445a48fd None admin] PATH_INFO: `/v3/projects/486401ae2e5f40d5a5f060242896ae53` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:40:59.901071 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-308bd214-1a83-4e7e-828f-8575445a48fd None admin] RBAC: Authorizing `identity:get_project(project_id=486401ae2e5f40d5a5f060242896ae53)` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:40:59.901298 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a85caf5-eeb8-41dc-8c76-fb91a3fa479a None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} May 23 12:40:59.902950 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-308bd214-1a83-4e7e-828f-8575445a48fd None admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:40:59.906007 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 179/346] 10.222.0.30 () {56 vars in 1186 bytes} [Mon May 23 12:40:59 2022] GET /identity/v3/projects/486401ae2e5f40d5a5f060242896ae53 => generated 298 bytes in 170 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) May 23 12:40:59.907421 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2be5876c-bff5-463f-b745-eb96fe5531fb None None] REQUEST_METHOD: `POST` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:40:59.907906 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2be5876c-bff5-463f-b745-eb96fe5531fb None None] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:40:59.908371 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2be5876c-bff5-463f-b745-eb96fe5531fb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:40:59.910605 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application [None req-3a85caf5-eeb8-41dc-8c76-fb91a3fa479a None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. May 23 12:40:59.910605 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application Traceback (most recent call last): May 23 12:40:59.910605 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request May 23 12:40:59.910605 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application rv = self.dispatch_request() May 23 12:40:59.910605 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request May 23 12:40:59.910605 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) May 23 12:40:59.910605 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper May 23 12:40:59.910605 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) May 23 12:40:59.910605 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view May 23 12:40:59.910605 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) May 23 12:40:59.910605 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request May 23 12:40:59.910605 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) May 23 12:40:59.910605 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get May 23 12:40:59.910605 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self._get_role(role_id) May 23 12:40:59.910605 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role May 23 12:40:59.910605 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') May 23 12:40:59.911606 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call May 23 12:40:59.911606 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( May 23 12:40:59.911606 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data May 23 12:40:59.911606 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) May 23 12:40:59.911606 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped May 23 12:40:59.911606 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) May 23 12:40:59.911606 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun May 23 12:40:59.911606 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) May 23 12:40:59.911606 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func May 23 12:40:59.911606 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self.get_or_create( May 23 12:40:59.911606 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create May 23 12:40:59.911606 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application with Lock( May 23 12:40:59.911606 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ May 23 12:40:59.911606 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self._enter() May 23 12:40:59.911606 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter May 23 12:40:59.911606 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) May 23 12:40:59.912431 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create May 23 12:40:59.912431 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self.creator() May 23 12:40:59.912431 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value May 23 12:40:59.912431 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application created_value = creator( May 23 12:40:59.912431 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role May 23 12:40:59.912431 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) May 23 12:40:59.912431 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role May 23 12:40:59.912431 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() May 23 12:40:59.912431 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role May 23 12:40:59.912431 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) May 23 12:40:59.912431 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. May 23 12:40:59.912431 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application May 23 12:40:59.912431 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 168/347] 10.222.0.30 () {56 vars in 1099 bytes} [Mon May 23 12:40:59 2022] GET /identity/v3/roles/member => generated 84 bytes in 182 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) May 23 12:40:59.913102 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a85caf5-eeb8-41dc-8c76-fb91a3fa479a None admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:40:59.931646 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.auth.core [None req-2be5876c-bff5-463f-b745-eb96fe5531fb None None] MFA Rules not processed for user `0f7119f8fa5c4445977671140ded91a4`. Rule list: `[]` (Enabled: `True`). {{(pid=92056) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 12:40:59.933319 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.fernet_utils [None req-2be5876c-bff5-463f-b745-eb96fe5531fb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92056) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 12:41:00.008333 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 180/348] 10.222.0.30 () {58 vars in 1023 bytes} [Mon May 23 12:40:59 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 101 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) May 23 12:41:00.010459 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2be5876c-bff5-463f-b745-eb96fe5531fb None None] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:41:00.031381 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=100 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:41:00.037711 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a85caf5-eeb8-41dc-8c76-fb91a3fa479a None admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:41:00.087386 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f551424-2b3f-471a-8701-c3d4894f4d35 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:41:00.091951 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f551424-2b3f-471a-8701-c3d4894f4d35 None admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:41:00.092369 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f551424-2b3f-471a-8701-c3d4894f4d35 None admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:41:00.092666 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f551424-2b3f-471a-8701-c3d4894f4d35 None admin] PATH_INFO: `/v3/roles` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:41:00.094566 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f551424-2b3f-471a-8701-c3d4894f4d35 None admin] RBAC: Adding query filter params (name=reader) {{(pid=92057) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} May 23 12:41:00.094566 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f551424-2b3f-471a-8701-c3d4894f4d35 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:41:00.095496 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f551424-2b3f-471a-8701-c3d4894f4d35 None admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:41:00.106749 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 169/349] 10.222.0.30 () {56 vars in 1101 bytes} [Mon May 23 12:40:59 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 194 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) May 23 12:41:00.107536 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f551424-2b3f-471a-8701-c3d4894f4d35 None admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:41:00.131908 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2be5876c-bff5-463f-b745-eb96fe5531fb None None] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:41:00.154977 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=100 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:41:00.177806 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b2a9b22-7c2d-4369-a0ac-0c76e31d4cdf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:41:00.178979 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b2a9b22-7c2d-4369-a0ac-0c76e31d4cdf None admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:41:00.179356 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b2a9b22-7c2d-4369-a0ac-0c76e31d4cdf None admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:41:00.179758 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b2a9b22-7c2d-4369-a0ac-0c76e31d4cdf None admin] PATH_INFO: `/v3/groups/b272e49032f04e138cd631884e9f8b93` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:41:00.187152 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b2a9b22-7c2d-4369-a0ac-0c76e31d4cdf None admin] RBAC: Authorizing `identity:get_group(group_id=b272e49032f04e138cd631884e9f8b93)` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:41:00.188744 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b2a9b22-7c2d-4369-a0ac-0c76e31d4cdf None admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:41:00.190751 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 181/350] 10.222.0.30 () {56 vars in 1180 bytes} [Mon May 23 12:41:00 2022] GET /identity/v3/groups/b272e49032f04e138cd631884e9f8b93 => generated 226 bytes in 181 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) May 23 12:41:00.191632 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b2a9b22-7c2d-4369-a0ac-0c76e31d4cdf None admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:41:00.232255 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f551424-2b3f-471a-8701-c3d4894f4d35 None admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:41:00.271166 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f901041-43cd-48b6-be44-d0a97362dcae None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:41:00.272234 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f901041-43cd-48b6-be44-d0a97362dcae None admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:41:00.273601 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f901041-43cd-48b6-be44-d0a97362dcae None admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:41:00.274120 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f901041-43cd-48b6-be44-d0a97362dcae None admin] PATH_INFO: `/v3/roles` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:41:00.276405 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f901041-43cd-48b6-be44-d0a97362dcae None admin] RBAC: Adding query filter params (name=member) {{(pid=92057) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} May 23 12:41:00.276944 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f901041-43cd-48b6-be44-d0a97362dcae None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:41:00.278584 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f901041-43cd-48b6-be44-d0a97362dcae None admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:41:00.292126 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 170/351] 10.222.0.30 () {56 vars in 1101 bytes} [Mon May 23 12:41:00 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 185 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) May 23 12:41:00.293620 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f901041-43cd-48b6-be44-d0a97362dcae None admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:41:00.315494 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b2a9b22-7c2d-4369-a0ac-0c76e31d4cdf None admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:41:00.334731 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=100 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:41:00.359157 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2677de20-f64e-4fd7-9356-7989b2f9225b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:41:00.359487 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2677de20-f64e-4fd7-9356-7989b2f9225b None admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:41:00.359777 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2677de20-f64e-4fd7-9356-7989b2f9225b None admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:41:00.360082 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2677de20-f64e-4fd7-9356-7989b2f9225b None admin] PATH_INFO: `/v3/roles/member` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:41:00.372941 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2677de20-f64e-4fd7-9356-7989b2f9225b None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} May 23 12:41:00.381766 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application [None req-2677de20-f64e-4fd7-9356-7989b2f9225b None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. May 23 12:41:00.381766 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application Traceback (most recent call last): May 23 12:41:00.381766 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request May 23 12:41:00.381766 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application rv = self.dispatch_request() May 23 12:41:00.381766 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request May 23 12:41:00.381766 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) May 23 12:41:00.381766 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper May 23 12:41:00.381766 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) May 23 12:41:00.381766 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view May 23 12:41:00.381766 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) May 23 12:41:00.381766 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request May 23 12:41:00.381766 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) May 23 12:41:00.381766 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get May 23 12:41:00.381766 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self._get_role(role_id) May 23 12:41:00.381766 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role May 23 12:41:00.381766 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') May 23 12:41:00.382743 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call May 23 12:41:00.382743 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( May 23 12:41:00.382743 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data May 23 12:41:00.382743 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) May 23 12:41:00.382743 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped May 23 12:41:00.382743 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) May 23 12:41:00.382743 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun May 23 12:41:00.382743 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) May 23 12:41:00.382743 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func May 23 12:41:00.382743 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self.get_or_create( May 23 12:41:00.382743 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create May 23 12:41:00.382743 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application with Lock( May 23 12:41:00.382743 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ May 23 12:41:00.382743 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self._enter() May 23 12:41:00.382743 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter May 23 12:41:00.382743 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) May 23 12:41:00.388388 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create May 23 12:41:00.388388 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self.creator() May 23 12:41:00.388388 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value May 23 12:41:00.388388 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application created_value = creator( May 23 12:41:00.388388 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role May 23 12:41:00.388388 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) May 23 12:41:00.388388 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role May 23 12:41:00.388388 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() May 23 12:41:00.388388 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role May 23 12:41:00.388388 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) May 23 12:41:00.388388 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. May 23 12:41:00.388388 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application May 23 12:41:00.388388 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 182/352] 10.222.0.30 () {56 vars in 1099 bytes} [Mon May 23 12:41:00 2022] GET /identity/v3/roles/member => generated 84 bytes in 192 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) May 23 12:41:00.388388 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2677de20-f64e-4fd7-9356-7989b2f9225b None admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:41:00.440516 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=100 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:41:00.458536 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f901041-43cd-48b6-be44-d0a97362dcae None admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:41:00.504528 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc3f026d-3f27-4851-b0b9-922fe248ae80 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:41:00.505422 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc3f026d-3f27-4851-b0b9-922fe248ae80 None admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:41:00.505700 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc3f026d-3f27-4851-b0b9-922fe248ae80 None admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:41:00.505998 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc3f026d-3f27-4851-b0b9-922fe248ae80 None admin] PATH_INFO: `/v3/users/4c82e0529aea4c23a9556941da8b78db` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:41:00.508182 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc3f026d-3f27-4851-b0b9-922fe248ae80 None admin] RBAC: Authorizing `identity:get_user(user_id=4c82e0529aea4c23a9556941da8b78db)` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:41:00.511243 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc3f026d-3f27-4851-b0b9-922fe248ae80 None admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:41:00.513210 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 171/353] 10.222.0.30 () {56 vars in 1177 bytes} [Mon May 23 12:41:00 2022] GET /identity/v3/users/4c82e0529aea4c23a9556941da8b78db => generated 293 bytes in 220 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) May 23 12:41:00.513788 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc3f026d-3f27-4851-b0b9-922fe248ae80 None admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:41:00.538372 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2677de20-f64e-4fd7-9356-7989b2f9225b None admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:41:00.582914 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a7ea6f9-6866-43a9-a263-6321926f8b1d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:41:00.583923 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a7ea6f9-6866-43a9-a263-6321926f8b1d None admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:41:00.584417 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a7ea6f9-6866-43a9-a263-6321926f8b1d None admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:41:00.584882 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a7ea6f9-6866-43a9-a263-6321926f8b1d None admin] PATH_INFO: `/v3/role_assignments` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:41:00.586053 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a7ea6f9-6866-43a9-a263-6321926f8b1d None admin] RBAC: Adding query filter params (group.id=b272e49032f04e138cd631884e9f8b93, role.id=e3b90948645643e888620ab1e5c774bf, scope.project.id=486401ae2e5f40d5a5f060242896ae53) {{(pid=92056) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} May 23 12:41:00.586578 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a7ea6f9-6866-43a9-a263-6321926f8b1d None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:41:00.588284 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a7ea6f9-6866-43a9-a263-6321926f8b1d None admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:41:00.594326 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 183/354] 10.222.0.30 () {56 vars in 1376 bytes} [Mon May 23 12:41:00 2022] GET /identity/v3/role_assignments?group.id=b272e49032f04e138cd631884e9f8b93&scope.project.id=486401ae2e5f40d5a5f060242896ae53&role.id=e3b90948645643e888620ab1e5c774bf => generated 260 bytes in 211 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) May 23 12:41:00.596406 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8dc5c1a6-69e0-4546-abf9-5d0c5fb23117 None None] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:41:00.596692 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8dc5c1a6-69e0-4546-abf9-5d0c5fb23117 None None] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:41:00.596967 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8dc5c1a6-69e0-4546-abf9-5d0c5fb23117 None None] PATH_INFO: `/` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:41:00.600679 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 184/355] 10.222.0.30 () {54 vars in 926 bytes} [Mon May 23 12:41:00 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) May 23 12:41:00.600679 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8dc5c1a6-69e0-4546-abf9-5d0c5fb23117 None None] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:41:00.631620 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=100 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:41:00.646652 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc3f026d-3f27-4851-b0b9-922fe248ae80 None admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:41:00.708577 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a6c7d71-2f02-40fe-8656-a76fef3c2f78 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:41:00.709478 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a6c7d71-2f02-40fe-8656-a76fef3c2f78 None admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:41:00.710047 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a6c7d71-2f02-40fe-8656-a76fef3c2f78 None admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:41:00.710399 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a6c7d71-2f02-40fe-8656-a76fef3c2f78 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:41:00.744545 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8dc5c1a6-69e0-4546-abf9-5d0c5fb23117 None None] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:41:00.747295 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=100 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:41:00.805594 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74863102-32f1-4d8e-9b7c-f22f4f68e9f0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:41:00.808023 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74863102-32f1-4d8e-9b7c-f22f4f68e9f0 None admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:41:00.808162 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74863102-32f1-4d8e-9b7c-f22f4f68e9f0 None admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:41:00.808436 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74863102-32f1-4d8e-9b7c-f22f4f68e9f0 None admin] PATH_INFO: `/v3/roles` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:41:00.809218 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74863102-32f1-4d8e-9b7c-f22f4f68e9f0 None admin] RBAC: Adding query filter params (name=member) {{(pid=92056) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} May 23 12:41:00.809528 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74863102-32f1-4d8e-9b7c-f22f4f68e9f0 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:41:00.810857 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a6c7d71-2f02-40fe-8656-a76fef3c2f78 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:41:00.811316 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74863102-32f1-4d8e-9b7c-f22f4f68e9f0 None admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:41:00.812351 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a6c7d71-2f02-40fe-8656-a76fef3c2f78 None admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:41:00.822307 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 185/356] 10.222.0.30 () {56 vars in 1101 bytes} [Mon May 23 12:41:00 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 222 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) May 23 12:41:00.823274 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74863102-32f1-4d8e-9b7c-f22f4f68e9f0 None admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:41:00.920938 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=100 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:41:00.935936 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 172/357] 10.222.0.30 () {58 vars in 1282 bytes} [Mon May 23 12:41:00 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 422 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) May 23 12:41:00.944620 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-259623c3-9def-46a1-bb30-bff6f30e8e83 None None] REQUEST_METHOD: `POST` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:41:00.944620 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-259623c3-9def-46a1-bb30-bff6f30e8e83 None None] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:41:00.944620 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-259623c3-9def-46a1-bb30-bff6f30e8e83 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:41:00.955958 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.auth.core [None req-259623c3-9def-46a1-bb30-bff6f30e8e83 None None] MFA Rules not processed for user `0f7119f8fa5c4445977671140ded91a4`. Rule list: `[]` (Enabled: `True`). {{(pid=92057) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 12:41:00.957149 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.fernet_utils [None req-259623c3-9def-46a1-bb30-bff6f30e8e83 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92057) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 12:41:00.966234 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74863102-32f1-4d8e-9b7c-f22f4f68e9f0 None admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:41:01.007799 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07bb5ca1-c614-478c-85eb-ae88717ea272 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:41:01.008638 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07bb5ca1-c614-478c-85eb-ae88717ea272 None admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:41:01.008638 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07bb5ca1-c614-478c-85eb-ae88717ea272 None admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:41:01.008918 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07bb5ca1-c614-478c-85eb-ae88717ea272 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:41:01.009736 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07bb5ca1-c614-478c-85eb-ae88717ea272 None admin] RBAC: Adding query filter params (role.id=c223286ed21a4654a0f2b4e5551831c7, user.id=4c82e0529aea4c23a9556941da8b78db, scope.system=all) {{(pid=92056) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} May 23 12:41:01.010082 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07bb5ca1-c614-478c-85eb-ae88717ea272 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:41:01.011536 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07bb5ca1-c614-478c-85eb-ae88717ea272 None admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:41:01.030463 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 186/358] 10.222.0.30 () {56 vars in 1308 bytes} [Mon May 23 12:41:00 2022] GET /identity/v3/role_assignments?user.id=4c82e0529aea4c23a9556941da8b78db&scope.system=all&role.id=c223286ed21a4654a0f2b4e5551831c7 => generated 510 bytes in 207 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) May 23 12:41:01.031711 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de3f0a13-2fe5-46a8-b0ce-5ffc486c0aa3 None None] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:41:01.032007 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de3f0a13-2fe5-46a8-b0ce-5ffc486c0aa3 None None] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:41:01.032290 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de3f0a13-2fe5-46a8-b0ce-5ffc486c0aa3 None None] PATH_INFO: `/` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:41:01.033804 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 187/359] 10.222.0.30 () {54 vars in 926 bytes} [Mon May 23 12:41:01 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) May 23 12:41:01.035641 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de3f0a13-2fe5-46a8-b0ce-5ffc486c0aa3 None None] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:41:01.063231 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 173/360] 10.222.0.30 () {58 vars in 1023 bytes} [Mon May 23 12:41:00 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 127 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) May 23 12:41:01.064786 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-259623c3-9def-46a1-bb30-bff6f30e8e83 None None] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:41:01.068985 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=100 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:41:01.160716 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de3f0a13-2fe5-46a8-b0ce-5ffc486c0aa3 None None] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:41:01.185929 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-259623c3-9def-46a1-bb30-bff6f30e8e83 None None] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:41:01.200671 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d02543c9-e14f-49ac-b3fa-d1136e45d3aa None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:41:01.201739 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d02543c9-e14f-49ac-b3fa-d1136e45d3aa None admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:41:01.202265 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d02543c9-e14f-49ac-b3fa-d1136e45d3aa None admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:41:01.202810 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d02543c9-e14f-49ac-b3fa-d1136e45d3aa None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:41:01.226193 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=100 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:41:01.230362 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be2c9d72-4176-4f2c-924d-25f55622f071 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:41:01.231304 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be2c9d72-4176-4f2c-924d-25f55622f071 None admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:41:01.231592 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be2c9d72-4176-4f2c-924d-25f55622f071 None admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:41:01.231928 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be2c9d72-4176-4f2c-924d-25f55622f071 None admin] PATH_INFO: `/v3/projects/486401ae2e5f40d5a5f060242896ae53` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:41:01.236779 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be2c9d72-4176-4f2c-924d-25f55622f071 None admin] RBAC: Authorizing `identity:get_project(project_id=486401ae2e5f40d5a5f060242896ae53)` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:41:01.236779 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be2c9d72-4176-4f2c-924d-25f55622f071 None admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:41:01.237391 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 174/361] 10.222.0.30 () {56 vars in 1186 bytes} [Mon May 23 12:41:01 2022] GET /identity/v3/projects/486401ae2e5f40d5a5f060242896ae53 => generated 298 bytes in 173 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) May 23 12:41:01.238736 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45f79cb8-21d2-4be9-8ca8-1346c34edf85 None None] REQUEST_METHOD: `POST` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:41:01.239022 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45f79cb8-21d2-4be9-8ca8-1346c34edf85 None None] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:41:01.239300 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45f79cb8-21d2-4be9-8ca8-1346c34edf85 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:41:01.261866 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.auth.core [None req-45f79cb8-21d2-4be9-8ca8-1346c34edf85 None None] MFA Rules not processed for user `0f7119f8fa5c4445977671140ded91a4`. Rule list: `[]` (Enabled: `True`). {{(pid=92057) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 12:41:01.263104 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.fernet_utils [None req-45f79cb8-21d2-4be9-8ca8-1346c34edf85 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92057) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 12:41:01.282585 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d02543c9-e14f-49ac-b3fa-d1136e45d3aa None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:41:01.284386 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d02543c9-e14f-49ac-b3fa-d1136e45d3aa None admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:41:01.309766 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=100 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:41:01.351631 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 175/362] 10.222.0.30 () {58 vars in 1023 bytes} [Mon May 23 12:41:01 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 114 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) May 23 12:41:01.353435 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45f79cb8-21d2-4be9-8ca8-1346c34edf85 None None] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:41:01.400556 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 188/363] 10.222.0.30 () {58 vars in 1282 bytes} [Mon May 23 12:41:01 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 366 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) May 23 12:41:01.402050 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d02543c9-e14f-49ac-b3fa-d1136e45d3aa None admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:41:01.483051 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45f79cb8-21d2-4be9-8ca8-1346c34edf85 None None] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:41:01.500762 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=100 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:41:01.523001 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98c3bd58-c0dc-4c2a-b7aa-940cdd9c871d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:41:01.524283 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98c3bd58-c0dc-4c2a-b7aa-940cdd9c871d None admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:41:01.524788 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98c3bd58-c0dc-4c2a-b7aa-940cdd9c871d None admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:41:01.524965 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98c3bd58-c0dc-4c2a-b7aa-940cdd9c871d None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:41:01.535195 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d02543c9-e14f-49ac-b3fa-d1136e45d3aa None admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:41:01.543672 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-98c3bd58-c0dc-4c2a-b7aa-940cdd9c871d None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} May 23 12:41:01.552142 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application [None req-98c3bd58-c0dc-4c2a-b7aa-940cdd9c871d None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. May 23 12:41:01.552142 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application Traceback (most recent call last): May 23 12:41:01.552142 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request May 23 12:41:01.552142 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application rv = self.dispatch_request() May 23 12:41:01.552142 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request May 23 12:41:01.552142 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) May 23 12:41:01.552142 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper May 23 12:41:01.552142 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) May 23 12:41:01.552142 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view May 23 12:41:01.552142 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) May 23 12:41:01.552142 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request May 23 12:41:01.552142 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) May 23 12:41:01.552142 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get May 23 12:41:01.552142 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self._get_role(role_id) May 23 12:41:01.552142 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role May 23 12:41:01.552142 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') May 23 12:41:01.553067 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call May 23 12:41:01.553067 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( May 23 12:41:01.553067 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data May 23 12:41:01.553067 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) May 23 12:41:01.553067 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped May 23 12:41:01.553067 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) May 23 12:41:01.553067 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun May 23 12:41:01.553067 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) May 23 12:41:01.553067 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func May 23 12:41:01.553067 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self.get_or_create( May 23 12:41:01.553067 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create May 23 12:41:01.553067 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application with Lock( May 23 12:41:01.553067 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ May 23 12:41:01.553067 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self._enter() May 23 12:41:01.553067 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter May 23 12:41:01.553067 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) May 23 12:41:01.553983 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create May 23 12:41:01.553983 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self.creator() May 23 12:41:01.553983 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value May 23 12:41:01.553983 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application created_value = creator( May 23 12:41:01.553983 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role May 23 12:41:01.553983 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) May 23 12:41:01.553983 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role May 23 12:41:01.553983 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() May 23 12:41:01.553983 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role May 23 12:41:01.553983 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) May 23 12:41:01.553983 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. May 23 12:41:01.553983 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application May 23 12:41:01.553983 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 176/364] 10.222.0.30 () {56 vars in 1114 bytes} [Mon May 23 12:41:01 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 201 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) May 23 12:41:01.559291 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98c3bd58-c0dc-4c2a-b7aa-940cdd9c871d None admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:41:01.572533 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=100 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:41:01.587040 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7ba1ee5-9931-4e24-9363-84d52625d87d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:41:01.588773 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7ba1ee5-9931-4e24-9363-84d52625d87d None admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:41:01.589211 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7ba1ee5-9931-4e24-9363-84d52625d87d None admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:41:01.589637 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7ba1ee5-9931-4e24-9363-84d52625d87d None admin] PATH_INFO: `/v3/groups/b272e49032f04e138cd631884e9f8b93` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:41:01.591056 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7ba1ee5-9931-4e24-9363-84d52625d87d None admin] RBAC: Authorizing `identity:get_group(group_id=b272e49032f04e138cd631884e9f8b93)` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:41:01.592664 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7ba1ee5-9931-4e24-9363-84d52625d87d None admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:41:01.594493 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 189/365] 10.222.0.30 () {56 vars in 1180 bytes} [Mon May 23 12:41:01 2022] GET /identity/v3/groups/b272e49032f04e138cd631884e9f8b93 => generated 226 bytes in 193 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) May 23 12:41:01.595818 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7ba1ee5-9931-4e24-9363-84d52625d87d None admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:41:01.691525 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98c3bd58-c0dc-4c2a-b7aa-940cdd9c871d None admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:41:01.728816 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7ba1ee5-9931-4e24-9363-84d52625d87d None admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:41:01.735253 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07dbfe31-bfa7-48a8-90b8-f4334f84f380 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:41:01.735923 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07dbfe31-bfa7-48a8-90b8-f4334f84f380 None admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:41:01.736203 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07dbfe31-bfa7-48a8-90b8-f4334f84f380 None admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:41:01.736475 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07dbfe31-bfa7-48a8-90b8-f4334f84f380 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:41:01.749367 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07dbfe31-bfa7-48a8-90b8-f4334f84f380 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} May 23 12:41:01.757470 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application [None req-07dbfe31-bfa7-48a8-90b8-f4334f84f380 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. May 23 12:41:01.757470 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application Traceback (most recent call last): May 23 12:41:01.757470 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request May 23 12:41:01.757470 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application rv = self.dispatch_request() May 23 12:41:01.757470 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request May 23 12:41:01.757470 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) May 23 12:41:01.757470 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper May 23 12:41:01.757470 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) May 23 12:41:01.757470 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view May 23 12:41:01.757470 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) May 23 12:41:01.757470 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request May 23 12:41:01.757470 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) May 23 12:41:01.757470 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get May 23 12:41:01.757470 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self._get_role(role_id) May 23 12:41:01.757470 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role May 23 12:41:01.757470 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') May 23 12:41:01.758478 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call May 23 12:41:01.758478 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( May 23 12:41:01.758478 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data May 23 12:41:01.758478 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) May 23 12:41:01.758478 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped May 23 12:41:01.758478 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) May 23 12:41:01.758478 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun May 23 12:41:01.758478 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) May 23 12:41:01.758478 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func May 23 12:41:01.758478 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self.get_or_create( May 23 12:41:01.758478 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create May 23 12:41:01.758478 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application with Lock( May 23 12:41:01.758478 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ May 23 12:41:01.758478 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self._enter() May 23 12:41:01.758478 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter May 23 12:41:01.758478 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) May 23 12:41:01.759357 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create May 23 12:41:01.759357 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self.creator() May 23 12:41:01.759357 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value May 23 12:41:01.759357 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application created_value = creator( May 23 12:41:01.759357 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role May 23 12:41:01.759357 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) May 23 12:41:01.759357 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role May 23 12:41:01.759357 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() May 23 12:41:01.759357 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role May 23 12:41:01.759357 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) May 23 12:41:01.759357 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. May 23 12:41:01.759357 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application May 23 12:41:01.759357 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 177/366] 10.222.0.30 () {56 vars in 1096 bytes} [Mon May 23 12:41:01 2022] GET /identity/v3/roles/admin => generated 83 bytes in 205 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) May 23 12:41:01.760055 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07dbfe31-bfa7-48a8-90b8-f4334f84f380 None admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:41:01.778562 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b9f4ac0-6158-4e5c-83bb-04811b086d06 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:41:01.781154 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b9f4ac0-6158-4e5c-83bb-04811b086d06 None admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:41:01.781447 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b9f4ac0-6158-4e5c-83bb-04811b086d06 None admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:41:01.781729 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b9f4ac0-6158-4e5c-83bb-04811b086d06 None admin] PATH_INFO: `/v3/projects/b39742b1d1b34dfe82a9ce4da39b7582` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:41:01.783249 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b9f4ac0-6158-4e5c-83bb-04811b086d06 None admin] RBAC: Authorizing `identity:get_project(project_id=b39742b1d1b34dfe82a9ce4da39b7582)` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:41:01.785054 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b9f4ac0-6158-4e5c-83bb-04811b086d06 None admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:41:01.787510 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 190/367] 10.222.0.30 () {56 vars in 1186 bytes} [Mon May 23 12:41:01 2022] GET /identity/v3/projects/b39742b1d1b34dfe82a9ce4da39b7582 => generated 302 bytes in 192 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) May 23 12:41:01.788215 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b9f4ac0-6158-4e5c-83bb-04811b086d06 None admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:41:01.801923 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=100 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:41:01.866070 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=100 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:41:01.898168 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07dbfe31-bfa7-48a8-90b8-f4334f84f380 None admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:41:01.922041 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b9f4ac0-6158-4e5c-83bb-04811b086d06 None admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:41:01.942602 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c6c3a08-9d33-4f75-8aea-257837cc6a69 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:41:01.943577 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c6c3a08-9d33-4f75-8aea-257837cc6a69 None admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:41:01.943863 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c6c3a08-9d33-4f75-8aea-257837cc6a69 None admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:41:01.944142 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c6c3a08-9d33-4f75-8aea-257837cc6a69 None admin] PATH_INFO: `/v3/roles` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:41:01.944892 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c6c3a08-9d33-4f75-8aea-257837cc6a69 None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=92057) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} May 23 12:41:01.945202 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c6c3a08-9d33-4f75-8aea-257837cc6a69 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:41:01.946728 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c6c3a08-9d33-4f75-8aea-257837cc6a69 None admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:41:01.959307 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 178/368] 10.222.0.30 () {56 vars in 1111 bytes} [Mon May 23 12:41:01 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 200 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) May 23 12:41:01.960233 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c6c3a08-9d33-4f75-8aea-257837cc6a69 None admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:41:01.969147 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09a1158f-e3e8-42a6-8557-bdff43ff00a8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:41:01.969809 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09a1158f-e3e8-42a6-8557-bdff43ff00a8 None admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:41:01.970121 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09a1158f-e3e8-42a6-8557-bdff43ff00a8 None admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:41:01.970421 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09a1158f-e3e8-42a6-8557-bdff43ff00a8 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:41:01.971234 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09a1158f-e3e8-42a6-8557-bdff43ff00a8 None admin] RBAC: Adding query filter params (group.id=b272e49032f04e138cd631884e9f8b93, role.id=80fb22507c9a4ff09d44d8f12f6fff1a, scope.project.id=486401ae2e5f40d5a5f060242896ae53) {{(pid=92056) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} May 23 12:41:01.971665 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09a1158f-e3e8-42a6-8557-bdff43ff00a8 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:41:01.973135 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09a1158f-e3e8-42a6-8557-bdff43ff00a8 None admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:41:01.981688 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 191/369] 10.222.0.30 () {56 vars in 1376 bytes} [Mon May 23 12:41:01 2022] GET /identity/v3/role_assignments?group.id=b272e49032f04e138cd631884e9f8b93&scope.project.id=486401ae2e5f40d5a5f060242896ae53&role.id=80fb22507c9a4ff09d44d8f12f6fff1a => generated 260 bytes in 194 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) May 23 12:41:01.982685 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09a1158f-e3e8-42a6-8557-bdff43ff00a8 None admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:41:02.115726 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=100 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:41:02.116558 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09a1158f-e3e8-42a6-8557-bdff43ff00a8 None admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:41:02.122528 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c6c3a08-9d33-4f75-8aea-257837cc6a69 None admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:41:02.150586 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=100 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:41:02.166846 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c77f923-024b-4db8-b4be-82d24c2dfa74 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:41:02.168735 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c77f923-024b-4db8-b4be-82d24c2dfa74 None admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:41:02.169199 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c77f923-024b-4db8-b4be-82d24c2dfa74 None admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:41:02.169964 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c77f923-024b-4db8-b4be-82d24c2dfa74 None admin] PATH_INFO: `/v3/groups/b272e49032f04e138cd631884e9f8b93` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:41:02.171814 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c77f923-024b-4db8-b4be-82d24c2dfa74 None admin] RBAC: Authorizing `identity:get_group(group_id=b272e49032f04e138cd631884e9f8b93)` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:41:02.173751 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c77f923-024b-4db8-b4be-82d24c2dfa74 None admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:41:02.176909 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 192/370] 10.222.0.30 () {56 vars in 1180 bytes} [Mon May 23 12:41:01 2022] GET /identity/v3/groups/b272e49032f04e138cd631884e9f8b93 => generated 226 bytes in 195 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) May 23 12:41:02.178395 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c77f923-024b-4db8-b4be-82d24c2dfa74 None admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:41:02.181704 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5992526c-aa6d-4edf-bb22-2f0033e6b949 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:41:02.182905 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5992526c-aa6d-4edf-bb22-2f0033e6b949 None admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:41:02.183284 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5992526c-aa6d-4edf-bb22-2f0033e6b949 None admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:41:02.183634 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5992526c-aa6d-4edf-bb22-2f0033e6b949 None admin] PATH_INFO: `/v3/roles` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:41:02.184817 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5992526c-aa6d-4edf-bb22-2f0033e6b949 None admin] RBAC: Adding query filter params (name=admin) {{(pid=92057) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} May 23 12:41:02.184927 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5992526c-aa6d-4edf-bb22-2f0033e6b949 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:41:02.186445 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5992526c-aa6d-4edf-bb22-2f0033e6b949 None admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:41:02.198775 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 179/371] 10.222.0.30 () {56 vars in 1099 bytes} [Mon May 23 12:41:01 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 239 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) May 23 12:41:02.199831 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5992526c-aa6d-4edf-bb22-2f0033e6b949 None admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:41:02.322635 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5992526c-aa6d-4edf-bb22-2f0033e6b949 None admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:41:02.322955 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c77f923-024b-4db8-b4be-82d24c2dfa74 None admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:41:02.364103 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ffa2c04-d83c-4517-ba01-49834b73ec24 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:41:02.364775 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ffa2c04-d83c-4517-ba01-49834b73ec24 None admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:41:02.365061 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ffa2c04-d83c-4517-ba01-49834b73ec24 None admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:41:02.365342 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ffa2c04-d83c-4517-ba01-49834b73ec24 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:41:02.366363 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ffa2c04-d83c-4517-ba01-49834b73ec24 None admin] RBAC: Adding query filter params (group.id=b272e49032f04e138cd631884e9f8b93, role.id=80fb22507c9a4ff09d44d8f12f6fff1a, scope.project.id=b39742b1d1b34dfe82a9ce4da39b7582) {{(pid=92057) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} May 23 12:41:02.366635 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ffa2c04-d83c-4517-ba01-49834b73ec24 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:41:02.368773 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ffa2c04-d83c-4517-ba01-49834b73ec24 None admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:41:02.377741 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fbd7e034-19ed-4c97-862a-9b21adafbb3a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:41:02.377975 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 180/372] 10.222.0.30 () {56 vars in 1376 bytes} [Mon May 23 12:41:02 2022] GET /identity/v3/role_assignments?group.id=b272e49032f04e138cd631884e9f8b93&scope.project.id=b39742b1d1b34dfe82a9ce4da39b7582&role.id=80fb22507c9a4ff09d44d8f12f6fff1a => generated 260 bytes in 178 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) May 23 12:41:02.378430 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbd7e034-19ed-4c97-862a-9b21adafbb3a None admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:41:02.378718 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbd7e034-19ed-4c97-862a-9b21adafbb3a None admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:41:02.378832 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ffa2c04-d83c-4517-ba01-49834b73ec24 None admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:41:02.379024 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbd7e034-19ed-4c97-862a-9b21adafbb3a None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:41:02.442814 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=100 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:41:02.458271 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=100 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:41:02.471988 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fbd7e034-19ed-4c97-862a-9b21adafbb3a None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:41:02.473483 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fbd7e034-19ed-4c97-862a-9b21adafbb3a None admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:41:02.612499 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ffa2c04-d83c-4517-ba01-49834b73ec24 None admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:41:02.657439 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c1aea91-d1e6-4b99-a6ac-b6db53312f08 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:41:02.658977 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c1aea91-d1e6-4b99-a6ac-b6db53312f08 None admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:41:02.659755 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c1aea91-d1e6-4b99-a6ac-b6db53312f08 None admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:41:02.660243 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c1aea91-d1e6-4b99-a6ac-b6db53312f08 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:41:02.688832 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 193/373] 10.222.0.30 () {58 vars in 1282 bytes} [Mon May 23 12:41:02 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 511 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) May 23 12:41:02.697069 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fbd7e034-19ed-4c97-862a-9b21adafbb3a None admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:41:02.759539 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c1aea91-d1e6-4b99-a6ac-b6db53312f08 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:41:02.761435 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c1aea91-d1e6-4b99-a6ac-b6db53312f08 None admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:41:02.792655 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=100 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:41:02.870451 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=100 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:41:02.887435 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 181/374] 10.222.0.30 () {58 vars in 1282 bytes} [Mon May 23 12:41:02 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 509 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) May 23 12:41:02.895013 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c1aea91-d1e6-4b99-a6ac-b6db53312f08 None admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:41:02.904875 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fbd7e034-19ed-4c97-862a-9b21adafbb3a None admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:41:02.959672 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab057171-1c92-4ace-b633-46ae2ca1df01 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:41:02.960482 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab057171-1c92-4ace-b633-46ae2ca1df01 None admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:41:02.960755 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab057171-1c92-4ace-b633-46ae2ca1df01 None admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:41:02.961031 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab057171-1c92-4ace-b633-46ae2ca1df01 None admin] PATH_INFO: `/v3/projects/b39742b1d1b34dfe82a9ce4da39b7582` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:41:02.962439 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab057171-1c92-4ace-b633-46ae2ca1df01 None admin] RBAC: Authorizing `identity:get_project(project_id=b39742b1d1b34dfe82a9ce4da39b7582)` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:41:02.963915 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab057171-1c92-4ace-b633-46ae2ca1df01 None admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:41:02.966048 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 194/375] 10.222.0.30 () {56 vars in 1186 bytes} [Mon May 23 12:41:02 2022] GET /identity/v3/projects/b39742b1d1b34dfe82a9ce4da39b7582 => generated 302 bytes in 269 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) May 23 12:41:02.970948 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab057171-1c92-4ace-b633-46ae2ca1df01 None admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:41:03.086996 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c1aea91-d1e6-4b99-a6ac-b6db53312f08 None admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:41:03.099978 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab057171-1c92-4ace-b633-46ae2ca1df01 None admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:41:03.140153 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2840e3b-fc5c-4299-b9ae-a937df4625b7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:41:03.141384 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2840e3b-fc5c-4299-b9ae-a937df4625b7 None admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:41:03.141901 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2840e3b-fc5c-4299-b9ae-a937df4625b7 None admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:41:03.142385 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2840e3b-fc5c-4299-b9ae-a937df4625b7 None admin] PATH_INFO: `/v3/groups/b272e49032f04e138cd631884e9f8b93` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:41:03.143846 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2840e3b-fc5c-4299-b9ae-a937df4625b7 None admin] RBAC: Authorizing `identity:get_group(group_id=b272e49032f04e138cd631884e9f8b93)` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:41:03.145479 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2840e3b-fc5c-4299-b9ae-a937df4625b7 None admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:41:03.147654 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 195/376] 10.222.0.30 () {56 vars in 1180 bytes} [Mon May 23 12:41:02 2022] GET /identity/v3/groups/b272e49032f04e138cd631884e9f8b93 => generated 226 bytes in 177 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) May 23 12:41:03.151259 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=100 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:41:03.153981 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2840e3b-fc5c-4299-b9ae-a937df4625b7 None admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:41:03.178800 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56a4a07c-3ea1-4457-9c2f-b60834090f02 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:41:03.179675 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56a4a07c-3ea1-4457-9c2f-b60834090f02 None admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:41:03.179964 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56a4a07c-3ea1-4457-9c2f-b60834090f02 None admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:41:03.180244 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56a4a07c-3ea1-4457-9c2f-b60834090f02 None admin] PATH_INFO: `/v3/projects/af8b84d1e3e44efba851eab6bed63282` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:41:03.182628 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56a4a07c-3ea1-4457-9c2f-b60834090f02 None admin] RBAC: Authorizing `identity:get_project(project_id=af8b84d1e3e44efba851eab6bed63282)` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:41:03.184119 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56a4a07c-3ea1-4457-9c2f-b60834090f02 None admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:41:03.185229 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=100 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:41:03.187029 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 182/377] 10.222.0.30 () {56 vars in 1186 bytes} [Mon May 23 12:41:02 2022] GET /identity/v3/projects/af8b84d1e3e44efba851eab6bed63282 => generated 344 bytes in 293 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) May 23 12:41:03.192685 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56a4a07c-3ea1-4457-9c2f-b60834090f02 None admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:41:03.295400 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2840e3b-fc5c-4299-b9ae-a937df4625b7 None admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:41:03.341610 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56a4a07c-3ea1-4457-9c2f-b60834090f02 None admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:41:03.344057 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72c1485b-88ea-4eee-bc46-b357dd5aa7fc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:41:03.345516 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72c1485b-88ea-4eee-bc46-b357dd5aa7fc None admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:41:03.345808 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72c1485b-88ea-4eee-bc46-b357dd5aa7fc None admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:41:03.346132 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72c1485b-88ea-4eee-bc46-b357dd5aa7fc None admin] PATH_INFO: `/v3/role_assignments` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:41:03.346966 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72c1485b-88ea-4eee-bc46-b357dd5aa7fc None admin] RBAC: Adding query filter params (group.id=b272e49032f04e138cd631884e9f8b93, role.id=e3b90948645643e888620ab1e5c774bf, scope.project.id=b39742b1d1b34dfe82a9ce4da39b7582) {{(pid=92056) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} May 23 12:41:03.347275 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72c1485b-88ea-4eee-bc46-b357dd5aa7fc None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:41:03.348721 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72c1485b-88ea-4eee-bc46-b357dd5aa7fc None admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:41:03.357452 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 196/378] 10.222.0.30 () {56 vars in 1376 bytes} [Mon May 23 12:41:03 2022] GET /identity/v3/role_assignments?group.id=b272e49032f04e138cd631884e9f8b93&scope.project.id=b39742b1d1b34dfe82a9ce4da39b7582&role.id=e3b90948645643e888620ab1e5c774bf => generated 260 bytes in 204 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) May 23 12:41:03.359194 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15268851-6723-4cc8-889a-830f46517d81 None None] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:41:03.359482 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15268851-6723-4cc8-889a-830f46517d81 None None] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:41:03.359927 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15268851-6723-4cc8-889a-830f46517d81 None None] PATH_INFO: `/` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:41:03.366001 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 197/379] 10.222.0.30 () {54 vars in 926 bytes} [Mon May 23 12:41:03 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) May 23 12:41:03.368591 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d275bdd2-15e7-434b-abbb-55835293f42b None None] REQUEST_METHOD: `POST` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:41:03.368863 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d275bdd2-15e7-434b-abbb-55835293f42b None None] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:41:03.369137 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d275bdd2-15e7-434b-abbb-55835293f42b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:41:03.389389 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33843138-1234-485a-bb5b-9965d7d818c5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:41:03.390303 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.auth.core [None req-d275bdd2-15e7-434b-abbb-55835293f42b None None] MFA Rules not processed for user `0f7119f8fa5c4445977671140ded91a4`. Rule list: `[]` (Enabled: `True`). {{(pid=92056) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 12:41:03.390939 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33843138-1234-485a-bb5b-9965d7d818c5 None admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:41:03.391434 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33843138-1234-485a-bb5b-9965d7d818c5 None admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:41:03.391556 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.fernet_utils [None req-d275bdd2-15e7-434b-abbb-55835293f42b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92056) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 12:41:03.392029 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33843138-1234-485a-bb5b-9965d7d818c5 None admin] PATH_INFO: `/v3/groups/33d7b939d9664f90b8df13bffb850ca4` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:41:03.399196 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33843138-1234-485a-bb5b-9965d7d818c5 None admin] RBAC: Authorizing `identity:get_group(group_id=33d7b939d9664f90b8df13bffb850ca4)` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:41:03.401597 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33843138-1234-485a-bb5b-9965d7d818c5 None admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:41:03.403880 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 183/380] 10.222.0.30 () {56 vars in 1180 bytes} [Mon May 23 12:41:03 2022] GET /identity/v3/groups/33d7b939d9664f90b8df13bffb850ca4 => generated 229 bytes in 212 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) May 23 12:41:03.410283 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33843138-1234-485a-bb5b-9965d7d818c5 None admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:41:03.481486 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=100 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:41:03.487995 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 198/381] 10.222.0.30 () {58 vars in 1023 bytes} [Mon May 23 12:41:03 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 120 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) May 23 12:41:03.527543 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d275bdd2-15e7-434b-abbb-55835293f42b None None] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:41:03.537906 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33843138-1234-485a-bb5b-9965d7d818c5 None admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:41:03.568167 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=100 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:41:03.576378 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7090661-f3b7-45af-89e7-9fea24e1a0c0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:41:03.577396 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7090661-f3b7-45af-89e7-9fea24e1a0c0 None admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:41:03.577908 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7090661-f3b7-45af-89e7-9fea24e1a0c0 None admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:41:03.581808 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7090661-f3b7-45af-89e7-9fea24e1a0c0 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:41:03.581808 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7090661-f3b7-45af-89e7-9fea24e1a0c0 None admin] RBAC: Adding query filter params (group.id=33d7b939d9664f90b8df13bffb850ca4, role.id=bc3c3c9d654940cbaa5177a6be23d1bc, scope.project.id=af8b84d1e3e44efba851eab6bed63282) {{(pid=92057) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} May 23 12:41:03.581808 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7090661-f3b7-45af-89e7-9fea24e1a0c0 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:41:03.581808 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7090661-f3b7-45af-89e7-9fea24e1a0c0 None admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:41:03.588701 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 184/382] 10.222.0.30 () {56 vars in 1376 bytes} [Mon May 23 12:41:03 2022] GET /identity/v3/role_assignments?group.id=33d7b939d9664f90b8df13bffb850ca4&scope.project.id=af8b84d1e3e44efba851eab6bed63282&role.id=bc3c3c9d654940cbaa5177a6be23d1bc => generated 260 bytes in 179 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) May 23 12:41:03.671947 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d275bdd2-15e7-434b-abbb-55835293f42b None None] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:41:03.713337 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-279c5659-28a8-4849-b84b-42d75da7eda2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:41:03.714273 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-279c5659-28a8-4849-b84b-42d75da7eda2 None admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:41:03.714545 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-279c5659-28a8-4849-b84b-42d75da7eda2 None admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:41:03.714815 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-279c5659-28a8-4849-b84b-42d75da7eda2 None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:41:03.727266 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-279c5659-28a8-4849-b84b-42d75da7eda2 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} May 23 12:41:03.736158 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application [None req-279c5659-28a8-4849-b84b-42d75da7eda2 None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. May 23 12:41:03.736158 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application Traceback (most recent call last): May 23 12:41:03.736158 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request May 23 12:41:03.736158 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application rv = self.dispatch_request() May 23 12:41:03.736158 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request May 23 12:41:03.736158 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) May 23 12:41:03.736158 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper May 23 12:41:03.736158 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) May 23 12:41:03.736158 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view May 23 12:41:03.736158 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) May 23 12:41:03.736158 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request May 23 12:41:03.736158 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) May 23 12:41:03.736158 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get May 23 12:41:03.736158 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self._get_role(role_id) May 23 12:41:03.736158 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role May 23 12:41:03.736158 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') May 23 12:41:03.737156 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call May 23 12:41:03.737156 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( May 23 12:41:03.737156 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data May 23 12:41:03.737156 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) May 23 12:41:03.737156 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped May 23 12:41:03.737156 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) May 23 12:41:03.737156 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun May 23 12:41:03.737156 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) May 23 12:41:03.737156 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func May 23 12:41:03.737156 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self.get_or_create( May 23 12:41:03.737156 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create May 23 12:41:03.737156 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application with Lock( May 23 12:41:03.737156 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ May 23 12:41:03.737156 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self._enter() May 23 12:41:03.737156 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter May 23 12:41:03.737156 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) May 23 12:41:03.738476 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create May 23 12:41:03.738476 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self.creator() May 23 12:41:03.738476 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value May 23 12:41:03.738476 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application created_value = creator( May 23 12:41:03.738476 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role May 23 12:41:03.738476 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) May 23 12:41:03.738476 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role May 23 12:41:03.738476 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() May 23 12:41:03.738476 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role May 23 12:41:03.738476 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) May 23 12:41:03.738476 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. May 23 12:41:03.738476 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application May 23 12:41:03.738476 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 199/383] 10.222.0.30 () {56 vars in 1114 bytes} [Mon May 23 12:41:03 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 212 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) May 23 12:41:03.744524 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7090661-f3b7-45af-89e7-9fea24e1a0c0 None admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:41:03.876673 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7090661-f3b7-45af-89e7-9fea24e1a0c0 None admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:41:03.898267 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=100 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:41:03.914127 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ced549e-1a85-4556-83fb-261fa06d0dd3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:41:03.915977 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ced549e-1a85-4556-83fb-261fa06d0dd3 None admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:41:03.922915 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ced549e-1a85-4556-83fb-261fa06d0dd3 None admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:41:03.922915 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ced549e-1a85-4556-83fb-261fa06d0dd3 None admin] PATH_INFO: `/v3/roles` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:41:03.922915 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ced549e-1a85-4556-83fb-261fa06d0dd3 None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=92057) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} May 23 12:41:03.922915 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ced549e-1a85-4556-83fb-261fa06d0dd3 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:41:03.922915 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ced549e-1a85-4556-83fb-261fa06d0dd3 None admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:41:03.928373 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 185/384] 10.222.0.30 () {56 vars in 1111 bytes} [Mon May 23 12:41:03 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 185 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) May 23 12:41:03.938051 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-279c5659-28a8-4849-b84b-42d75da7eda2 None admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:41:04.058026 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=100 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:41:04.077405 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-279c5659-28a8-4849-b84b-42d75da7eda2 None admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:41:04.118022 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66b3528a-2de6-46c6-9288-9f9c4c1a2d25 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:41:04.119230 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66b3528a-2de6-46c6-9288-9f9c4c1a2d25 None admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:41:04.119514 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66b3528a-2de6-46c6-9288-9f9c4c1a2d25 None admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:41:04.119791 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66b3528a-2de6-46c6-9288-9f9c4c1a2d25 None admin] PATH_INFO: `/v3/groups/b272e49032f04e138cd631884e9f8b93` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:41:04.121006 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66b3528a-2de6-46c6-9288-9f9c4c1a2d25 None admin] RBAC: Authorizing `identity:get_group(group_id=b272e49032f04e138cd631884e9f8b93)` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:41:04.122492 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66b3528a-2de6-46c6-9288-9f9c4c1a2d25 None admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:41:04.124462 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 200/385] 10.222.0.30 () {56 vars in 1180 bytes} [Mon May 23 12:41:03 2022] GET /identity/v3/groups/b272e49032f04e138cd631884e9f8b93 => generated 226 bytes in 187 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) May 23 12:41:04.129797 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ced549e-1a85-4556-83fb-261fa06d0dd3 None admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:41:04.247973 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ced549e-1a85-4556-83fb-261fa06d0dd3 None admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:41:04.293306 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f359fbc2-c69a-45ba-9796-57d35d4864aa None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:41:04.294530 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f359fbc2-c69a-45ba-9796-57d35d4864aa None admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:41:04.295042 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f359fbc2-c69a-45ba-9796-57d35d4864aa None admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:41:04.295513 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f359fbc2-c69a-45ba-9796-57d35d4864aa None admin] PATH_INFO: `/v3/projects/486401ae2e5f40d5a5f060242896ae53` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:41:04.302598 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f359fbc2-c69a-45ba-9796-57d35d4864aa None admin] RBAC: Authorizing `identity:get_project(project_id=486401ae2e5f40d5a5f060242896ae53)` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:41:04.304263 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f359fbc2-c69a-45ba-9796-57d35d4864aa None admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:41:04.306337 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 186/386] 10.222.0.30 () {56 vars in 1186 bytes} [Mon May 23 12:41:04 2022] GET /identity/v3/projects/486401ae2e5f40d5a5f060242896ae53 => generated 298 bytes in 177 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) May 23 12:41:04.312131 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66b3528a-2de6-46c6-9288-9f9c4c1a2d25 None admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:41:04.433888 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66b3528a-2de6-46c6-9288-9f9c4c1a2d25 None admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:41:04.469495 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6542403-f5cb-4c50-ae21-206835cf448c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:41:04.470401 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6542403-f5cb-4c50-ae21-206835cf448c None admin] REQUEST_METHOD: `PUT` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:41:04.470689 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6542403-f5cb-4c50-ae21-206835cf448c None admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:41:04.471000 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6542403-f5cb-4c50-ae21-206835cf448c None admin] PATH_INFO: `/v3/projects/486401ae2e5f40d5a5f060242896ae53/groups/b272e49032f04e138cd631884e9f8b93/roles/e3b90948645643e888620ab1e5c774bf` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:41:04.473181 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6542403-f5cb-4c50-ae21-206835cf448c None admin] RBAC: Authorizing `identity:create_grant(project_id=486401ae2e5f40d5a5f060242896ae53, group_id=b272e49032f04e138cd631884e9f8b93, role_id=e3b90948645643e888620ab1e5c774bf)` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:41:04.474661 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6542403-f5cb-4c50-ae21-206835cf448c None admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:41:04.566383 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 201/387] 10.222.0.30 () {58 vars in 1442 bytes} [Mon May 23 12:41:04 2022] PUT /identity/v3/projects/486401ae2e5f40d5a5f060242896ae53/groups/b272e49032f04e138cd631884e9f8b93/roles/e3b90948645643e888620ab1e5c774bf => generated 0 bytes in 254 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) May 23 12:41:04.873402 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2309aa33-707e-491a-811e-183212234b23 None None] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:41:04.874098 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2309aa33-707e-491a-811e-183212234b23 None None] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:41:04.874570 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2309aa33-707e-491a-811e-183212234b23 None None] PATH_INFO: `/` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:41:04.876214 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 187/388] 10.222.0.30 () {54 vars in 926 bytes} [Mon May 23 12:41:04 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) May 23 12:41:04.882442 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c330396-32fb-48c1-94b7-0d29e053a1dd None None] REQUEST_METHOD: `POST` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:41:04.882734 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c330396-32fb-48c1-94b7-0d29e053a1dd None None] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:41:04.883015 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c330396-32fb-48c1-94b7-0d29e053a1dd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:41:04.901557 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.auth.core [None req-2c330396-32fb-48c1-94b7-0d29e053a1dd None None] MFA Rules not processed for user `0f7119f8fa5c4445977671140ded91a4`. Rule list: `[]` (Enabled: `True`). {{(pid=92056) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 12:41:04.902769 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.fernet_utils [None req-2c330396-32fb-48c1-94b7-0d29e053a1dd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92056) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 12:41:04.905056 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:41:04.974571 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 202/389] 10.222.0.30 () {58 vars in 1023 bytes} [Mon May 23 12:41:04 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 93 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) May 23 12:41:04.997457 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2309aa33-707e-491a-811e-183212234b23 None None] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:41:05.059864 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=100 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:41:05.127565 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2309aa33-707e-491a-811e-183212234b23 None None] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:41:05.183834 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a30623b-4eda-45d5-990b-83bfd930ce35 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:41:05.184647 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a30623b-4eda-45d5-990b-83bfd930ce35 None admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:41:05.184934 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a30623b-4eda-45d5-990b-83bfd930ce35 None admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:41:05.185213 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a30623b-4eda-45d5-990b-83bfd930ce35 None admin] PATH_INFO: `/v3/roles/member` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:41:05.204856 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a30623b-4eda-45d5-990b-83bfd930ce35 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} May 23 12:41:05.212875 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application [None req-0a30623b-4eda-45d5-990b-83bfd930ce35 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. May 23 12:41:05.212875 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application Traceback (most recent call last): May 23 12:41:05.212875 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request May 23 12:41:05.212875 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application rv = self.dispatch_request() May 23 12:41:05.212875 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request May 23 12:41:05.212875 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) May 23 12:41:05.212875 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper May 23 12:41:05.212875 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) May 23 12:41:05.212875 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view May 23 12:41:05.212875 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) May 23 12:41:05.212875 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request May 23 12:41:05.212875 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) May 23 12:41:05.212875 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get May 23 12:41:05.212875 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self._get_role(role_id) May 23 12:41:05.212875 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role May 23 12:41:05.212875 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') May 23 12:41:05.213819 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call May 23 12:41:05.213819 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( May 23 12:41:05.213819 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data May 23 12:41:05.213819 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) May 23 12:41:05.213819 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped May 23 12:41:05.213819 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) May 23 12:41:05.213819 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun May 23 12:41:05.213819 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) May 23 12:41:05.213819 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func May 23 12:41:05.213819 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self.get_or_create( May 23 12:41:05.213819 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create May 23 12:41:05.213819 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application with Lock( May 23 12:41:05.213819 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ May 23 12:41:05.213819 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self._enter() May 23 12:41:05.213819 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter May 23 12:41:05.213819 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) May 23 12:41:05.215069 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create May 23 12:41:05.215069 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self.creator() May 23 12:41:05.215069 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value May 23 12:41:05.215069 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application created_value = creator( May 23 12:41:05.215069 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role May 23 12:41:05.215069 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) May 23 12:41:05.215069 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role May 23 12:41:05.215069 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() May 23 12:41:05.215069 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role May 23 12:41:05.215069 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) May 23 12:41:05.215069 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. May 23 12:41:05.215069 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application May 23 12:41:05.215069 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 188/390] 10.222.0.30 () {56 vars in 1099 bytes} [Mon May 23 12:41:04 2022] GET /identity/v3/roles/member => generated 84 bytes in 217 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) May 23 12:41:05.220008 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c330396-32fb-48c1-94b7-0d29e053a1dd None None] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:41:05.373487 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c330396-32fb-48c1-94b7-0d29e053a1dd None None] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:41:05.377180 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-830ec845-419c-40f9-aabd-51f6e2ebf512 None None] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:41:05.377474 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-830ec845-419c-40f9-aabd-51f6e2ebf512 None None] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:41:05.377760 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-830ec845-419c-40f9-aabd-51f6e2ebf512 None None] PATH_INFO: `/` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:41:05.379191 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 189/391] 10.222.0.30 () {54 vars in 926 bytes} [Mon May 23 12:41:05 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) May 23 12:41:05.385284 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aed2ba42-ceb9-4141-b110-ba83a01fd6b4 None None] REQUEST_METHOD: `POST` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:41:05.385574 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aed2ba42-ceb9-4141-b110-ba83a01fd6b4 None None] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:41:05.385863 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aed2ba42-ceb9-4141-b110-ba83a01fd6b4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:41:05.411513 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.auth.core [None req-aed2ba42-ceb9-4141-b110-ba83a01fd6b4 None None] MFA Rules not processed for user `0f7119f8fa5c4445977671140ded91a4`. Rule list: `[]` (Enabled: `True`). {{(pid=92057) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 12:41:05.412784 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.fernet_utils [None req-aed2ba42-ceb9-4141-b110-ba83a01fd6b4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92057) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 12:41:05.441493 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc6ec3f9-e9a2-4117-8848-0696883babab None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:41:05.442186 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc6ec3f9-e9a2-4117-8848-0696883babab None admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:41:05.442467 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc6ec3f9-e9a2-4117-8848-0696883babab None admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:41:05.442737 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc6ec3f9-e9a2-4117-8848-0696883babab None admin] PATH_INFO: `/v3/roles` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:41:05.443484 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc6ec3f9-e9a2-4117-8848-0696883babab None admin] RBAC: Adding query filter params (name=member) {{(pid=92056) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} May 23 12:41:05.443788 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc6ec3f9-e9a2-4117-8848-0696883babab None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:41:05.445162 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc6ec3f9-e9a2-4117-8848-0696883babab None admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:41:05.453871 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=100 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:41:05.467852 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 203/392] 10.222.0.30 () {56 vars in 1101 bytes} [Mon May 23 12:41:05 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 247 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) May 23 12:41:05.469763 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc6ec3f9-e9a2-4117-8848-0696883babab None admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:41:05.568178 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 190/393] 10.222.0.30 () {58 vars in 1023 bytes} [Mon May 23 12:41:05 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 184 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) May 23 12:41:05.575970 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc6ec3f9-e9a2-4117-8848-0696883babab None admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:41:05.602931 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aed2ba42-ceb9-4141-b110-ba83a01fd6b4 None None] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:41:05.615710 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16b77297-7c34-40a0-b3ae-04618128e448 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:41:05.616628 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16b77297-7c34-40a0-b3ae-04618128e448 None admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:41:05.616917 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16b77297-7c34-40a0-b3ae-04618128e448 None admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:41:05.617195 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16b77297-7c34-40a0-b3ae-04618128e448 None admin] PATH_INFO: `/v3/groups/b272e49032f04e138cd631884e9f8b93` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:41:05.619016 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16b77297-7c34-40a0-b3ae-04618128e448 None admin] RBAC: Authorizing `identity:get_group(group_id=b272e49032f04e138cd631884e9f8b93)` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:41:05.620508 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16b77297-7c34-40a0-b3ae-04618128e448 None admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:41:05.622455 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 204/394] 10.222.0.30 () {56 vars in 1180 bytes} [Mon May 23 12:41:05 2022] GET /identity/v3/groups/b272e49032f04e138cd631884e9f8b93 => generated 226 bytes in 153 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) May 23 12:41:05.627955 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16b77297-7c34-40a0-b3ae-04618128e448 None admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:41:05.691889 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=100 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:41:05.706405 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=100 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:41:05.737150 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16b77297-7c34-40a0-b3ae-04618128e448 None admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:41:05.776353 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74b17bbc-d888-4350-b62d-ea7eb3afe189 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:41:05.777218 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74b17bbc-d888-4350-b62d-ea7eb3afe189 None admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:41:05.777494 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74b17bbc-d888-4350-b62d-ea7eb3afe189 None admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:41:05.777765 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74b17bbc-d888-4350-b62d-ea7eb3afe189 None admin] PATH_INFO: `/v3/projects/486401ae2e5f40d5a5f060242896ae53` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:41:05.779459 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74b17bbc-d888-4350-b62d-ea7eb3afe189 None admin] RBAC: Authorizing `identity:get_project(project_id=486401ae2e5f40d5a5f060242896ae53)` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:41:05.781856 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74b17bbc-d888-4350-b62d-ea7eb3afe189 None admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:41:05.786187 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 205/395] 10.222.0.30 () {56 vars in 1186 bytes} [Mon May 23 12:41:05 2022] GET /identity/v3/projects/486401ae2e5f40d5a5f060242896ae53 => generated 298 bytes in 159 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) May 23 12:41:05.797750 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74b17bbc-d888-4350-b62d-ea7eb3afe189 None admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:41:05.828563 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aed2ba42-ceb9-4141-b110-ba83a01fd6b4 None None] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:41:05.919581 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45f65341-d56e-434c-a9c6-5458ce0da215 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:41:05.920733 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45f65341-d56e-434c-a9c6-5458ce0da215 None admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:41:05.921212 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45f65341-d56e-434c-a9c6-5458ce0da215 None admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:41:05.921698 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45f65341-d56e-434c-a9c6-5458ce0da215 None admin] PATH_INFO: `/v3/roles/member` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:41:05.928473 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74b17bbc-d888-4350-b62d-ea7eb3afe189 None admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:41:05.954842 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45f65341-d56e-434c-a9c6-5458ce0da215 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} May 23 12:41:05.966892 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application [None req-45f65341-d56e-434c-a9c6-5458ce0da215 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. May 23 12:41:05.966892 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application Traceback (most recent call last): May 23 12:41:05.966892 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request May 23 12:41:05.966892 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application rv = self.dispatch_request() May 23 12:41:05.966892 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request May 23 12:41:05.966892 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) May 23 12:41:05.966892 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper May 23 12:41:05.966892 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) May 23 12:41:05.966892 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view May 23 12:41:05.966892 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) May 23 12:41:05.966892 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request May 23 12:41:05.966892 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) May 23 12:41:05.966892 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get May 23 12:41:05.966892 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self._get_role(role_id) May 23 12:41:05.966892 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role May 23 12:41:05.966892 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') May 23 12:41:05.967896 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call May 23 12:41:05.967896 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( May 23 12:41:05.967896 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data May 23 12:41:05.967896 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) May 23 12:41:05.967896 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped May 23 12:41:05.967896 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) May 23 12:41:05.967896 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun May 23 12:41:05.967896 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) May 23 12:41:05.967896 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func May 23 12:41:05.967896 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self.get_or_create( May 23 12:41:05.967896 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create May 23 12:41:05.967896 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application with Lock( May 23 12:41:05.967896 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ May 23 12:41:05.967896 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self._enter() May 23 12:41:05.967896 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter May 23 12:41:05.967896 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) May 23 12:41:05.968718 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create May 23 12:41:05.968718 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self.creator() May 23 12:41:05.968718 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value May 23 12:41:05.968718 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application created_value = creator( May 23 12:41:05.968718 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role May 23 12:41:05.968718 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) May 23 12:41:05.968718 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role May 23 12:41:05.968718 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() May 23 12:41:05.968718 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role May 23 12:41:05.968718 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) May 23 12:41:05.968718 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. May 23 12:41:05.968718 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application May 23 12:41:05.972939 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8362a80c-77dd-4f75-8cb7-e15b6f930570 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:41:05.974101 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8362a80c-77dd-4f75-8cb7-e15b6f930570 None admin] REQUEST_METHOD: `PUT` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:41:05.974324 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 191/396] 10.222.0.30 () {56 vars in 1099 bytes} [Mon May 23 12:41:05 2022] GET /identity/v3/roles/member => generated 84 bytes in 370 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) May 23 12:41:05.974795 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8362a80c-77dd-4f75-8cb7-e15b6f930570 None admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:41:05.975598 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8362a80c-77dd-4f75-8cb7-e15b6f930570 None admin] PATH_INFO: `/v3/projects/486401ae2e5f40d5a5f060242896ae53/groups/b272e49032f04e138cd631884e9f8b93/roles/80fb22507c9a4ff09d44d8f12f6fff1a` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:41:05.978768 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45f65341-d56e-434c-a9c6-5458ce0da215 None admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:41:05.980420 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8362a80c-77dd-4f75-8cb7-e15b6f930570 None admin] RBAC: Authorizing `identity:create_grant(project_id=486401ae2e5f40d5a5f060242896ae53, group_id=b272e49032f04e138cd631884e9f8b93, role_id=80fb22507c9a4ff09d44d8f12f6fff1a)` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:41:05.984223 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8362a80c-77dd-4f75-8cb7-e15b6f930570 None admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:41:05.989189 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:41:06.070772 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 206/397] 10.222.0.30 () {58 vars in 1442 bytes} [Mon May 23 12:41:05 2022] PUT /identity/v3/projects/486401ae2e5f40d5a5f060242896ae53/groups/b272e49032f04e138cd631884e9f8b93/roles/80fb22507c9a4ff09d44d8f12f6fff1a => generated 0 bytes in 273 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) May 23 12:41:06.120651 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=100 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:41:06.127121 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45f65341-d56e-434c-a9c6-5458ce0da215 None admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:41:06.178626 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ceb72b7-edea-42e5-8a01-8a66813bbcec None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:41:06.179981 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ceb72b7-edea-42e5-8a01-8a66813bbcec None admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:41:06.181103 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ceb72b7-edea-42e5-8a01-8a66813bbcec None admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:41:06.181629 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ceb72b7-edea-42e5-8a01-8a66813bbcec None admin] PATH_INFO: `/v3/roles` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:41:06.182768 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ceb72b7-edea-42e5-8a01-8a66813bbcec None admin] RBAC: Adding query filter params (name=member) {{(pid=92057) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} May 23 12:41:06.183287 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ceb72b7-edea-42e5-8a01-8a66813bbcec None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:41:06.184975 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ceb72b7-edea-42e5-8a01-8a66813bbcec None admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:41:06.195978 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 192/398] 10.222.0.30 () {56 vars in 1101 bytes} [Mon May 23 12:41:05 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 219 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) May 23 12:41:06.202264 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8362a80c-77dd-4f75-8cb7-e15b6f930570 None admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:41:06.328096 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8362a80c-77dd-4f75-8cb7-e15b6f930570 None admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:41:06.365872 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60255fdd-7bda-4543-b00e-b55666850895 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:41:06.367152 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60255fdd-7bda-4543-b00e-b55666850895 None admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:41:06.367869 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60255fdd-7bda-4543-b00e-b55666850895 None admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:41:06.368551 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60255fdd-7bda-4543-b00e-b55666850895 None admin] PATH_INFO: `/v3/groups/b272e49032f04e138cd631884e9f8b93` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:41:06.370008 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60255fdd-7bda-4543-b00e-b55666850895 None admin] RBAC: Authorizing `identity:get_group(group_id=b272e49032f04e138cd631884e9f8b93)` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:41:06.371745 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60255fdd-7bda-4543-b00e-b55666850895 None admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:41:06.373739 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 207/399] 10.222.0.30 () {56 vars in 1180 bytes} [Mon May 23 12:41:06 2022] GET /identity/v3/groups/b272e49032f04e138cd631884e9f8b93 => generated 226 bytes in 172 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) May 23 12:41:06.383328 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ceb72b7-edea-42e5-8a01-8a66813bbcec None admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:41:06.459128 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d19cc24-e1f5-4574-8e83-cd5f666be75d None None] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:41:06.459400 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d19cc24-e1f5-4574-8e83-cd5f666be75d None None] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:41:06.459661 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d19cc24-e1f5-4574-8e83-cd5f666be75d None None] PATH_INFO: `/` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:41:06.461301 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 208/400] 10.222.0.30 () {54 vars in 926 bytes} [Mon May 23 12:41:06 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) May 23 12:41:06.467033 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb3c8d42-ee37-4b9d-a1b4-f968e151b844 None None] REQUEST_METHOD: `POST` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:41:06.467324 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb3c8d42-ee37-4b9d-a1b4-f968e151b844 None None] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:41:06.467604 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb3c8d42-ee37-4b9d-a1b4-f968e151b844 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:41:06.485937 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.auth.core [None req-eb3c8d42-ee37-4b9d-a1b4-f968e151b844 None None] MFA Rules not processed for user `0f7119f8fa5c4445977671140ded91a4`. Rule list: `[]` (Enabled: `True`). {{(pid=92056) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 12:41:06.487111 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.fernet_utils [None req-eb3c8d42-ee37-4b9d-a1b4-f968e151b844 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92056) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 12:41:06.501715 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ceb72b7-edea-42e5-8a01-8a66813bbcec None admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:41:06.539358 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-360a58f7-914a-44bf-b430-5f3f6c612dcf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:41:06.541238 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-360a58f7-914a-44bf-b430-5f3f6c612dcf None admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:41:06.541662 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-360a58f7-914a-44bf-b430-5f3f6c612dcf None admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:41:06.542216 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-360a58f7-914a-44bf-b430-5f3f6c612dcf None admin] PATH_INFO: `/v3/projects/b39742b1d1b34dfe82a9ce4da39b7582` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:41:06.543425 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-360a58f7-914a-44bf-b430-5f3f6c612dcf None admin] RBAC: Authorizing `identity:get_project(project_id=b39742b1d1b34dfe82a9ce4da39b7582)` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:41:06.545014 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-360a58f7-914a-44bf-b430-5f3f6c612dcf None admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:41:06.547204 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 193/401] 10.222.0.30 () {56 vars in 1186 bytes} [Mon May 23 12:41:06 2022] GET /identity/v3/projects/b39742b1d1b34dfe82a9ce4da39b7582 => generated 302 bytes in 164 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) May 23 12:41:06.548470 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e9b4fbd-d269-42fc-a03e-59a368368722 None None] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:41:06.548757 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e9b4fbd-d269-42fc-a03e-59a368368722 None None] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:41:06.549029 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e9b4fbd-d269-42fc-a03e-59a368368722 None None] PATH_INFO: `/` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:41:06.550665 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 194/402] 10.222.0.30 () {54 vars in 926 bytes} [Mon May 23 12:41:06 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) May 23 12:41:06.555585 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e9b4fbd-d269-42fc-a03e-59a368368722 None None] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:41:06.564688 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=100 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:41:06.575870 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 209/403] 10.222.0.30 () {58 vars in 1023 bytes} [Mon May 23 12:41:06 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 110 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) May 23 12:41:06.578513 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5406285-c9f8-47be-b0a5-6e218e1625ff None None] REQUEST_METHOD: `POST` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:41:06.578801 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5406285-c9f8-47be-b0a5-6e218e1625ff None None] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:41:06.579071 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5406285-c9f8-47be-b0a5-6e218e1625ff None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:41:06.606029 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=100 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:41:06.615285 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.auth.core [None req-c5406285-c9f8-47be-b0a5-6e218e1625ff None None] MFA Rules not processed for user `0f7119f8fa5c4445977671140ded91a4`. Rule list: `[]` (Enabled: `True`). {{(pid=92056) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 12:41:06.616708 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.fernet_utils [None req-c5406285-c9f8-47be-b0a5-6e218e1625ff None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92056) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 12:41:06.706671 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e9b4fbd-d269-42fc-a03e-59a368368722 None None] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:41:06.714142 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 210/404] 10.222.0.30 () {58 vars in 1023 bytes} [Mon May 23 12:41:06 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 131 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) May 23 12:41:06.715595 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5406285-c9f8-47be-b0a5-6e218e1625ff None None] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:41:06.747794 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6642c243-8b18-428c-9064-99adcb030a88 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:41:06.749321 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6642c243-8b18-428c-9064-99adcb030a88 None admin] REQUEST_METHOD: `PUT` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:41:06.750221 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6642c243-8b18-428c-9064-99adcb030a88 None admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:41:06.750704 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6642c243-8b18-428c-9064-99adcb030a88 None admin] PATH_INFO: `/v3/projects/b39742b1d1b34dfe82a9ce4da39b7582/groups/b272e49032f04e138cd631884e9f8b93/roles/80fb22507c9a4ff09d44d8f12f6fff1a` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:41:06.753436 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6642c243-8b18-428c-9064-99adcb030a88 None admin] RBAC: Authorizing `identity:create_grant(project_id=b39742b1d1b34dfe82a9ce4da39b7582, group_id=b272e49032f04e138cd631884e9f8b93, role_id=80fb22507c9a4ff09d44d8f12f6fff1a)` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:41:06.755670 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6642c243-8b18-428c-9064-99adcb030a88 None admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:41:06.839596 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5406285-c9f8-47be-b0a5-6e218e1625ff None None] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:41:06.860416 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 195/405] 10.222.0.30 () {58 vars in 1442 bytes} [Mon May 23 12:41:06 2022] PUT /identity/v3/projects/b39742b1d1b34dfe82a9ce4da39b7582/groups/b272e49032f04e138cd631884e9f8b93/roles/80fb22507c9a4ff09d44d8f12f6fff1a => generated 0 bytes in 305 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) May 23 12:41:06.862355 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6642c243-8b18-428c-9064-99adcb030a88 None admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:41:06.876751 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=100 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:41:06.898646 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3253135d-9670-4e69-a8e3-b05937afd75e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:41:06.901806 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3253135d-9670-4e69-a8e3-b05937afd75e None admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:41:06.901806 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3253135d-9670-4e69-a8e3-b05937afd75e None admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:41:06.901806 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3253135d-9670-4e69-a8e3-b05937afd75e None admin] PATH_INFO: `/v3/roles/admin` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:41:06.928242 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3253135d-9670-4e69-a8e3-b05937afd75e None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} May 23 12:41:06.942365 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application [None req-3253135d-9670-4e69-a8e3-b05937afd75e None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. May 23 12:41:06.942365 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application Traceback (most recent call last): May 23 12:41:06.942365 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request May 23 12:41:06.942365 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application rv = self.dispatch_request() May 23 12:41:06.942365 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request May 23 12:41:06.942365 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) May 23 12:41:06.942365 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper May 23 12:41:06.942365 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) May 23 12:41:06.942365 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view May 23 12:41:06.942365 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) May 23 12:41:06.942365 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request May 23 12:41:06.942365 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) May 23 12:41:06.942365 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get May 23 12:41:06.942365 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self._get_role(role_id) May 23 12:41:06.942365 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role May 23 12:41:06.942365 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') May 23 12:41:06.943411 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call May 23 12:41:06.943411 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( May 23 12:41:06.943411 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data May 23 12:41:06.943411 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) May 23 12:41:06.943411 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped May 23 12:41:06.943411 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) May 23 12:41:06.943411 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun May 23 12:41:06.943411 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) May 23 12:41:06.943411 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func May 23 12:41:06.943411 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self.get_or_create( May 23 12:41:06.943411 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create May 23 12:41:06.943411 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application with Lock( May 23 12:41:06.943411 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ May 23 12:41:06.943411 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self._enter() May 23 12:41:06.943411 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter May 23 12:41:06.943411 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) May 23 12:41:06.944266 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create May 23 12:41:06.944266 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self.creator() May 23 12:41:06.944266 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value May 23 12:41:06.944266 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application created_value = creator( May 23 12:41:06.944266 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role May 23 12:41:06.944266 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) May 23 12:41:06.944266 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role May 23 12:41:06.944266 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() May 23 12:41:06.944266 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role May 23 12:41:06.944266 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) May 23 12:41:06.944266 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. May 23 12:41:06.944266 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application May 23 12:41:06.944266 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 211/406] 10.222.0.30 () {56 vars in 1096 bytes} [Mon May 23 12:41:06 2022] GET /identity/v3/roles/admin => generated 83 bytes in 229 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) May 23 12:41:06.952704 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3253135d-9670-4e69-a8e3-b05937afd75e None admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:41:07.013077 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:41:07.030759 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6642c243-8b18-428c-9064-99adcb030a88 None admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:41:07.076156 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-457b6952-4920-40c2-a339-59878e8ceefb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:41:07.077059 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-457b6952-4920-40c2-a339-59878e8ceefb None admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:41:07.077421 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-457b6952-4920-40c2-a339-59878e8ceefb None admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:41:07.077759 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-457b6952-4920-40c2-a339-59878e8ceefb None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:41:07.093388 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-457b6952-4920-40c2-a339-59878e8ceefb None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} May 23 12:41:07.103965 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application [None req-457b6952-4920-40c2-a339-59878e8ceefb None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. May 23 12:41:07.103965 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application Traceback (most recent call last): May 23 12:41:07.103965 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request May 23 12:41:07.103965 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application rv = self.dispatch_request() May 23 12:41:07.103965 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request May 23 12:41:07.103965 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) May 23 12:41:07.103965 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper May 23 12:41:07.103965 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) May 23 12:41:07.103965 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view May 23 12:41:07.103965 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) May 23 12:41:07.103965 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request May 23 12:41:07.103965 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) May 23 12:41:07.103965 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get May 23 12:41:07.103965 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self._get_role(role_id) May 23 12:41:07.103965 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role May 23 12:41:07.103965 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') May 23 12:41:07.105667 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call May 23 12:41:07.105667 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( May 23 12:41:07.105667 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data May 23 12:41:07.105667 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) May 23 12:41:07.105667 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped May 23 12:41:07.105667 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) May 23 12:41:07.105667 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun May 23 12:41:07.105667 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) May 23 12:41:07.105667 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func May 23 12:41:07.105667 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self.get_or_create( May 23 12:41:07.105667 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create May 23 12:41:07.105667 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application with Lock( May 23 12:41:07.105667 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ May 23 12:41:07.105667 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self._enter() May 23 12:41:07.105667 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter May 23 12:41:07.105667 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) May 23 12:41:07.106819 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create May 23 12:41:07.106819 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self.creator() May 23 12:41:07.106819 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value May 23 12:41:07.106819 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application created_value = creator( May 23 12:41:07.106819 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role May 23 12:41:07.106819 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) May 23 12:41:07.106819 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role May 23 12:41:07.106819 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() May 23 12:41:07.106819 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role May 23 12:41:07.106819 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) May 23 12:41:07.106819 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. May 23 12:41:07.106819 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application May 23 12:41:07.106819 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 196/407] 10.222.0.30 () {56 vars in 1114 bytes} [Mon May 23 12:41:06 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 244 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) May 23 12:41:07.112046 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-457b6952-4920-40c2-a339-59878e8ceefb None admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:41:07.208377 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3253135d-9670-4e69-a8e3-b05937afd75e None admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:41:07.248182 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-457b6952-4920-40c2-a339-59878e8ceefb None admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:41:07.291126 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1a5a30e-6596-415e-b1bb-16e10831e041 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:41:07.292284 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1a5a30e-6596-415e-b1bb-16e10831e041 None admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:41:07.292952 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1a5a30e-6596-415e-b1bb-16e10831e041 None admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:41:07.293727 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1a5a30e-6596-415e-b1bb-16e10831e041 None admin] PATH_INFO: `/v3/roles` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:41:07.294732 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1a5a30e-6596-415e-b1bb-16e10831e041 None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=92057) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} May 23 12:41:07.295469 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1a5a30e-6596-415e-b1bb-16e10831e041 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:41:07.298144 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1a5a30e-6596-415e-b1bb-16e10831e041 None admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:41:07.311435 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 197/408] 10.222.0.30 () {56 vars in 1111 bytes} [Mon May 23 12:41:07 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 201 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) May 23 12:41:07.318126 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1a5a30e-6596-415e-b1bb-16e10831e041 None admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:41:07.330600 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7f87c6f-3e37-446e-9ba1-5170f2a984df None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:41:07.331287 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7f87c6f-3e37-446e-9ba1-5170f2a984df None admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:41:07.331569 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7f87c6f-3e37-446e-9ba1-5170f2a984df None admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:41:07.331842 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7f87c6f-3e37-446e-9ba1-5170f2a984df None admin] PATH_INFO: `/v3/roles` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:41:07.332609 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7f87c6f-3e37-446e-9ba1-5170f2a984df None admin] RBAC: Adding query filter params (name=admin) {{(pid=92056) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} May 23 12:41:07.332914 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7f87c6f-3e37-446e-9ba1-5170f2a984df None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:41:07.334572 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7f87c6f-3e37-446e-9ba1-5170f2a984df None admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:41:07.347025 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=100 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:41:07.355857 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 212/409] 10.222.0.30 () {56 vars in 1099 bytes} [Mon May 23 12:41:06 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 404 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) May 23 12:41:07.361577 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7f87c6f-3e37-446e-9ba1-5170f2a984df None admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:41:07.455351 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1a5a30e-6596-415e-b1bb-16e10831e041 None admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:41:07.462981 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=100 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:41:07.503165 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8216e28a-685e-4b0a-b0ce-d5da9b9dba58 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:41:07.504466 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8216e28a-685e-4b0a-b0ce-d5da9b9dba58 None admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:41:07.504944 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8216e28a-685e-4b0a-b0ce-d5da9b9dba58 None admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:41:07.505403 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8216e28a-685e-4b0a-b0ce-d5da9b9dba58 None admin] PATH_INFO: `/v3/groups/b272e49032f04e138cd631884e9f8b93` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:41:07.506955 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8216e28a-685e-4b0a-b0ce-d5da9b9dba58 None admin] RBAC: Authorizing `identity:get_group(group_id=b272e49032f04e138cd631884e9f8b93)` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:41:07.509547 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8216e28a-685e-4b0a-b0ce-d5da9b9dba58 None admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:41:07.512084 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 198/410] 10.222.0.30 () {56 vars in 1180 bytes} [Mon May 23 12:41:07 2022] GET /identity/v3/groups/b272e49032f04e138cd631884e9f8b93 => generated 226 bytes in 195 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) May 23 12:41:07.536669 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8216e28a-685e-4b0a-b0ce-d5da9b9dba58 None admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:41:07.566506 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7f87c6f-3e37-446e-9ba1-5170f2a984df None admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:41:07.602584 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-842e5728-c74d-42e7-8d5c-284989e82d03 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:41:07.603454 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-842e5728-c74d-42e7-8d5c-284989e82d03 None admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:41:07.603788 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-842e5728-c74d-42e7-8d5c-284989e82d03 None admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:41:07.604011 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-842e5728-c74d-42e7-8d5c-284989e82d03 None admin] PATH_INFO: `/v3/groups/33d7b939d9664f90b8df13bffb850ca4` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:41:07.605553 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-842e5728-c74d-42e7-8d5c-284989e82d03 None admin] RBAC: Authorizing `identity:get_group(group_id=33d7b939d9664f90b8df13bffb850ca4)` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:41:07.606720 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-842e5728-c74d-42e7-8d5c-284989e82d03 None admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:41:07.608736 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 213/411] 10.222.0.30 () {56 vars in 1180 bytes} [Mon May 23 12:41:07 2022] GET /identity/v3/groups/33d7b939d9664f90b8df13bffb850ca4 => generated 229 bytes in 247 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) May 23 12:41:07.614765 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-842e5728-c74d-42e7-8d5c-284989e82d03 None admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:41:07.661196 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=100 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:41:07.710083 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8216e28a-685e-4b0a-b0ce-d5da9b9dba58 None admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:41:07.754711 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82aadda6-c33f-4f6f-8b83-6028d9d15b25 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:41:07.755582 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82aadda6-c33f-4f6f-8b83-6028d9d15b25 None admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:41:07.755863 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82aadda6-c33f-4f6f-8b83-6028d9d15b25 None admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:41:07.756136 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82aadda6-c33f-4f6f-8b83-6028d9d15b25 None admin] PATH_INFO: `/v3/projects/b39742b1d1b34dfe82a9ce4da39b7582` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:41:07.756443 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-842e5728-c74d-42e7-8d5c-284989e82d03 None admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:41:07.757347 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82aadda6-c33f-4f6f-8b83-6028d9d15b25 None admin] RBAC: Authorizing `identity:get_project(project_id=b39742b1d1b34dfe82a9ce4da39b7582)` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:41:07.759482 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82aadda6-c33f-4f6f-8b83-6028d9d15b25 None admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:41:07.768729 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 199/412] 10.222.0.30 () {56 vars in 1186 bytes} [Mon May 23 12:41:07 2022] GET /identity/v3/projects/b39742b1d1b34dfe82a9ce4da39b7582 => generated 302 bytes in 226 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) May 23 12:41:07.768729 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-706d8aa9-132d-4405-9235-7be2f15c86d0 None None] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:41:07.768729 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-706d8aa9-132d-4405-9235-7be2f15c86d0 None None] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:41:07.768729 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-706d8aa9-132d-4405-9235-7be2f15c86d0 None None] PATH_INFO: `/` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:41:07.768729 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 200/413] 10.222.0.30 () {54 vars in 926 bytes} [Mon May 23 12:41:07 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) May 23 12:41:07.768729 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-706d8aa9-132d-4405-9235-7be2f15c86d0 None None] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:41:07.791959 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=100 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:41:07.823541 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ab22237-a55e-4fe9-8381-42efd6d1767f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:41:07.824931 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ab22237-a55e-4fe9-8381-42efd6d1767f None admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:41:07.825868 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ab22237-a55e-4fe9-8381-42efd6d1767f None admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:41:07.826393 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ab22237-a55e-4fe9-8381-42efd6d1767f None admin] PATH_INFO: `/v3/projects/af8b84d1e3e44efba851eab6bed63282` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:41:07.828562 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ab22237-a55e-4fe9-8381-42efd6d1767f None admin] RBAC: Authorizing `identity:get_project(project_id=af8b84d1e3e44efba851eab6bed63282)` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:41:07.831137 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ab22237-a55e-4fe9-8381-42efd6d1767f None admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:41:07.834052 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 214/414] 10.222.0.30 () {56 vars in 1186 bytes} [Mon May 23 12:41:07 2022] GET /identity/v3/projects/af8b84d1e3e44efba851eab6bed63282 => generated 344 bytes in 220 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) May 23 12:41:07.838325 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-489cc07d-8a35-474e-9dfa-1bf5350a928f None None] REQUEST_METHOD: `POST` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:41:07.838820 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-489cc07d-8a35-474e-9dfa-1bf5350a928f None None] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:41:07.839276 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-489cc07d-8a35-474e-9dfa-1bf5350a928f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:41:07.859084 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.auth.core [None req-489cc07d-8a35-474e-9dfa-1bf5350a928f None None] MFA Rules not processed for user `0f7119f8fa5c4445977671140ded91a4`. Rule list: `[]` (Enabled: `True`). {{(pid=92056) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 12:41:07.860343 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.fernet_utils [None req-489cc07d-8a35-474e-9dfa-1bf5350a928f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92056) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 12:41:07.908407 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-706d8aa9-132d-4405-9235-7be2f15c86d0 None None] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:41:07.943310 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 215/415] 10.222.0.30 () {58 vars in 1023 bytes} [Mon May 23 12:41:07 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 108 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) May 23 12:41:07.944401 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-489cc07d-8a35-474e-9dfa-1bf5350a928f None None] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:41:07.963190 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a7ac12e-1930-4097-bdba-73d2c06ff433 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:41:07.964081 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a7ac12e-1930-4097-bdba-73d2c06ff433 None admin] REQUEST_METHOD: `PUT` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:41:07.964361 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a7ac12e-1930-4097-bdba-73d2c06ff433 None admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:41:07.964632 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a7ac12e-1930-4097-bdba-73d2c06ff433 None admin] PATH_INFO: `/v3/projects/b39742b1d1b34dfe82a9ce4da39b7582/groups/b272e49032f04e138cd631884e9f8b93/roles/e3b90948645643e888620ab1e5c774bf` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:41:07.973123 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a7ac12e-1930-4097-bdba-73d2c06ff433 None admin] RBAC: Authorizing `identity:create_grant(project_id=b39742b1d1b34dfe82a9ce4da39b7582, group_id=b272e49032f04e138cd631884e9f8b93, role_id=e3b90948645643e888620ab1e5c774bf)` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:41:07.974746 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a7ac12e-1930-4097-bdba-73d2c06ff433 None admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:41:08.056101 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 201/416] 10.222.0.30 () {58 vars in 1442 bytes} [Mon May 23 12:41:07 2022] PUT /identity/v3/projects/b39742b1d1b34dfe82a9ce4da39b7582/groups/b272e49032f04e138cd631884e9f8b93/roles/e3b90948645643e888620ab1e5c774bf => generated 0 bytes in 282 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) May 23 12:41:08.057190 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a7ac12e-1930-4097-bdba-73d2c06ff433 None admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:41:08.058678 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-489cc07d-8a35-474e-9dfa-1bf5350a928f None None] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:41:08.082390 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:41:08.095660 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=100 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:41:08.150568 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f50461a-4c39-44ca-a207-a223fabdc55d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:41:08.151516 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f50461a-4c39-44ca-a207-a223fabdc55d None admin] REQUEST_METHOD: `PUT` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:41:08.151806 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f50461a-4c39-44ca-a207-a223fabdc55d None admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:41:08.152090 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f50461a-4c39-44ca-a207-a223fabdc55d None admin] PATH_INFO: `/v3/projects/af8b84d1e3e44efba851eab6bed63282/groups/33d7b939d9664f90b8df13bffb850ca4/roles/bc3c3c9d654940cbaa5177a6be23d1bc` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:41:08.157328 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f50461a-4c39-44ca-a207-a223fabdc55d None admin] RBAC: Authorizing `identity:create_grant(project_id=af8b84d1e3e44efba851eab6bed63282, group_id=33d7b939d9664f90b8df13bffb850ca4, role_id=bc3c3c9d654940cbaa5177a6be23d1bc)` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:41:08.160850 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f50461a-4c39-44ca-a207-a223fabdc55d None admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:41:08.214227 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 216/417] 10.222.0.30 () {58 vars in 1442 bytes} [Mon May 23 12:41:07 2022] PUT /identity/v3/projects/af8b84d1e3e44efba851eab6bed63282/groups/33d7b939d9664f90b8df13bffb850ca4/roles/bc3c3c9d654940cbaa5177a6be23d1bc => generated 0 bytes in 270 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) May 23 12:41:08.252532 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a7ac12e-1930-4097-bdba-73d2c06ff433 None admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:41:08.295065 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54ae4f30-1048-4213-bd30-015a6dfc1026 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:41:08.295964 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54ae4f30-1048-4213-bd30-015a6dfc1026 None admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:41:08.296254 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54ae4f30-1048-4213-bd30-015a6dfc1026 None admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:41:08.296529 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54ae4f30-1048-4213-bd30-015a6dfc1026 None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:41:08.309264 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54ae4f30-1048-4213-bd30-015a6dfc1026 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} May 23 12:41:08.317839 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application [None req-54ae4f30-1048-4213-bd30-015a6dfc1026 None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. May 23 12:41:08.317839 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application Traceback (most recent call last): May 23 12:41:08.317839 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request May 23 12:41:08.317839 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application rv = self.dispatch_request() May 23 12:41:08.317839 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request May 23 12:41:08.317839 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) May 23 12:41:08.317839 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper May 23 12:41:08.317839 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) May 23 12:41:08.317839 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view May 23 12:41:08.317839 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) May 23 12:41:08.317839 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request May 23 12:41:08.317839 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) May 23 12:41:08.317839 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get May 23 12:41:08.317839 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self._get_role(role_id) May 23 12:41:08.317839 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role May 23 12:41:08.317839 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') May 23 12:41:08.318841 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call May 23 12:41:08.318841 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( May 23 12:41:08.318841 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data May 23 12:41:08.318841 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) May 23 12:41:08.318841 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped May 23 12:41:08.318841 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) May 23 12:41:08.318841 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun May 23 12:41:08.318841 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) May 23 12:41:08.318841 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func May 23 12:41:08.318841 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self.get_or_create( May 23 12:41:08.318841 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create May 23 12:41:08.318841 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application with Lock( May 23 12:41:08.318841 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ May 23 12:41:08.318841 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self._enter() May 23 12:41:08.318841 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter May 23 12:41:08.318841 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) May 23 12:41:08.320078 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create May 23 12:41:08.320078 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self.creator() May 23 12:41:08.320078 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value May 23 12:41:08.320078 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application created_value = creator( May 23 12:41:08.320078 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role May 23 12:41:08.320078 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) May 23 12:41:08.320078 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role May 23 12:41:08.320078 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() May 23 12:41:08.320078 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role May 23 12:41:08.320078 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) May 23 12:41:08.320078 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. May 23 12:41:08.320078 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application May 23 12:41:08.320078 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 202/418] 10.222.0.30 () {56 vars in 1114 bytes} [Mon May 23 12:41:08 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 262 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) May 23 12:41:08.325927 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f50461a-4c39-44ca-a207-a223fabdc55d None admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:41:08.505181 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f50461a-4c39-44ca-a207-a223fabdc55d None admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:41:08.557651 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57d1ef95-bed8-4a24-8262-8be5c975cb53 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:41:08.558747 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57d1ef95-bed8-4a24-8262-8be5c975cb53 None admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:41:08.559106 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57d1ef95-bed8-4a24-8262-8be5c975cb53 None admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:41:08.559578 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57d1ef95-bed8-4a24-8262-8be5c975cb53 None admin] PATH_INFO: `/v3/roles` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:41:08.560693 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57d1ef95-bed8-4a24-8262-8be5c975cb53 None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=92056) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} May 23 12:41:08.561172 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57d1ef95-bed8-4a24-8262-8be5c975cb53 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:41:08.563495 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57d1ef95-bed8-4a24-8262-8be5c975cb53 None admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:41:08.578640 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 217/419] 10.222.0.30 () {56 vars in 1111 bytes} [Mon May 23 12:41:08 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 253 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) May 23 12:41:08.586852 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54ae4f30-1048-4213-bd30-015a6dfc1026 None admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:41:08.716202 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=100 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:41:08.753268 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54ae4f30-1048-4213-bd30-015a6dfc1026 None admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:41:08.768615 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef00d3bb-7d59-455a-85ea-50fa2e8f42fa None None] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:41:08.768822 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef00d3bb-7d59-455a-85ea-50fa2e8f42fa None None] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:41:08.769104 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef00d3bb-7d59-455a-85ea-50fa2e8f42fa None None] PATH_INFO: `/` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:41:08.770742 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 218/420] 10.222.0.30 () {54 vars in 926 bytes} [Mon May 23 12:41:08 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) May 23 12:41:08.776665 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbdf7f8f-672e-4a76-ac85-ce9a166b7a89 None None] REQUEST_METHOD: `POST` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:41:08.776951 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbdf7f8f-672e-4a76-ac85-ce9a166b7a89 None None] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:41:08.777231 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbdf7f8f-672e-4a76-ac85-ce9a166b7a89 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:41:08.795099 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.auth.core [None req-cbdf7f8f-672e-4a76-ac85-ce9a166b7a89 None None] MFA Rules not processed for user `0f7119f8fa5c4445977671140ded91a4`. Rule list: `[]` (Enabled: `True`). {{(pid=92056) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 12:41:08.796381 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.fernet_utils [None req-cbdf7f8f-672e-4a76-ac85-ce9a166b7a89 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92056) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 12:41:08.803356 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0924589f-1d86-464d-beb1-f80ad498da7b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:41:08.804199 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0924589f-1d86-464d-beb1-f80ad498da7b None admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:41:08.804627 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0924589f-1d86-464d-beb1-f80ad498da7b None admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:41:08.805072 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0924589f-1d86-464d-beb1-f80ad498da7b None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:41:08.825365 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:41:08.878968 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 219/421] 10.222.0.30 () {58 vars in 1023 bytes} [Mon May 23 12:41:08 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 103 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) May 23 12:41:08.902855 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cbdf7f8f-672e-4a76-ac85-ce9a166b7a89 None None] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:41:08.911550 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0924589f-1d86-464d-beb1-f80ad498da7b None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:41:08.913345 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0924589f-1d86-464d-beb1-f80ad498da7b None admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:41:09.009933 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cbdf7f8f-672e-4a76-ac85-ce9a166b7a89 None None] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:41:09.045836 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16f5f59c-f39a-469a-8628-80021532abf1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:41:09.047642 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16f5f59c-f39a-469a-8628-80021532abf1 None admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:41:09.047642 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16f5f59c-f39a-469a-8628-80021532abf1 None admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:41:09.047642 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16f5f59c-f39a-469a-8628-80021532abf1 None admin] PATH_INFO: `/v3/roles/member` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:41:09.059867 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16f5f59c-f39a-469a-8628-80021532abf1 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} May 23 12:41:09.067099 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=100 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:41:09.068517 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application [None req-16f5f59c-f39a-469a-8628-80021532abf1 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. May 23 12:41:09.068517 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application Traceback (most recent call last): May 23 12:41:09.068517 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request May 23 12:41:09.068517 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application rv = self.dispatch_request() May 23 12:41:09.068517 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request May 23 12:41:09.068517 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) May 23 12:41:09.068517 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper May 23 12:41:09.068517 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) May 23 12:41:09.068517 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view May 23 12:41:09.068517 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) May 23 12:41:09.068517 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request May 23 12:41:09.068517 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) May 23 12:41:09.068517 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get May 23 12:41:09.068517 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self._get_role(role_id) May 23 12:41:09.068517 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role May 23 12:41:09.068517 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') May 23 12:41:09.069617 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call May 23 12:41:09.069617 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( May 23 12:41:09.069617 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data May 23 12:41:09.069617 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) May 23 12:41:09.069617 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped May 23 12:41:09.069617 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) May 23 12:41:09.069617 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun May 23 12:41:09.069617 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) May 23 12:41:09.069617 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func May 23 12:41:09.069617 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self.get_or_create( May 23 12:41:09.069617 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create May 23 12:41:09.069617 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application with Lock( May 23 12:41:09.069617 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ May 23 12:41:09.069617 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self._enter() May 23 12:41:09.069617 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter May 23 12:41:09.069617 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) May 23 12:41:09.070675 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create May 23 12:41:09.070675 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self.creator() May 23 12:41:09.070675 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value May 23 12:41:09.070675 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application created_value = creator( May 23 12:41:09.070675 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role May 23 12:41:09.070675 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) May 23 12:41:09.070675 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role May 23 12:41:09.070675 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() May 23 12:41:09.070675 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role May 23 12:41:09.070675 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) May 23 12:41:09.070675 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. May 23 12:41:09.070675 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application May 23 12:41:09.072555 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 220/422] 10.222.0.30 () {56 vars in 1099 bytes} [Mon May 23 12:41:08 2022] GET /identity/v3/roles/member => generated 84 bytes in 171 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) May 23 12:41:09.078390 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16f5f59c-f39a-469a-8628-80021532abf1 None admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:41:09.107251 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=100 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:41:09.145391 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 203/423] 10.222.0.30 () {58 vars in 1282 bytes} [Mon May 23 12:41:08 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 559 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) May 23 12:41:09.151058 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0924589f-1d86-464d-beb1-f80ad498da7b None admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:41:09.185993 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16f5f59c-f39a-469a-8628-80021532abf1 None admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:41:09.223627 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3903f921-caa6-42e2-b9e7-7d479d573e19 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:41:09.224441 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3903f921-caa6-42e2-b9e7-7d479d573e19 None admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:41:09.224644 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3903f921-caa6-42e2-b9e7-7d479d573e19 None admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:41:09.224920 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3903f921-caa6-42e2-b9e7-7d479d573e19 None admin] PATH_INFO: `/v3/roles` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:41:09.225736 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3903f921-caa6-42e2-b9e7-7d479d573e19 None admin] RBAC: Adding query filter params (name=member) {{(pid=92056) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} May 23 12:41:09.226081 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3903f921-caa6-42e2-b9e7-7d479d573e19 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:41:09.228163 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3903f921-caa6-42e2-b9e7-7d479d573e19 None admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:41:09.242257 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 221/424] 10.222.0.30 () {56 vars in 1101 bytes} [Mon May 23 12:41:09 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 164 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) May 23 12:41:09.244694 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3903f921-caa6-42e2-b9e7-7d479d573e19 None admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:41:09.334479 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=100 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:41:09.350998 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3903f921-caa6-42e2-b9e7-7d479d573e19 None admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:41:09.396779 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c53b5567-1b48-44b1-8fb0-6d4f8f2a3d0d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:41:09.397427 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c53b5567-1b48-44b1-8fb0-6d4f8f2a3d0d None admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:41:09.397707 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c53b5567-1b48-44b1-8fb0-6d4f8f2a3d0d None admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:41:09.398008 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c53b5567-1b48-44b1-8fb0-6d4f8f2a3d0d None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:41:09.445360 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0924589f-1d86-464d-beb1-f80ad498da7b None admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:41:09.471918 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c53b5567-1b48-44b1-8fb0-6d4f8f2a3d0d None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:41:09.473379 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c53b5567-1b48-44b1-8fb0-6d4f8f2a3d0d None admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:41:09.540025 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-240c8ba2-7d6c-444e-9684-3a8fcb002c7b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:41:09.541323 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-240c8ba2-7d6c-444e-9684-3a8fcb002c7b None admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:41:09.541825 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-240c8ba2-7d6c-444e-9684-3a8fcb002c7b None admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:41:09.542333 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-240c8ba2-7d6c-444e-9684-3a8fcb002c7b None admin] PATH_INFO: `/v3/projects/486401ae2e5f40d5a5f060242896ae53` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:41:09.550662 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-240c8ba2-7d6c-444e-9684-3a8fcb002c7b None admin] RBAC: Authorizing `identity:get_project(project_id=486401ae2e5f40d5a5f060242896ae53)` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:41:09.552379 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-240c8ba2-7d6c-444e-9684-3a8fcb002c7b None admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:41:09.554671 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 204/425] 10.222.0.30 () {56 vars in 1186 bytes} [Mon May 23 12:41:09 2022] GET /identity/v3/projects/486401ae2e5f40d5a5f060242896ae53 => generated 298 bytes in 404 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) May 23 12:41:09.560642 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-240c8ba2-7d6c-444e-9684-3a8fcb002c7b None admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:41:09.571947 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=100 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:41:09.580995 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 222/426] 10.222.0.30 () {58 vars in 1282 bytes} [Mon May 23 12:41:09 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 337 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) May 23 12:41:09.587147 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c53b5567-1b48-44b1-8fb0-6d4f8f2a3d0d None admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:41:09.666987 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=100 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:41:09.699301 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c53b5567-1b48-44b1-8fb0-6d4f8f2a3d0d None admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:41:09.737512 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f69c05e8-0c70-49ac-9516-b152cbfc3293 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:41:09.738882 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f69c05e8-0c70-49ac-9516-b152cbfc3293 None admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:41:09.739359 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f69c05e8-0c70-49ac-9516-b152cbfc3293 None admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:41:09.739845 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f69c05e8-0c70-49ac-9516-b152cbfc3293 None admin] PATH_INFO: `/v3/projects/486401ae2e5f40d5a5f060242896ae53` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:41:09.741272 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f69c05e8-0c70-49ac-9516-b152cbfc3293 None admin] RBAC: Authorizing `identity:get_project(project_id=486401ae2e5f40d5a5f060242896ae53)` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:41:09.743155 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f69c05e8-0c70-49ac-9516-b152cbfc3293 None admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:41:09.745453 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 223/427] 10.222.0.30 () {56 vars in 1186 bytes} [Mon May 23 12:41:09 2022] GET /identity/v3/projects/486401ae2e5f40d5a5f060242896ae53 => generated 298 bytes in 159 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) May 23 12:41:09.751018 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f69c05e8-0c70-49ac-9516-b152cbfc3293 None admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:41:09.810340 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=100 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:41:09.846851 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-240c8ba2-7d6c-444e-9684-3a8fcb002c7b None admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:41:09.857738 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f69c05e8-0c70-49ac-9516-b152cbfc3293 None admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:41:09.888885 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e009cb1-d5a5-4174-b955-d6984abddfd7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:41:09.889750 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e009cb1-d5a5-4174-b955-d6984abddfd7 None admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:41:09.890081 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e009cb1-d5a5-4174-b955-d6984abddfd7 None admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:41:09.890361 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e009cb1-d5a5-4174-b955-d6984abddfd7 None admin] PATH_INFO: `/v3/groups/b272e49032f04e138cd631884e9f8b93` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:41:09.891523 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e009cb1-d5a5-4174-b955-d6984abddfd7 None admin] RBAC: Authorizing `identity:get_group(group_id=b272e49032f04e138cd631884e9f8b93)` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:41:09.893206 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e009cb1-d5a5-4174-b955-d6984abddfd7 None admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:41:09.894932 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 205/428] 10.222.0.30 () {56 vars in 1180 bytes} [Mon May 23 12:41:09 2022] GET /identity/v3/groups/b272e49032f04e138cd631884e9f8b93 => generated 226 bytes in 334 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) May 23 12:41:09.905910 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e009cb1-d5a5-4174-b955-d6984abddfd7 None admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:41:09.926816 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-346294c9-c9dd-414a-ae7b-e492efb4d16c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:41:09.928110 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-346294c9-c9dd-414a-ae7b-e492efb4d16c None admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:41:09.928585 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-346294c9-c9dd-414a-ae7b-e492efb4d16c None admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:41:09.929046 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-346294c9-c9dd-414a-ae7b-e492efb4d16c None admin] PATH_INFO: `/v3/groups/b272e49032f04e138cd631884e9f8b93` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:41:09.933646 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-346294c9-c9dd-414a-ae7b-e492efb4d16c None admin] RBAC: Authorizing `identity:get_group(group_id=b272e49032f04e138cd631884e9f8b93)` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:41:09.935334 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-346294c9-c9dd-414a-ae7b-e492efb4d16c None admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:41:09.937453 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 224/429] 10.222.0.30 () {56 vars in 1180 bytes} [Mon May 23 12:41:09 2022] GET /identity/v3/groups/b272e49032f04e138cd631884e9f8b93 => generated 226 bytes in 187 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) May 23 12:41:09.943432 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-346294c9-c9dd-414a-ae7b-e492efb4d16c None admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:41:10.005367 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e009cb1-d5a5-4174-b955-d6984abddfd7 None admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:41:10.007552 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=100 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:41:10.041060 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4db10c9a-0cc3-4e8d-90b9-e8e3e6bf04bc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:41:10.042295 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4db10c9a-0cc3-4e8d-90b9-e8e3e6bf04bc None admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:41:10.042585 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4db10c9a-0cc3-4e8d-90b9-e8e3e6bf04bc None admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:41:10.043499 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4db10c9a-0cc3-4e8d-90b9-e8e3e6bf04bc None admin] PATH_INFO: `/v3/role_assignments` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:41:10.044352 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4db10c9a-0cc3-4e8d-90b9-e8e3e6bf04bc None admin] RBAC: Adding query filter params (group.id=b272e49032f04e138cd631884e9f8b93, role.id=e3b90948645643e888620ab1e5c774bf, scope.project.id=486401ae2e5f40d5a5f060242896ae53) {{(pid=92057) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} May 23 12:41:10.051302 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4db10c9a-0cc3-4e8d-90b9-e8e3e6bf04bc None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:41:10.051302 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4db10c9a-0cc3-4e8d-90b9-e8e3e6bf04bc None admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:41:10.052566 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 206/430] 10.222.0.30 () {56 vars in 1376 bytes} [Mon May 23 12:41:09 2022] GET /identity/v3/role_assignments?group.id=b272e49032f04e138cd631884e9f8b93&scope.project.id=486401ae2e5f40d5a5f060242896ae53&role.id=e3b90948645643e888620ab1e5c774bf => generated 611 bytes in 152 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) May 23 12:41:10.053816 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8751a605-c1f5-4814-a8b2-110b1bf4d965 None None] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:41:10.062352 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8751a605-c1f5-4814-a8b2-110b1bf4d965 None None] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:41:10.062675 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8751a605-c1f5-4814-a8b2-110b1bf4d965 None None] PATH_INFO: `/` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:41:10.062927 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-346294c9-c9dd-414a-ae7b-e492efb4d16c None admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:41:10.064439 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 207/431] 10.222.0.30 () {54 vars in 926 bytes} [Mon May 23 12:41:10 2022] GET /identity => generated 270 bytes in 11 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) May 23 12:41:10.071279 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d55a8cf-0e72-492a-8231-fb3e73bc659e None None] REQUEST_METHOD: `POST` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:41:10.071279 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d55a8cf-0e72-492a-8231-fb3e73bc659e None None] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:41:10.071279 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d55a8cf-0e72-492a-8231-fb3e73bc659e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:41:10.102512 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.auth.core [None req-3d55a8cf-0e72-492a-8231-fb3e73bc659e None None] MFA Rules not processed for user `0f7119f8fa5c4445977671140ded91a4`. Rule list: `[]` (Enabled: `True`). {{(pid=92057) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 12:41:10.103633 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.fernet_utils [None req-3d55a8cf-0e72-492a-8231-fb3e73bc659e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92057) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 12:41:10.106449 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=100 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:41:10.119958 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5362f800-39ee-42c9-979c-304b0f0f10f0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:41:10.121027 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5362f800-39ee-42c9-979c-304b0f0f10f0 None admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:41:10.121512 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5362f800-39ee-42c9-979c-304b0f0f10f0 None admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:41:10.122256 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5362f800-39ee-42c9-979c-304b0f0f10f0 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:41:10.123326 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5362f800-39ee-42c9-979c-304b0f0f10f0 None admin] RBAC: Adding query filter params (group.id=b272e49032f04e138cd631884e9f8b93, role.id=80fb22507c9a4ff09d44d8f12f6fff1a, scope.project.id=486401ae2e5f40d5a5f060242896ae53) {{(pid=92056) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} May 23 12:41:10.123851 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5362f800-39ee-42c9-979c-304b0f0f10f0 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:41:10.125702 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5362f800-39ee-42c9-979c-304b0f0f10f0 None admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:41:10.138072 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 225/432] 10.222.0.30 () {56 vars in 1376 bytes} [Mon May 23 12:41:09 2022] GET /identity/v3/role_assignments?group.id=b272e49032f04e138cd631884e9f8b93&scope.project.id=486401ae2e5f40d5a5f060242896ae53&role.id=80fb22507c9a4ff09d44d8f12f6fff1a => generated 611 bytes in 190 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) May 23 12:41:10.241911 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 208/433] 10.222.0.30 () {58 vars in 1023 bytes} [Mon May 23 12:41:10 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 172 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) May 23 12:41:10.281888 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5362f800-39ee-42c9-979c-304b0f0f10f0 None admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:41:10.411906 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5362f800-39ee-42c9-979c-304b0f0f10f0 None admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:41:10.450190 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85a89f3f-fe31-46b1-9be9-acac2cd844fd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:41:10.451372 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85a89f3f-fe31-46b1-9be9-acac2cd844fd None admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:41:10.451857 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85a89f3f-fe31-46b1-9be9-acac2cd844fd None admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:41:10.452332 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85a89f3f-fe31-46b1-9be9-acac2cd844fd None admin] PATH_INFO: `/v3/roles/member` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:41:10.465258 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85a89f3f-fe31-46b1-9be9-acac2cd844fd None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} May 23 12:41:10.473936 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application [None req-85a89f3f-fe31-46b1-9be9-acac2cd844fd None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. May 23 12:41:10.473936 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application Traceback (most recent call last): May 23 12:41:10.473936 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request May 23 12:41:10.473936 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application rv = self.dispatch_request() May 23 12:41:10.473936 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request May 23 12:41:10.473936 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) May 23 12:41:10.473936 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper May 23 12:41:10.473936 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) May 23 12:41:10.473936 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view May 23 12:41:10.473936 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) May 23 12:41:10.473936 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request May 23 12:41:10.473936 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) May 23 12:41:10.473936 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get May 23 12:41:10.473936 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self._get_role(role_id) May 23 12:41:10.473936 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role May 23 12:41:10.473936 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') May 23 12:41:10.474929 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call May 23 12:41:10.474929 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( May 23 12:41:10.474929 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data May 23 12:41:10.474929 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) May 23 12:41:10.474929 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped May 23 12:41:10.474929 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) May 23 12:41:10.474929 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun May 23 12:41:10.474929 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) May 23 12:41:10.474929 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func May 23 12:41:10.474929 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self.get_or_create( May 23 12:41:10.474929 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create May 23 12:41:10.474929 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application with Lock( May 23 12:41:10.474929 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ May 23 12:41:10.474929 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self._enter() May 23 12:41:10.474929 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter May 23 12:41:10.474929 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) May 23 12:41:10.475868 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create May 23 12:41:10.475868 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self.creator() May 23 12:41:10.475868 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value May 23 12:41:10.475868 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application created_value = creator( May 23 12:41:10.475868 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role May 23 12:41:10.475868 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) May 23 12:41:10.475868 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role May 23 12:41:10.475868 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() May 23 12:41:10.475868 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role May 23 12:41:10.475868 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) May 23 12:41:10.475868 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. May 23 12:41:10.475868 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application May 23 12:41:10.479139 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 226/434] 10.222.0.30 () {56 vars in 1099 bytes} [Mon May 23 12:41:10 2022] GET /identity/v3/roles/member => generated 84 bytes in 198 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) May 23 12:41:10.484817 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d55a8cf-0e72-492a-8231-fb3e73bc659e None None] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:41:10.598209 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=100 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:41:10.602841 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d55a8cf-0e72-492a-8231-fb3e73bc659e None None] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:41:10.637895 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc6f71a9-2718-480f-af93-a242b9299e5c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:41:10.638589 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc6f71a9-2718-480f-af93-a242b9299e5c None admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:41:10.638876 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc6f71a9-2718-480f-af93-a242b9299e5c None admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:41:10.639153 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc6f71a9-2718-480f-af93-a242b9299e5c None admin] PATH_INFO: `/v3/roles` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:41:10.639962 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc6f71a9-2718-480f-af93-a242b9299e5c None admin] RBAC: Adding query filter params (name=member) {{(pid=92057) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} May 23 12:41:10.640274 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc6f71a9-2718-480f-af93-a242b9299e5c None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:41:10.641766 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc6f71a9-2718-480f-af93-a242b9299e5c None admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:41:10.651585 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 209/435] 10.222.0.30 () {56 vars in 1101 bytes} [Mon May 23 12:41:10 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 167 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) May 23 12:41:10.657054 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85a89f3f-fe31-46b1-9be9-acac2cd844fd None admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:41:10.716116 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a96fb7d3-12be-4014-9203-6e772e88cb25 None None] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:41:10.716450 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a96fb7d3-12be-4014-9203-6e772e88cb25 None None] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:41:10.716689 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a96fb7d3-12be-4014-9203-6e772e88cb25 None None] PATH_INFO: `/` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:41:10.719615 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 210/436] 10.222.0.30 () {54 vars in 926 bytes} [Mon May 23 12:41:10 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) May 23 12:41:10.724095 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=100 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:41:10.724229 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96bc5bb6-f9d8-4328-b9da-0f1b58d89239 None None] REQUEST_METHOD: `POST` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:41:10.725199 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96bc5bb6-f9d8-4328-b9da-0f1b58d89239 None None] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:41:10.725491 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96bc5bb6-f9d8-4328-b9da-0f1b58d89239 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:41:10.745935 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.auth.core [None req-96bc5bb6-f9d8-4328-b9da-0f1b58d89239 None None] MFA Rules not processed for user `0f7119f8fa5c4445977671140ded91a4`. Rule list: `[]` (Enabled: `True`). {{(pid=92057) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 12:41:10.747102 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.fernet_utils [None req-96bc5bb6-f9d8-4328-b9da-0f1b58d89239 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92057) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 12:41:10.790876 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85a89f3f-fe31-46b1-9be9-acac2cd844fd None admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:41:10.828827 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 211/437] 10.222.0.30 () {58 vars in 1023 bytes} [Mon May 23 12:41:10 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 106 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) May 23 12:41:10.838089 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-833cff54-4d59-4e98-bb29-dae9ad9547c7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:41:10.838867 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-833cff54-4d59-4e98-bb29-dae9ad9547c7 None admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:41:10.839040 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-833cff54-4d59-4e98-bb29-dae9ad9547c7 None admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:41:10.839276 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-833cff54-4d59-4e98-bb29-dae9ad9547c7 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:41:10.852291 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96bc5bb6-f9d8-4328-b9da-0f1b58d89239 None None] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:41:10.915282 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-833cff54-4d59-4e98-bb29-dae9ad9547c7 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:41:10.916752 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-833cff54-4d59-4e98-bb29-dae9ad9547c7 None admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:41:10.960474 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=100 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:41:10.967948 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=100 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:41:10.970104 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96bc5bb6-f9d8-4328-b9da-0f1b58d89239 None None] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:41:11.004531 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08ebaadf-840e-47fa-9c88-9ea39c4e63d3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:41:11.005417 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08ebaadf-840e-47fa-9c88-9ea39c4e63d3 None admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:41:11.005767 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08ebaadf-840e-47fa-9c88-9ea39c4e63d3 None admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:41:11.006161 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08ebaadf-840e-47fa-9c88-9ea39c4e63d3 None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:41:11.017379 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08ebaadf-840e-47fa-9c88-9ea39c4e63d3 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} May 23 12:41:11.025673 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 227/438] 10.222.0.30 () {58 vars in 1282 bytes} [Mon May 23 12:41:10 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 365 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) May 23 12:41:11.025673 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab3c1306-72ba-4b1a-b80e-ba37fc08a10e None None] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:41:11.025673 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab3c1306-72ba-4b1a-b80e-ba37fc08a10e None None] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:41:11.025673 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab3c1306-72ba-4b1a-b80e-ba37fc08a10e None None] PATH_INFO: `/` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:41:11.025673 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 228/439] 10.222.0.30 () {54 vars in 926 bytes} [Mon May 23 12:41:11 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) May 23 12:41:11.026797 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application [None req-08ebaadf-840e-47fa-9c88-9ea39c4e63d3 None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. May 23 12:41:11.026797 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application Traceback (most recent call last): May 23 12:41:11.026797 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request May 23 12:41:11.026797 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application rv = self.dispatch_request() May 23 12:41:11.026797 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request May 23 12:41:11.026797 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) May 23 12:41:11.026797 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper May 23 12:41:11.026797 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) May 23 12:41:11.026797 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view May 23 12:41:11.026797 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) May 23 12:41:11.026797 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request May 23 12:41:11.026797 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) May 23 12:41:11.026797 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get May 23 12:41:11.026797 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self._get_role(role_id) May 23 12:41:11.026797 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role May 23 12:41:11.026797 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') May 23 12:41:11.028598 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call May 23 12:41:11.028598 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( May 23 12:41:11.028598 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data May 23 12:41:11.028598 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) May 23 12:41:11.028598 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped May 23 12:41:11.028598 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) May 23 12:41:11.028598 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun May 23 12:41:11.028598 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) May 23 12:41:11.028598 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func May 23 12:41:11.028598 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self.get_or_create( May 23 12:41:11.028598 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create May 23 12:41:11.028598 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application with Lock( May 23 12:41:11.028598 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ May 23 12:41:11.028598 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self._enter() May 23 12:41:11.028598 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter May 23 12:41:11.028598 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) May 23 12:41:11.029622 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create May 23 12:41:11.029622 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self.creator() May 23 12:41:11.029622 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value May 23 12:41:11.029622 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application created_value = creator( May 23 12:41:11.029622 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role May 23 12:41:11.029622 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) May 23 12:41:11.029622 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role May 23 12:41:11.029622 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() May 23 12:41:11.029622 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role May 23 12:41:11.029622 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) May 23 12:41:11.029622 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. May 23 12:41:11.029622 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application May 23 12:41:11.030271 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab3c1306-72ba-4b1a-b80e-ba37fc08a10e None None] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:41:11.030340 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 212/440] 10.222.0.30 () {56 vars in 1114 bytes} [Mon May 23 12:41:10 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 175 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) May 23 12:41:11.030677 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fc9ee29-2be4-4659-959f-9186f4107b69 None None] REQUEST_METHOD: `POST` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:41:11.030837 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fc9ee29-2be4-4659-959f-9186f4107b69 None None] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:41:11.031121 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fc9ee29-2be4-4659-959f-9186f4107b69 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:41:11.047826 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.auth.core [None req-8fc9ee29-2be4-4659-959f-9186f4107b69 None None] MFA Rules not processed for user `0f7119f8fa5c4445977671140ded91a4`. Rule list: `[]` (Enabled: `True`). {{(pid=92057) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 12:41:11.049001 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.fernet_utils [None req-8fc9ee29-2be4-4659-959f-9186f4107b69 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92057) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 12:41:11.116801 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 213/441] 10.222.0.30 () {58 vars in 1023 bytes} [Mon May 23 12:41:11 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 87 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) May 23 12:41:11.118221 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8fc9ee29-2be4-4659-959f-9186f4107b69 None None] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:41:11.129230 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab3c1306-72ba-4b1a-b80e-ba37fc08a10e None None] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:41:11.164943 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7afbc0ac-6dce-4765-a85b-e8a092ea1e39 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:41:11.165798 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7afbc0ac-6dce-4765-a85b-e8a092ea1e39 None admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:41:11.166103 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7afbc0ac-6dce-4765-a85b-e8a092ea1e39 None admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:41:11.166379 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7afbc0ac-6dce-4765-a85b-e8a092ea1e39 None admin] PATH_INFO: `/v3/projects/b39742b1d1b34dfe82a9ce4da39b7582` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:41:11.167655 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7afbc0ac-6dce-4765-a85b-e8a092ea1e39 None admin] RBAC: Authorizing `identity:get_project(project_id=b39742b1d1b34dfe82a9ce4da39b7582)` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:41:11.169196 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7afbc0ac-6dce-4765-a85b-e8a092ea1e39 None admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:41:11.170903 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 229/442] 10.222.0.30 () {56 vars in 1186 bytes} [Mon May 23 12:41:11 2022] GET /identity/v3/projects/b39742b1d1b34dfe82a9ce4da39b7582 => generated 302 bytes in 144 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) May 23 12:41:11.171653 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7afbc0ac-6dce-4765-a85b-e8a092ea1e39 None admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:41:11.189260 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=100 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:41:11.209805 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=100 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:41:11.228414 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8fc9ee29-2be4-4659-959f-9186f4107b69 None None] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:41:11.261570 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54ee2f19-0d34-4ba1-9f25-2b357725314d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:41:11.262270 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54ee2f19-0d34-4ba1-9f25-2b357725314d None admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:41:11.262552 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54ee2f19-0d34-4ba1-9f25-2b357725314d None admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:41:11.262827 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54ee2f19-0d34-4ba1-9f25-2b357725314d None admin] PATH_INFO: `/v3/roles` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:41:11.263566 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54ee2f19-0d34-4ba1-9f25-2b357725314d None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=92057) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} May 23 12:41:11.263872 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54ee2f19-0d34-4ba1-9f25-2b357725314d None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:41:11.265272 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54ee2f19-0d34-4ba1-9f25-2b357725314d None admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:41:11.273513 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 214/443] 10.222.0.30 () {56 vars in 1111 bytes} [Mon May 23 12:41:11 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 156 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) May 23 12:41:11.274352 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7afbc0ac-6dce-4765-a85b-e8a092ea1e39 None admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:41:11.274975 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54ee2f19-0d34-4ba1-9f25-2b357725314d None admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:41:11.309264 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98b765ce-34e8-4091-9ac9-d22e5d63ef18 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:41:11.310086 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98b765ce-34e8-4091-9ac9-d22e5d63ef18 None admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:41:11.310370 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98b765ce-34e8-4091-9ac9-d22e5d63ef18 None admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:41:11.310645 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98b765ce-34e8-4091-9ac9-d22e5d63ef18 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:41:11.321072 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-98b765ce-34e8-4091-9ac9-d22e5d63ef18 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} May 23 12:41:11.329301 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application [None req-98b765ce-34e8-4091-9ac9-d22e5d63ef18 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. May 23 12:41:11.329301 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application Traceback (most recent call last): May 23 12:41:11.329301 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request May 23 12:41:11.329301 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application rv = self.dispatch_request() May 23 12:41:11.329301 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request May 23 12:41:11.329301 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) May 23 12:41:11.329301 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper May 23 12:41:11.329301 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) May 23 12:41:11.329301 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view May 23 12:41:11.329301 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) May 23 12:41:11.329301 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request May 23 12:41:11.329301 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) May 23 12:41:11.329301 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get May 23 12:41:11.329301 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self._get_role(role_id) May 23 12:41:11.329301 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role May 23 12:41:11.329301 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') May 23 12:41:11.330411 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call May 23 12:41:11.330411 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( May 23 12:41:11.330411 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data May 23 12:41:11.330411 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) May 23 12:41:11.330411 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped May 23 12:41:11.330411 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) May 23 12:41:11.330411 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun May 23 12:41:11.330411 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) May 23 12:41:11.330411 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func May 23 12:41:11.330411 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self.get_or_create( May 23 12:41:11.330411 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create May 23 12:41:11.330411 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application with Lock( May 23 12:41:11.330411 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ May 23 12:41:11.330411 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self._enter() May 23 12:41:11.330411 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter May 23 12:41:11.330411 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) May 23 12:41:11.331473 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create May 23 12:41:11.331473 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self.creator() May 23 12:41:11.331473 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value May 23 12:41:11.331473 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application created_value = creator( May 23 12:41:11.331473 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role May 23 12:41:11.331473 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) May 23 12:41:11.331473 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role May 23 12:41:11.331473 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() May 23 12:41:11.331473 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role May 23 12:41:11.331473 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) May 23 12:41:11.331473 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. May 23 12:41:11.331473 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application May 23 12:41:11.331473 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 230/444] 10.222.0.30 () {56 vars in 1096 bytes} [Mon May 23 12:41:11 2022] GET /identity/v3/roles/admin => generated 83 bytes in 159 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) May 23 12:41:11.331473 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98b765ce-34e8-4091-9ac9-d22e5d63ef18 None admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:41:11.378783 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54ee2f19-0d34-4ba1-9f25-2b357725314d None admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:41:11.409748 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=100 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:41:11.412191 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8acb179e-ffc2-4b38-9f30-ded2fea46191 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:41:11.413041 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8acb179e-ffc2-4b38-9f30-ded2fea46191 None admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:41:11.413322 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8acb179e-ffc2-4b38-9f30-ded2fea46191 None admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:41:11.413596 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8acb179e-ffc2-4b38-9f30-ded2fea46191 None admin] PATH_INFO: `/v3/groups/b272e49032f04e138cd631884e9f8b93` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:41:11.415317 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8acb179e-ffc2-4b38-9f30-ded2fea46191 None admin] RBAC: Authorizing `identity:get_group(group_id=b272e49032f04e138cd631884e9f8b93)` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:41:11.416771 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8acb179e-ffc2-4b38-9f30-ded2fea46191 None admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:41:11.418408 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 215/445] 10.222.0.30 () {56 vars in 1180 bytes} [Mon May 23 12:41:11 2022] GET /identity/v3/groups/b272e49032f04e138cd631884e9f8b93 => generated 226 bytes in 144 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) May 23 12:41:11.419158 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8acb179e-ffc2-4b38-9f30-ded2fea46191 None admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:41:11.434335 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=100 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:41:11.436670 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98b765ce-34e8-4091-9ac9-d22e5d63ef18 None admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:41:11.470451 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f4d21fa-dc4e-4a55-a9c7-603057e497d5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:41:11.471086 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f4d21fa-dc4e-4a55-a9c7-603057e497d5 None admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:41:11.471368 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f4d21fa-dc4e-4a55-a9c7-603057e497d5 None admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:41:11.471639 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f4d21fa-dc4e-4a55-a9c7-603057e497d5 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:41:11.520404 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8acb179e-ffc2-4b38-9f30-ded2fea46191 None admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:41:11.538695 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f4d21fa-dc4e-4a55-a9c7-603057e497d5 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:41:11.540376 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f4d21fa-dc4e-4a55-a9c7-603057e497d5 None admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:41:11.552803 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fbd99473-3950-482a-857c-7f33b0e12b7f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:41:11.553487 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbd99473-3950-482a-857c-7f33b0e12b7f None admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:41:11.553766 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbd99473-3950-482a-857c-7f33b0e12b7f None admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:41:11.554073 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbd99473-3950-482a-857c-7f33b0e12b7f None admin] PATH_INFO: `/v3/roles` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:41:11.554809 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fbd99473-3950-482a-857c-7f33b0e12b7f None admin] RBAC: Adding query filter params (name=admin) {{(pid=92057) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} May 23 12:41:11.555714 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fbd99473-3950-482a-857c-7f33b0e12b7f None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:41:11.557431 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fbd99473-3950-482a-857c-7f33b0e12b7f None admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:41:11.565440 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 216/446] 10.222.0.30 () {56 vars in 1099 bytes} [Mon May 23 12:41:11 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 146 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) May 23 12:41:11.566074 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fbd99473-3950-482a-857c-7f33b0e12b7f None admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:41:11.613001 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=100 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:41:11.635323 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=100 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:41:11.638445 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 231/447] 10.222.0.30 () {58 vars in 1282 bytes} [Mon May 23 12:41:11 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 307 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) May 23 12:41:11.639270 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f4d21fa-dc4e-4a55-a9c7-603057e497d5 None admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:41:11.667173 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fbd99473-3950-482a-857c-7f33b0e12b7f None admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:41:11.701807 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98c985b4-0ffd-4e22-ac1a-211df209c768 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:41:11.702482 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98c985b4-0ffd-4e22-ac1a-211df209c768 None admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:41:11.702763 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98c985b4-0ffd-4e22-ac1a-211df209c768 None admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:41:11.703040 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98c985b4-0ffd-4e22-ac1a-211df209c768 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:41:11.703857 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-98c985b4-0ffd-4e22-ac1a-211df209c768 None admin] RBAC: Adding query filter params (group.id=b272e49032f04e138cd631884e9f8b93, role.id=80fb22507c9a4ff09d44d8f12f6fff1a, scope.project.id=b39742b1d1b34dfe82a9ce4da39b7582) {{(pid=92057) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} May 23 12:41:11.704165 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-98c985b4-0ffd-4e22-ac1a-211df209c768 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:41:11.705598 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-98c985b4-0ffd-4e22-ac1a-211df209c768 None admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:41:11.711307 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 217/448] 10.222.0.30 () {56 vars in 1376 bytes} [Mon May 23 12:41:11 2022] GET /identity/v3/role_assignments?group.id=b272e49032f04e138cd631884e9f8b93&scope.project.id=b39742b1d1b34dfe82a9ce4da39b7582&role.id=80fb22507c9a4ff09d44d8f12f6fff1a => generated 611 bytes in 145 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) May 23 12:41:11.721678 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98c985b4-0ffd-4e22-ac1a-211df209c768 None admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:41:11.751893 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f4d21fa-dc4e-4a55-a9c7-603057e497d5 None admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:41:11.790446 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-335a92e7-2e67-41dc-b678-8cb00dbff616 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:41:11.791046 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-335a92e7-2e67-41dc-b678-8cb00dbff616 None admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:41:11.791332 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-335a92e7-2e67-41dc-b678-8cb00dbff616 None admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:41:11.791615 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-335a92e7-2e67-41dc-b678-8cb00dbff616 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:41:11.829937 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98c985b4-0ffd-4e22-ac1a-211df209c768 None admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:41:11.842913 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=100 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:41:11.869300 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce6322a8-c7cc-4361-b923-e787e69d6753 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:41:11.870310 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce6322a8-c7cc-4361-b923-e787e69d6753 None admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:41:11.870660 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce6322a8-c7cc-4361-b923-e787e69d6753 None admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:41:11.871038 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce6322a8-c7cc-4361-b923-e787e69d6753 None admin] PATH_INFO: `/v3/projects/b39742b1d1b34dfe82a9ce4da39b7582` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:41:11.871473 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-335a92e7-2e67-41dc-b678-8cb00dbff616 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:41:11.872363 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce6322a8-c7cc-4361-b923-e787e69d6753 None admin] RBAC: Authorizing `identity:get_project(project_id=b39742b1d1b34dfe82a9ce4da39b7582)` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:41:11.872959 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-335a92e7-2e67-41dc-b678-8cb00dbff616 None admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:41:11.873823 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce6322a8-c7cc-4361-b923-e787e69d6753 None admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:41:11.875795 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 218/449] 10.222.0.30 () {56 vars in 1186 bytes} [Mon May 23 12:41:11 2022] GET /identity/v3/projects/b39742b1d1b34dfe82a9ce4da39b7582 => generated 302 bytes in 164 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) May 23 12:41:11.881137 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce6322a8-c7cc-4361-b923-e787e69d6753 None admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:41:11.886331 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=100 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:41:11.995813 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 232/450] 10.222.0.30 () {58 vars in 1282 bytes} [Mon May 23 12:41:11 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 357 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) May 23 12:41:12.000942 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce6322a8-c7cc-4361-b923-e787e69d6753 None admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:41:12.001636 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-335a92e7-2e67-41dc-b678-8cb00dbff616 None admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:41:12.042813 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb4718ae-0433-40ec-a7f3-22076bbe00f8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:41:12.043609 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb4718ae-0433-40ec-a7f3-22076bbe00f8 None admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:41:12.043908 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb4718ae-0433-40ec-a7f3-22076bbe00f8 None admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:41:12.044182 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb4718ae-0433-40ec-a7f3-22076bbe00f8 None admin] PATH_INFO: `/v3/groups/b272e49032f04e138cd631884e9f8b93` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:41:12.045397 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb4718ae-0433-40ec-a7f3-22076bbe00f8 None admin] RBAC: Authorizing `identity:get_group(group_id=b272e49032f04e138cd631884e9f8b93)` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:41:12.046973 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb4718ae-0433-40ec-a7f3-22076bbe00f8 None admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:41:12.053307 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 219/451] 10.222.0.30 () {56 vars in 1180 bytes} [Mon May 23 12:41:11 2022] GET /identity/v3/groups/b272e49032f04e138cd631884e9f8b93 => generated 226 bytes in 168 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) May 23 12:41:12.054326 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb4718ae-0433-40ec-a7f3-22076bbe00f8 None admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:41:12.089766 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=100 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:41:12.117694 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-335a92e7-2e67-41dc-b678-8cb00dbff616 None admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:41:12.137822 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=100 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:41:12.152810 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a926e6df-53ab-4228-b522-880b6ed89a9e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:41:12.153966 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a926e6df-53ab-4228-b522-880b6ed89a9e None admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:41:12.154247 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a926e6df-53ab-4228-b522-880b6ed89a9e None admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:41:12.154519 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a926e6df-53ab-4228-b522-880b6ed89a9e None admin] PATH_INFO: `/v3/projects/af8b84d1e3e44efba851eab6bed63282` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:41:12.155684 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a926e6df-53ab-4228-b522-880b6ed89a9e None admin] RBAC: Authorizing `identity:get_project(project_id=af8b84d1e3e44efba851eab6bed63282)` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:41:12.157118 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a926e6df-53ab-4228-b522-880b6ed89a9e None admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:41:12.158833 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 233/452] 10.222.0.30 () {56 vars in 1186 bytes} [Mon May 23 12:41:12 2022] GET /identity/v3/projects/af8b84d1e3e44efba851eab6bed63282 => generated 344 bytes in 158 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) May 23 12:41:12.164252 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a926e6df-53ab-4228-b522-880b6ed89a9e None admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:41:12.167698 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb4718ae-0433-40ec-a7f3-22076bbe00f8 None admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:41:12.201527 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c4d8128-99cd-4e18-a9a1-71762a75c046 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:41:12.202210 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c4d8128-99cd-4e18-a9a1-71762a75c046 None admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:41:12.202499 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c4d8128-99cd-4e18-a9a1-71762a75c046 None admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:41:12.202779 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c4d8128-99cd-4e18-a9a1-71762a75c046 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:41:12.203609 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c4d8128-99cd-4e18-a9a1-71762a75c046 None admin] RBAC: Adding query filter params (group.id=b272e49032f04e138cd631884e9f8b93, role.id=e3b90948645643e888620ab1e5c774bf, scope.project.id=b39742b1d1b34dfe82a9ce4da39b7582) {{(pid=92057) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} May 23 12:41:12.203925 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c4d8128-99cd-4e18-a9a1-71762a75c046 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:41:12.205457 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c4d8128-99cd-4e18-a9a1-71762a75c046 None admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:41:12.212165 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 220/453] 10.222.0.30 () {56 vars in 1376 bytes} [Mon May 23 12:41:12 2022] GET /identity/v3/role_assignments?group.id=b272e49032f04e138cd631884e9f8b93&scope.project.id=b39742b1d1b34dfe82a9ce4da39b7582&role.id=e3b90948645643e888620ab1e5c774bf => generated 611 bytes in 158 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) May 23 12:41:12.286406 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a926e6df-53ab-4228-b522-880b6ed89a9e None admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:41:12.322575 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-371a8130-da44-4d4f-bec1-1cdbf781f7eb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:41:12.323362 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-371a8130-da44-4d4f-bec1-1cdbf781f7eb None admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:41:12.323643 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-371a8130-da44-4d4f-bec1-1cdbf781f7eb None admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:41:12.323918 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-371a8130-da44-4d4f-bec1-1cdbf781f7eb None admin] PATH_INFO: `/v3/groups/33d7b939d9664f90b8df13bffb850ca4` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:41:12.325139 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-371a8130-da44-4d4f-bec1-1cdbf781f7eb None admin] RBAC: Authorizing `identity:get_group(group_id=33d7b939d9664f90b8df13bffb850ca4)` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:41:12.326672 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-371a8130-da44-4d4f-bec1-1cdbf781f7eb None admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:41:12.328331 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 234/454] 10.222.0.30 () {56 vars in 1180 bytes} [Mon May 23 12:41:12 2022] GET /identity/v3/groups/33d7b939d9664f90b8df13bffb850ca4 => generated 229 bytes in 164 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) May 23 12:41:12.334292 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c4d8128-99cd-4e18-a9a1-71762a75c046 None admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:41:12.437760 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c4d8128-99cd-4e18-a9a1-71762a75c046 None admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:41:12.472942 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3a2aebf-aa7d-498e-8b54-150ac5e6a2c3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:41:12.473605 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3a2aebf-aa7d-498e-8b54-150ac5e6a2c3 None admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:41:12.473912 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3a2aebf-aa7d-498e-8b54-150ac5e6a2c3 None admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:41:12.474198 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3a2aebf-aa7d-498e-8b54-150ac5e6a2c3 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:41:12.475034 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c3a2aebf-aa7d-498e-8b54-150ac5e6a2c3 None admin] RBAC: Adding query filter params (group.id=33d7b939d9664f90b8df13bffb850ca4, role.id=bc3c3c9d654940cbaa5177a6be23d1bc, scope.project.id=af8b84d1e3e44efba851eab6bed63282) {{(pid=92057) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} May 23 12:41:12.475352 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c3a2aebf-aa7d-498e-8b54-150ac5e6a2c3 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:41:12.476849 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c3a2aebf-aa7d-498e-8b54-150ac5e6a2c3 None admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:41:12.480262 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=100 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:41:12.483672 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 221/455] 10.222.0.30 () {56 vars in 1376 bytes} [Mon May 23 12:41:12 2022] GET /identity/v3/role_assignments?group.id=33d7b939d9664f90b8df13bffb850ca4&scope.project.id=af8b84d1e3e44efba851eab6bed63282&role.id=bc3c3c9d654940cbaa5177a6be23d1bc => generated 611 bytes in 150 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) May 23 12:41:15.130998 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3c0f721-3a7e-40d0-befd-b701c2af4bdd None None] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:41:15.131946 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3c0f721-3a7e-40d0-befd-b701c2af4bdd None None] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:41:15.132432 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3c0f721-3a7e-40d0-befd-b701c2af4bdd None None] PATH_INFO: `/` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:41:15.134977 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 235/456] 10.222.0.30 () {54 vars in 926 bytes} [Mon May 23 12:41:15 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) May 23 12:41:15.140076 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-265fcd3c-fa36-4025-9801-f9609141f87c None None] REQUEST_METHOD: `POST` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:41:15.140514 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-265fcd3c-fa36-4025-9801-f9609141f87c None None] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:41:15.140638 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-265fcd3c-fa36-4025-9801-f9609141f87c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:41:15.166914 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.auth.core [None req-265fcd3c-fa36-4025-9801-f9609141f87c None None] MFA Rules not processed for user `0f7119f8fa5c4445977671140ded91a4`. Rule list: `[]` (Enabled: `True`). {{(pid=92057) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 12:41:15.168342 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.fernet_utils [None req-265fcd3c-fa36-4025-9801-f9609141f87c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92057) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 12:41:15.268520 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 222/457] 10.222.0.30 () {58 vars in 1023 bytes} [Mon May 23 12:41:15 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 129 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) May 23 12:41:15.316633 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3c0f721-3a7e-40d0-befd-b701c2af4bdd None None] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:41:15.334055 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=100 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:41:15.423324 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ed2abb4-8d70-4c3b-90a5-4ea27a382fd2 None None] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:41:15.423826 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ed2abb4-8d70-4c3b-90a5-4ea27a382fd2 None None] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:41:15.424247 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ed2abb4-8d70-4c3b-90a5-4ea27a382fd2 None None] PATH_INFO: `/` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:41:15.426073 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 223/458] 10.222.0.30 () {54 vars in 926 bytes} [Mon May 23 12:41:15 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) May 23 12:41:15.432369 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8409c2e9-8efe-47b8-bab2-b618d8610790 None None] REQUEST_METHOD: `POST` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:41:15.432839 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8409c2e9-8efe-47b8-bab2-b618d8610790 None None] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:41:15.433257 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8409c2e9-8efe-47b8-bab2-b618d8610790 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:41:15.446159 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3c0f721-3a7e-40d0-befd-b701c2af4bdd None None] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:41:15.466157 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.auth.core [None req-8409c2e9-8efe-47b8-bab2-b618d8610790 None None] MFA Rules not processed for user `0f7119f8fa5c4445977671140ded91a4`. Rule list: `[]` (Enabled: `True`). {{(pid=92057) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 12:41:15.467715 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.fernet_utils [None req-8409c2e9-8efe-47b8-bab2-b618d8610790 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92057) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 12:41:15.482500 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b01cc5f-ed05-46a0-96c5-8b22901e79f5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:41:15.483513 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b01cc5f-ed05-46a0-96c5-8b22901e79f5 None admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:41:15.483929 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b01cc5f-ed05-46a0-96c5-8b22901e79f5 None admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:41:15.484360 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b01cc5f-ed05-46a0-96c5-8b22901e79f5 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:41:15.496038 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b01cc5f-ed05-46a0-96c5-8b22901e79f5 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:41:15.497661 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b01cc5f-ed05-46a0-96c5-8b22901e79f5 None admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:41:15.516924 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application [None req-9b01cc5f-ed05-46a0-96c5-8b22901e79f5 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. May 23 12:41:15.516924 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application Traceback (most recent call last): May 23 12:41:15.516924 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request May 23 12:41:15.516924 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application rv = self.dispatch_request() May 23 12:41:15.516924 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request May 23 12:41:15.516924 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) May 23 12:41:15.516924 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper May 23 12:41:15.516924 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) May 23 12:41:15.516924 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view May 23 12:41:15.516924 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) May 23 12:41:15.516924 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request May 23 12:41:15.516924 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) May 23 12:41:15.516924 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get May 23 12:41:15.516924 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self._get_domain(domain_id) May 23 12:41:15.516924 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain May 23 12:41:15.518057 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) May 23 12:41:15.518057 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped May 23 12:41:15.518057 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) May 23 12:41:15.518057 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun May 23 12:41:15.518057 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) May 23 12:41:15.518057 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func May 23 12:41:15.518057 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self.get_or_create( May 23 12:41:15.518057 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create May 23 12:41:15.518057 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application with Lock( May 23 12:41:15.518057 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ May 23 12:41:15.518057 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self._enter() May 23 12:41:15.518057 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter May 23 12:41:15.518057 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) May 23 12:41:15.518057 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create May 23 12:41:15.518057 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self.creator() May 23 12:41:15.518057 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value May 23 12:41:15.518057 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application created_value = creator( May 23 12:41:15.519121 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain May 23 12:41:15.519121 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) May 23 12:41:15.519121 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. May 23 12:41:15.519121 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application May 23 12:41:15.521053 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 236/459] 10.222.0.30 () {56 vars in 1108 bytes} [Mon May 23 12:41:15 2022] GET /identity/v3/domains/Default => generated 87 bytes in 205 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) May 23 12:41:15.529741 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b01cc5f-ed05-46a0-96c5-8b22901e79f5 None admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:41:15.618915 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 224/460] 10.222.0.30 () {58 vars in 1023 bytes} [Mon May 23 12:41:15 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 188 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) May 23 12:41:15.643555 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8409c2e9-8efe-47b8-bab2-b618d8610790 None None] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:41:15.663247 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=100 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:41:15.719629 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=100 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:41:15.759933 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8409c2e9-8efe-47b8-bab2-b618d8610790 None None] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:41:15.777690 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b01cc5f-ed05-46a0-96c5-8b22901e79f5 None admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:41:15.823486 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc147770-bc54-4973-b2db-4bc40bce7cce None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:41:15.824563 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc147770-bc54-4973-b2db-4bc40bce7cce None admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:41:15.825017 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc147770-bc54-4973-b2db-4bc40bce7cce None admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:41:15.825433 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc147770-bc54-4973-b2db-4bc40bce7cce None admin] PATH_INFO: `/v3/domains/Default` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:41:15.831673 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc58bc9a-0f74-49fc-854b-ee493e2df9ff None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:41:15.832343 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc58bc9a-0f74-49fc-854b-ee493e2df9ff None admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:41:15.832627 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc58bc9a-0f74-49fc-854b-ee493e2df9ff None admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:41:15.832902 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc58bc9a-0f74-49fc-854b-ee493e2df9ff None admin] PATH_INFO: `/v3/domains` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:41:15.834215 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc58bc9a-0f74-49fc-854b-ee493e2df9ff None admin] RBAC: Adding query filter params (name=Default) {{(pid=92056) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} May 23 12:41:15.834529 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc58bc9a-0f74-49fc-854b-ee493e2df9ff None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:41:15.837196 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc58bc9a-0f74-49fc-854b-ee493e2df9ff None admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:41:15.838895 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc147770-bc54-4973-b2db-4bc40bce7cce None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:41:15.840498 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc147770-bc54-4973-b2db-4bc40bce7cce None admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:41:15.852358 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 237/461] 10.222.0.30 () {56 vars in 1109 bytes} [Mon May 23 12:41:15 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 323 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) May 23 12:41:15.853478 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0b991c0-fca6-4063-bfc7-d607f4552495 None None] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:41:15.853771 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0b991c0-fca6-4063-bfc7-d607f4552495 None None] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:41:15.854075 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0b991c0-fca6-4063-bfc7-d607f4552495 None None] PATH_INFO: `/` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:41:15.859448 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 238/462] 10.222.0.30 () {54 vars in 926 bytes} [Mon May 23 12:41:15 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) May 23 12:41:15.859448 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0b991c0-fca6-4063-bfc7-d607f4552495 None None] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:41:15.865793 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application [None req-dc147770-bc54-4973-b2db-4bc40bce7cce None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. May 23 12:41:15.865793 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application Traceback (most recent call last): May 23 12:41:15.865793 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request May 23 12:41:15.865793 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application rv = self.dispatch_request() May 23 12:41:15.865793 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request May 23 12:41:15.865793 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) May 23 12:41:15.865793 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper May 23 12:41:15.865793 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) May 23 12:41:15.865793 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view May 23 12:41:15.865793 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) May 23 12:41:15.865793 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request May 23 12:41:15.865793 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) May 23 12:41:15.865793 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get May 23 12:41:15.865793 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self._get_domain(domain_id) May 23 12:41:15.865793 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain May 23 12:41:15.866946 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) May 23 12:41:15.866946 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped May 23 12:41:15.866946 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) May 23 12:41:15.866946 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun May 23 12:41:15.866946 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) May 23 12:41:15.866946 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func May 23 12:41:15.866946 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self.get_or_create( May 23 12:41:15.866946 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create May 23 12:41:15.866946 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application with Lock( May 23 12:41:15.866946 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ May 23 12:41:15.866946 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self._enter() May 23 12:41:15.866946 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter May 23 12:41:15.866946 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) May 23 12:41:15.866946 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create May 23 12:41:15.866946 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self.creator() May 23 12:41:15.866946 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value May 23 12:41:15.866946 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application created_value = creator( May 23 12:41:15.868663 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain May 23 12:41:15.868663 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) May 23 12:41:15.868663 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. May 23 12:41:15.868663 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application May 23 12:41:15.870518 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 225/463] 10.222.0.30 () {56 vars in 1108 bytes} [Mon May 23 12:41:15 2022] GET /identity/v3/domains/Default => generated 87 bytes in 227 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) May 23 12:41:15.872069 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b1cc824-433e-47b6-9b7f-07a9671be982 None None] REQUEST_METHOD: `POST` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:41:15.872498 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b1cc824-433e-47b6-9b7f-07a9671be982 None None] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:41:15.872915 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b1cc824-433e-47b6-9b7f-07a9671be982 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:41:15.891072 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.auth.core [None req-5b1cc824-433e-47b6-9b7f-07a9671be982 None None] MFA Rules not processed for user `0f7119f8fa5c4445977671140ded91a4`. Rule list: `[]` (Enabled: `True`). {{(pid=92057) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 12:41:15.892445 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.fernet_utils [None req-5b1cc824-433e-47b6-9b7f-07a9671be982 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92057) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 12:41:15.966964 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 226/464] 10.222.0.30 () {58 vars in 1023 bytes} [Mon May 23 12:41:15 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 96 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) May 23 12:41:15.968017 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b1cc824-433e-47b6-9b7f-07a9671be982 None None] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:41:15.983685 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0b991c0-fca6-4063-bfc7-d607f4552495 None None] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:41:16.009365 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=100 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:41:16.018396 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=100 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:41:16.020740 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a31cbcfa-c664-4e14-b6af-6b567266a1bb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:41:16.021403 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a31cbcfa-c664-4e14-b6af-6b567266a1bb None admin] REQUEST_METHOD: `POST` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:41:16.021693 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a31cbcfa-c664-4e14-b6af-6b567266a1bb None admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:41:16.022005 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a31cbcfa-c664-4e14-b6af-6b567266a1bb None admin] PATH_INFO: `/v3/users` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:41:16.022757 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a31cbcfa-c664-4e14-b6af-6b567266a1bb None admin] RBAC: Authorizing `identity:create_user()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:41:16.025322 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a31cbcfa-c664-4e14-b6af-6b567266a1bb None admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:41:16.075043 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b1cc824-433e-47b6-9b7f-07a9671be982 None None] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:41:16.108805 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b88bed80-e82f-4c13-bc95-2c785cbf4e89 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:41:16.109468 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b88bed80-e82f-4c13-bc95-2c785cbf4e89 None admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:41:16.109757 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b88bed80-e82f-4c13-bc95-2c785cbf4e89 None admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:41:16.110057 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b88bed80-e82f-4c13-bc95-2c785cbf4e89 None admin] PATH_INFO: `/v3/domains` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:41:16.110777 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b88bed80-e82f-4c13-bc95-2c785cbf4e89 None admin] RBAC: Adding query filter params (name=Default) {{(pid=92057) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} May 23 12:41:16.111088 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b88bed80-e82f-4c13-bc95-2c785cbf4e89 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:41:16.112518 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b88bed80-e82f-4c13-bc95-2c785cbf4e89 None admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:41:16.133912 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 227/465] 10.222.0.30 () {56 vars in 1109 bytes} [Mon May 23 12:41:15 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 156 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) May 23 12:41:16.134109 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 239/466] 10.222.0.30 () {60 vars in 1132 bytes} [Mon May 23 12:41:15 2022] POST /identity/v3/users => generated 246 bytes in 268 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) May 23 12:41:16.134524 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0797023-8090-4c79-901e-4dc660e7d075 None None] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:41:16.134524 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0797023-8090-4c79-901e-4dc660e7d075 None None] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:41:16.134524 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0797023-8090-4c79-901e-4dc660e7d075 None None] PATH_INFO: `/` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:41:16.134768 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a31cbcfa-c664-4e14-b6af-6b567266a1bb None admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:41:16.134854 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 228/467] 10.222.0.30 () {54 vars in 926 bytes} [Mon May 23 12:41:16 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) May 23 12:41:16.134854 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0797023-8090-4c79-901e-4dc660e7d075 None None] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:41:16.256404 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0797023-8090-4c79-901e-4dc660e7d075 None None] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:41:16.260168 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=100 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:41:16.284356 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a31cbcfa-c664-4e14-b6af-6b567266a1bb None admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:41:16.302665 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9fdf1517-afba-461b-9c97-fccd5c96ef4d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:41:16.303615 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fdf1517-afba-461b-9c97-fccd5c96ef4d None admin] REQUEST_METHOD: `POST` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:41:16.304196 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fdf1517-afba-461b-9c97-fccd5c96ef4d None admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:41:16.304657 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fdf1517-afba-461b-9c97-fccd5c96ef4d None admin] PATH_INFO: `/v3/users` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:41:16.305711 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9fdf1517-afba-461b-9c97-fccd5c96ef4d None admin] RBAC: Authorizing `identity:create_user()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:41:16.307742 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9fdf1517-afba-461b-9c97-fccd5c96ef4d None admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:41:16.331454 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82ea53e5-ad03-4ad5-890e-9bc1a7d9fdaa None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:41:16.332262 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82ea53e5-ad03-4ad5-890e-9bc1a7d9fdaa None admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:41:16.332543 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82ea53e5-ad03-4ad5-890e-9bc1a7d9fdaa None admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:41:16.332817 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82ea53e5-ad03-4ad5-890e-9bc1a7d9fdaa None admin] PATH_INFO: `/v3/domains/Default` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:41:16.345985 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82ea53e5-ad03-4ad5-890e-9bc1a7d9fdaa None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:41:16.347466 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82ea53e5-ad03-4ad5-890e-9bc1a7d9fdaa None admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:41:16.366864 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application [None req-82ea53e5-ad03-4ad5-890e-9bc1a7d9fdaa None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. May 23 12:41:16.366864 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application Traceback (most recent call last): May 23 12:41:16.366864 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request May 23 12:41:16.366864 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application rv = self.dispatch_request() May 23 12:41:16.366864 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request May 23 12:41:16.366864 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) May 23 12:41:16.366864 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper May 23 12:41:16.366864 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) May 23 12:41:16.366864 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view May 23 12:41:16.366864 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) May 23 12:41:16.366864 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request May 23 12:41:16.366864 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) May 23 12:41:16.366864 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get May 23 12:41:16.366864 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self._get_domain(domain_id) May 23 12:41:16.366864 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain May 23 12:41:16.367990 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) May 23 12:41:16.367990 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped May 23 12:41:16.367990 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) May 23 12:41:16.367990 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun May 23 12:41:16.367990 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) May 23 12:41:16.367990 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func May 23 12:41:16.367990 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self.get_or_create( May 23 12:41:16.367990 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create May 23 12:41:16.367990 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application with Lock( May 23 12:41:16.367990 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ May 23 12:41:16.367990 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self._enter() May 23 12:41:16.367990 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter May 23 12:41:16.367990 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) May 23 12:41:16.367990 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create May 23 12:41:16.367990 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self.creator() May 23 12:41:16.367990 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value May 23 12:41:16.367990 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application created_value = creator( May 23 12:41:16.368997 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain May 23 12:41:16.368997 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) May 23 12:41:16.368997 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. May 23 12:41:16.368997 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application May 23 12:41:16.368997 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 240/468] 10.222.0.30 () {56 vars in 1108 bytes} [Mon May 23 12:41:16 2022] GET /identity/v3/domains/Default => generated 87 bytes in 243 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) May 23 12:41:16.370982 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95be82ad-0896-4d5e-b9a8-c8fa88b2b338 None None] REQUEST_METHOD: `POST` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:41:16.370982 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95be82ad-0896-4d5e-b9a8-c8fa88b2b338 None None] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:41:16.370982 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95be82ad-0896-4d5e-b9a8-c8fa88b2b338 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:41:16.389870 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.auth.core [None req-95be82ad-0896-4d5e-b9a8-c8fa88b2b338 None None] MFA Rules not processed for user `0f7119f8fa5c4445977671140ded91a4`. Rule list: `[]` (Enabled: `True`). {{(pid=92056) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 12:41:16.390869 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 229/469] 10.222.0.30 () {60 vars in 1132 bytes} [Mon May 23 12:41:16 2022] POST /identity/v3/users => generated 248 bytes in 262 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) May 23 12:41:16.391242 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.fernet_utils [None req-95be82ad-0896-4d5e-b9a8-c8fa88b2b338 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92056) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 12:41:16.397902 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9fdf1517-afba-461b-9c97-fccd5c96ef4d None admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:41:16.424144 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:41:16.483475 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 241/470] 10.222.0.30 () {58 vars in 1023 bytes} [Mon May 23 12:41:16 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 114 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) May 23 12:41:16.512975 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95be82ad-0896-4d5e-b9a8-c8fa88b2b338 None None] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:41:16.553027 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9fdf1517-afba-461b-9c97-fccd5c96ef4d None admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:41:16.614809 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72373e1b-6b14-4622-83f7-beca63ac344b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:41:16.615553 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72373e1b-6b14-4622-83f7-beca63ac344b None admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:41:16.615845 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72373e1b-6b14-4622-83f7-beca63ac344b None admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:41:16.616126 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72373e1b-6b14-4622-83f7-beca63ac344b None admin] PATH_INFO: `/v3/domains` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:41:16.616862 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72373e1b-6b14-4622-83f7-beca63ac344b None admin] RBAC: Adding query filter params (name=Default) {{(pid=92057) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} May 23 12:41:16.617734 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72373e1b-6b14-4622-83f7-beca63ac344b None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:41:16.619973 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72373e1b-6b14-4622-83f7-beca63ac344b None admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:41:16.634877 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 230/471] 10.222.0.30 () {56 vars in 1109 bytes} [Mon May 23 12:41:16 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 243 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) May 23 12:41:16.641534 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72373e1b-6b14-4622-83f7-beca63ac344b None admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:41:16.667271 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:41:16.670302 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95be82ad-0896-4d5e-b9a8-c8fa88b2b338 None None] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:41:16.714412 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ce2d34e-023f-4dd9-913f-ff0181498cde None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:41:16.716332 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ce2d34e-023f-4dd9-913f-ff0181498cde None admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:41:16.716848 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ce2d34e-023f-4dd9-913f-ff0181498cde None admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:41:16.717358 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ce2d34e-023f-4dd9-913f-ff0181498cde None admin] PATH_INFO: `/v3/domains/Default` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:41:16.729352 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ce2d34e-023f-4dd9-913f-ff0181498cde None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:41:16.731141 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ce2d34e-023f-4dd9-913f-ff0181498cde None admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:41:16.751194 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application [None req-3ce2d34e-023f-4dd9-913f-ff0181498cde None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. May 23 12:41:16.751194 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application Traceback (most recent call last): May 23 12:41:16.751194 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request May 23 12:41:16.751194 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application rv = self.dispatch_request() May 23 12:41:16.751194 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request May 23 12:41:16.751194 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) May 23 12:41:16.751194 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper May 23 12:41:16.751194 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) May 23 12:41:16.751194 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view May 23 12:41:16.751194 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) May 23 12:41:16.751194 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request May 23 12:41:16.751194 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) May 23 12:41:16.751194 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get May 23 12:41:16.751194 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self._get_domain(domain_id) May 23 12:41:16.751194 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain May 23 12:41:16.752181 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) May 23 12:41:16.752181 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped May 23 12:41:16.752181 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) May 23 12:41:16.752181 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun May 23 12:41:16.752181 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) May 23 12:41:16.752181 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func May 23 12:41:16.752181 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self.get_or_create( May 23 12:41:16.752181 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create May 23 12:41:16.752181 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application with Lock( May 23 12:41:16.752181 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ May 23 12:41:16.752181 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self._enter() May 23 12:41:16.752181 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter May 23 12:41:16.752181 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) May 23 12:41:16.752181 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create May 23 12:41:16.752181 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self.creator() May 23 12:41:16.752181 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value May 23 12:41:16.752181 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application created_value = creator( May 23 12:41:16.753099 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain May 23 12:41:16.753099 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) May 23 12:41:16.753099 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. May 23 12:41:16.753099 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application May 23 12:41:16.753317 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 242/472] 10.222.0.30 () {56 vars in 1108 bytes} [Mon May 23 12:41:16 2022] GET /identity/v3/domains/Default => generated 87 bytes in 241 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) May 23 12:41:16.754623 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ff1b5cf-50a1-41ea-adcb-4304bb7ab1a7 None None] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:41:16.754909 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ff1b5cf-50a1-41ea-adcb-4304bb7ab1a7 None None] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:41:16.755189 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ff1b5cf-50a1-41ea-adcb-4304bb7ab1a7 None None] PATH_INFO: `/` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:41:16.756700 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 243/473] 10.222.0.30 () {54 vars in 926 bytes} [Mon May 23 12:41:16 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) May 23 12:41:16.763135 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ff1b5cf-50a1-41ea-adcb-4304bb7ab1a7 None None] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:41:16.781273 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72373e1b-6b14-4622-83f7-beca63ac344b None admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:41:16.782230 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=100 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:41:16.823545 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-508e2f52-e98b-435d-b075-9af02059c09e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:41:16.825070 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-508e2f52-e98b-435d-b075-9af02059c09e None admin] REQUEST_METHOD: `POST` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:41:16.825532 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-508e2f52-e98b-435d-b075-9af02059c09e None admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:41:16.826000 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-508e2f52-e98b-435d-b075-9af02059c09e None admin] PATH_INFO: `/v3/users` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:41:16.826850 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-508e2f52-e98b-435d-b075-9af02059c09e None admin] RBAC: Authorizing `identity:create_user()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:41:16.829033 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-508e2f52-e98b-435d-b075-9af02059c09e None admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:41:16.882100 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ff1b5cf-50a1-41ea-adcb-4304bb7ab1a7 None None] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:41:16.912790 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 231/474] 10.222.0.30 () {60 vars in 1132 bytes} [Mon May 23 12:41:16 2022] POST /identity/v3/users => generated 248 bytes in 272 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) May 23 12:41:16.914968 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c9c3364-7c43-426a-886e-fc8193158ab6 None None] REQUEST_METHOD: `POST` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:41:16.915443 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c9c3364-7c43-426a-886e-fc8193158ab6 None None] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:41:16.915923 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c9c3364-7c43-426a-886e-fc8193158ab6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:41:16.935685 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2eb4724c-7385-45d9-a2a7-e16509b97e70 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:41:16.935685 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2eb4724c-7385-45d9-a2a7-e16509b97e70 None admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:41:16.935685 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2eb4724c-7385-45d9-a2a7-e16509b97e70 None admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:41:16.935685 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2eb4724c-7385-45d9-a2a7-e16509b97e70 None admin] PATH_INFO: `/v3/domains` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:41:16.936424 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2eb4724c-7385-45d9-a2a7-e16509b97e70 None admin] RBAC: Adding query filter params (name=Default) {{(pid=92056) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} May 23 12:41:16.936786 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2eb4724c-7385-45d9-a2a7-e16509b97e70 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:41:16.938216 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2eb4724c-7385-45d9-a2a7-e16509b97e70 None admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:41:16.939220 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.auth.core [None req-1c9c3364-7c43-426a-886e-fc8193158ab6 None None] MFA Rules not processed for user `0f7119f8fa5c4445977671140ded91a4`. Rule list: `[]` (Enabled: `True`). {{(pid=92057) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 12:41:16.941604 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.fernet_utils [None req-1c9c3364-7c43-426a-886e-fc8193158ab6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92057) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 12:41:16.953632 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 244/475] 10.222.0.30 () {56 vars in 1109 bytes} [Mon May 23 12:41:16 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 191 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) May 23 12:41:16.959054 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2eb4724c-7385-45d9-a2a7-e16509b97e70 None admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:41:17.098656 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=100 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:41:17.127738 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 232/476] 10.222.0.30 () {58 vars in 1023 bytes} [Mon May 23 12:41:16 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 214 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) May 23 12:41:17.158516 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2eb4724c-7385-45d9-a2a7-e16509b97e70 None admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:41:17.166954 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c9c3364-7c43-426a-886e-fc8193158ab6 None None] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:41:17.172187 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:41:17.221348 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e188033a-816f-45b6-b23e-2ea54ddb1681 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:41:17.222086 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e188033a-816f-45b6-b23e-2ea54ddb1681 None admin] REQUEST_METHOD: `POST` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:41:17.222407 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e188033a-816f-45b6-b23e-2ea54ddb1681 None admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:41:17.222685 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e188033a-816f-45b6-b23e-2ea54ddb1681 None admin] PATH_INFO: `/v3/users` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:41:17.223554 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e188033a-816f-45b6-b23e-2ea54ddb1681 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:41:17.225143 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e188033a-816f-45b6-b23e-2ea54ddb1681 None admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:41:17.331669 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c9c3364-7c43-426a-886e-fc8193158ab6 None None] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:41:17.370022 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-302b5738-ff4a-40c7-aca6-34be12c92811 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:41:17.370701 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-302b5738-ff4a-40c7-aca6-34be12c92811 None admin] REQUEST_METHOD: `POST` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:41:17.370987 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-302b5738-ff4a-40c7-aca6-34be12c92811 None admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:41:17.371267 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-302b5738-ff4a-40c7-aca6-34be12c92811 None admin] PATH_INFO: `/v3/roles` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:41:17.372026 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-302b5738-ff4a-40c7-aca6-34be12c92811 None admin] RBAC: Authorizing `identity:create_role()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:41:17.373415 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-302b5738-ff4a-40c7-aca6-34be12c92811 None admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:41:17.395521 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 245/477] 10.222.0.30 () {60 vars in 1132 bytes} [Mon May 23 12:41:16 2022] POST /identity/v3/users => generated 249 bytes in 437 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) May 23 12:41:17.412406 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.sql.core [None req-302b5738-ff4a-40c7-aca6-34be12c92811 None admin] Conflict role: (pymysql.err.IntegrityError) (1062, "Duplicate entry 'anotherrole-<>' for key 'role.ixu_role_name_domain_id'") May 23 12:41:17.412406 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [SQL: INSERT INTO `role` (id, name, description, extra) VALUES (%(id)s, %(name)s, %(description)s, %(extra)s)] May 23 12:41:17.412406 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [parameters: {'id': 'a9621f42cecd42d4afd9436cff7502a6', 'name': 'anotherrole', 'description': None, 'extra': '{}'}] May 23 12:41:17.412406 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: (Background on this error at: https://sqlalche.me/e/14/gkpj) {{(pid=92057) wrapper /opt/stack/keystone/keystone/common/sql/core.py:533}} May 23 12:41:17.433744 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application [None req-302b5738-ff4a-40c7-aca6-34be12c92811 None admin] Conflict occurred attempting to store role - Duplicate entry found with name anotherrole.: keystone.exception.Conflict: Conflict occurred attempting to store role - Duplicate entry found with name anotherrole. May 23 12:41:17.433744 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application Traceback (most recent call last): May 23 12:41:17.433744 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1819, in _execute_context May 23 12:41:17.433744 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application self.dialect.do_execute( May 23 12:41:17.433744 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/default.py", line 732, in do_execute May 23 12:41:17.433744 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application cursor.execute(statement, parameters) May 23 12:41:17.433744 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 148, in execute May 23 12:41:17.433744 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application result = self._query(query) May 23 12:41:17.433744 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 310, in _query May 23 12:41:17.433744 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application conn.query(q) May 23 12:41:17.433744 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 548, in query May 23 12:41:17.433744 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application self._affected_rows = self._read_query_result(unbuffered=unbuffered) May 23 12:41:17.433744 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 775, in _read_query_result May 23 12:41:17.433744 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application result.read() May 23 12:41:17.433744 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 1156, in read May 23 12:41:17.434734 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application first_packet = self.connection._read_packet() May 23 12:41:17.434734 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 725, in _read_packet May 23 12:41:17.434734 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application packet.raise_for_error() May 23 12:41:17.434734 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/protocol.py", line 221, in raise_for_error May 23 12:41:17.434734 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application err.raise_mysql_exception(self._data) May 23 12:41:17.434734 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/err.py", line 143, in raise_mysql_exception May 23 12:41:17.434734 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application raise errorclass(errno, errval) May 23 12:41:17.434734 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application pymysql.err.IntegrityError: (1062, "Duplicate entry 'anotherrole-<>' for key 'role.ixu_role_name_domain_id'") May 23 12:41:17.434734 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application May 23 12:41:17.434734 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application The above exception was the direct cause of the following exception: May 23 12:41:17.434734 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application May 23 12:41:17.434734 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application Traceback (most recent call last): May 23 12:41:17.434734 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/sql/core.py", line 528, in wrapper May 23 12:41:17.434734 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return method(*args, **kwargs) May 23 12:41:17.434734 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 33, in create_role May 23 12:41:17.434734 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return ref.to_dict() May 23 12:41:17.434734 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ May 23 12:41:17.434734 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application next(self.gen) May 23 12:41:17.435679 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 1060, in _transaction_scope May 23 12:41:17.435679 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application yield resource May 23 12:41:17.435679 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ May 23 12:41:17.435679 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application next(self.gen) May 23 12:41:17.435679 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 662, in _session May 23 12:41:17.435679 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application self.session.rollback() May 23 12:41:17.435679 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ May 23 12:41:17.435679 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application self.force_reraise() May 23 12:41:17.435679 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise May 23 12:41:17.435679 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application raise self.value May 23 12:41:17.435679 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 659, in _session May 23 12:41:17.435679 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application self._end_session_transaction(self.session) May 23 12:41:17.435679 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 687, in _end_session_transaction May 23 12:41:17.435679 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application session.commit() May 23 12:41:17.435679 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 1435, in commit May 23 12:41:17.435679 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application self._transaction.commit(_to_root=self.future) May 23 12:41:17.435679 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 829, in commit May 23 12:41:17.436570 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application self._prepare_impl() May 23 12:41:17.436570 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 808, in _prepare_impl May 23 12:41:17.436570 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application self.session.flush() May 23 12:41:17.436570 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3367, in flush May 23 12:41:17.436570 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application self._flush(objects) May 23 12:41:17.436570 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3507, in _flush May 23 12:41:17.436570 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application transaction.rollback(_capture_exception=True) May 23 12:41:17.436570 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/langhelpers.py", line 70, in __exit__ May 23 12:41:17.436570 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application compat.raise_( May 23 12:41:17.436570 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/compat.py", line 207, in raise_ May 23 12:41:17.436570 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application raise exception May 23 12:41:17.436570 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3467, in _flush May 23 12:41:17.436570 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application flush_context.execute() May 23 12:41:17.436570 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/unitofwork.py", line 456, in execute May 23 12:41:17.436570 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application rec.execute(self) May 23 12:41:17.436570 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/unitofwork.py", line 630, in execute May 23 12:41:17.436570 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application util.preloaded.orm_persistence.save_obj( May 23 12:41:17.438237 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/persistence.py", line 245, in save_obj May 23 12:41:17.438237 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application _emit_insert_statements( May 23 12:41:17.438237 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/persistence.py", line 1097, in _emit_insert_statements May 23 12:41:17.438237 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application c = connection._execute_20( May 23 12:41:17.438237 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1631, in _execute_20 May 23 12:41:17.438237 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return meth(self, args_10style, kwargs_10style, execution_options) May 23 12:41:17.438237 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/sql/elements.py", line 325, in _execute_on_connection May 23 12:41:17.438237 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return connection._execute_clauseelement( May 23 12:41:17.438237 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1498, in _execute_clauseelement May 23 12:41:17.438237 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application ret = self._execute_context( May 23 12:41:17.438237 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1862, in _execute_context May 23 12:41:17.438237 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application self._handle_dbapi_exception( May 23 12:41:17.438237 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 2041, in _handle_dbapi_exception May 23 12:41:17.438237 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application util.raise_(newraise, with_traceback=exc_info[2], from_=e) May 23 12:41:17.438237 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/compat.py", line 207, in raise_ May 23 12:41:17.438237 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application raise exception May 23 12:41:17.439099 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1819, in _execute_context May 23 12:41:17.439099 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application self.dialect.do_execute( May 23 12:41:17.439099 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/default.py", line 732, in do_execute May 23 12:41:17.439099 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application cursor.execute(statement, parameters) May 23 12:41:17.439099 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 148, in execute May 23 12:41:17.439099 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application result = self._query(query) May 23 12:41:17.439099 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 310, in _query May 23 12:41:17.439099 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application conn.query(q) May 23 12:41:17.439099 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 548, in query May 23 12:41:17.439099 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application self._affected_rows = self._read_query_result(unbuffered=unbuffered) May 23 12:41:17.439099 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 775, in _read_query_result May 23 12:41:17.439099 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application result.read() May 23 12:41:17.439099 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 1156, in read May 23 12:41:17.439099 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application first_packet = self.connection._read_packet() May 23 12:41:17.439099 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 725, in _read_packet May 23 12:41:17.439099 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application packet.raise_for_error() May 23 12:41:17.439915 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/protocol.py", line 221, in raise_for_error May 23 12:41:17.439915 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application err.raise_mysql_exception(self._data) May 23 12:41:17.439915 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/err.py", line 143, in raise_mysql_exception May 23 12:41:17.439915 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application raise errorclass(errno, errval) May 23 12:41:17.439915 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application oslo_db.exception.DBDuplicateEntry: (pymysql.err.IntegrityError) (1062, "Duplicate entry 'anotherrole-<>' for key 'role.ixu_role_name_domain_id'") May 23 12:41:17.439915 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application [SQL: INSERT INTO `role` (id, name, description, extra) VALUES (%(id)s, %(name)s, %(description)s, %(extra)s)] May 23 12:41:17.439915 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application [parameters: {'id': 'a9621f42cecd42d4afd9436cff7502a6', 'name': 'anotherrole', 'description': None, 'extra': '{}'}] May 23 12:41:17.439915 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application (Background on this error at: https://sqlalche.me/e/14/gkpj) May 23 12:41:17.439915 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application May 23 12:41:17.439915 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: May 23 12:41:17.439915 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application May 23 12:41:17.439915 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application Traceback (most recent call last): May 23 12:41:17.439915 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request May 23 12:41:17.439915 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application rv = self.dispatch_request() May 23 12:41:17.439915 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request May 23 12:41:17.439915 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) May 23 12:41:17.440745 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper May 23 12:41:17.440745 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) May 23 12:41:17.440745 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view May 23 12:41:17.440745 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) May 23 12:41:17.440745 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request May 23 12:41:17.440745 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) May 23 12:41:17.440745 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 115, in post May 23 12:41:17.440745 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application ref = PROVIDERS.role_api.create_role( May 23 12:41:17.440745 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped May 23 12:41:17.440745 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) May 23 12:41:17.440745 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1298, in create_role May 23 12:41:17.440745 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application ret = self.driver.create_role(role_id, role) May 23 12:41:17.440745 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/sql/core.py", line 563, in wrapper May 23 12:41:17.440745 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application raise exception.Conflict(type=conflict_type, May 23 12:41:17.440745 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application keystone.exception.Conflict: Conflict occurred attempting to store role - Duplicate entry found with name anotherrole. May 23 12:41:17.440745 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application May 23 12:41:17.441607 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 233/478] 10.222.0.30 () {60 vars in 1131 bytes} [Mon May 23 12:41:17 2022] POST /identity/v3/roles => generated 144 bytes in 269 msecs (HTTP/1.1 409) 5 headers in 183 bytes (1 switches on core 0) May 23 12:41:17.442527 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e188033a-816f-45b6-b23e-2ea54ddb1681 None admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:41:17.623751 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e188033a-816f-45b6-b23e-2ea54ddb1681 None admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:41:17.671882 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:41:17.687772 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38eb5d8f-b8d5-4c05-9460-0c7420957d95 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:41:17.688604 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38eb5d8f-b8d5-4c05-9460-0c7420957d95 None admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:41:17.688893 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38eb5d8f-b8d5-4c05-9460-0c7420957d95 None admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:41:17.693985 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38eb5d8f-b8d5-4c05-9460-0c7420957d95 None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:41:17.705928 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38eb5d8f-b8d5-4c05-9460-0c7420957d95 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} May 23 12:41:17.715421 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application [None req-38eb5d8f-b8d5-4c05-9460-0c7420957d95 None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. May 23 12:41:17.715421 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application Traceback (most recent call last): May 23 12:41:17.715421 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request May 23 12:41:17.715421 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application rv = self.dispatch_request() May 23 12:41:17.715421 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request May 23 12:41:17.715421 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) May 23 12:41:17.715421 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper May 23 12:41:17.715421 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) May 23 12:41:17.715421 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view May 23 12:41:17.715421 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) May 23 12:41:17.715421 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request May 23 12:41:17.715421 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) May 23 12:41:17.715421 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get May 23 12:41:17.715421 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self._get_role(role_id) May 23 12:41:17.715421 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role May 23 12:41:17.715421 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') May 23 12:41:17.716437 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call May 23 12:41:17.716437 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( May 23 12:41:17.716437 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data May 23 12:41:17.716437 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) May 23 12:41:17.716437 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped May 23 12:41:17.716437 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) May 23 12:41:17.716437 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun May 23 12:41:17.716437 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) May 23 12:41:17.716437 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func May 23 12:41:17.716437 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self.get_or_create( May 23 12:41:17.716437 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create May 23 12:41:17.716437 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application with Lock( May 23 12:41:17.716437 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ May 23 12:41:17.716437 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self._enter() May 23 12:41:17.716437 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter May 23 12:41:17.716437 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) May 23 12:41:17.717258 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create May 23 12:41:17.717258 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self.creator() May 23 12:41:17.717258 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value May 23 12:41:17.717258 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application created_value = creator( May 23 12:41:17.717258 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role May 23 12:41:17.717258 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) May 23 12:41:17.717258 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role May 23 12:41:17.717258 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() May 23 12:41:17.717258 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role May 23 12:41:17.717258 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) May 23 12:41:17.717258 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. May 23 12:41:17.717258 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application May 23 12:41:17.717258 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 246/479] 10.222.0.30 () {56 vars in 1114 bytes} [Mon May 23 12:41:17 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 276 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) May 23 12:41:17.723821 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-302b5738-ff4a-40c7-aca6-34be12c92811 None admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:41:17.801564 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:41:17.839415 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-302b5738-ff4a-40c7-aca6-34be12c92811 None admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:41:17.875908 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19ba4328-bf22-485d-b828-f3c76adef3de None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:41:17.876513 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19ba4328-bf22-485d-b828-f3c76adef3de None admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:41:17.876802 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19ba4328-bf22-485d-b828-f3c76adef3de None admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:41:17.877082 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19ba4328-bf22-485d-b828-f3c76adef3de None admin] PATH_INFO: `/v3/roles` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:41:17.877832 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19ba4328-bf22-485d-b828-f3c76adef3de None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=92057) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} May 23 12:41:17.878186 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19ba4328-bf22-485d-b828-f3c76adef3de None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:41:17.879590 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19ba4328-bf22-485d-b828-f3c76adef3de None admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:41:17.889476 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 234/480] 10.222.0.30 () {56 vars in 1111 bytes} [Mon May 23 12:41:17 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 166 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) May 23 12:41:19.105729 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00380fca-b818-4dd2-9061-df808021dba8 None None] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:41:19.106238 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00380fca-b818-4dd2-9061-df808021dba8 None None] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:41:19.106238 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00380fca-b818-4dd2-9061-df808021dba8 None None] PATH_INFO: `/` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:41:19.108158 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 247/481] 10.222.0.30 () {54 vars in 926 bytes} [Mon May 23 12:41:19 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) May 23 12:41:19.113713 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-801ed90f-60f1-4b10-ab8c-54adb3c43788 None None] REQUEST_METHOD: `POST` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:41:19.114131 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-801ed90f-60f1-4b10-ab8c-54adb3c43788 None None] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:41:19.114534 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-801ed90f-60f1-4b10-ab8c-54adb3c43788 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:41:19.120979 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7313c4bb-b401-4443-a51f-100596152d7c None None] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:41:19.121264 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7313c4bb-b401-4443-a51f-100596152d7c None None] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:41:19.121533 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7313c4bb-b401-4443-a51f-100596152d7c None None] PATH_INFO: `/` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:41:19.123159 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 248/482] 10.222.0.30 () {54 vars in 926 bytes} [Mon May 23 12:41:19 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) May 23 12:41:19.127708 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f45f1a0-365d-48d0-a0ff-dc1bab83766a None None] REQUEST_METHOD: `POST` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:41:19.128000 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f45f1a0-365d-48d0-a0ff-dc1bab83766a None None] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:41:19.128279 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f45f1a0-365d-48d0-a0ff-dc1bab83766a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:41:19.150599 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.auth.core [None req-801ed90f-60f1-4b10-ab8c-54adb3c43788 None None] MFA Rules not processed for user `0f7119f8fa5c4445977671140ded91a4`. Rule list: `[]` (Enabled: `True`). {{(pid=92057) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 12:41:19.151791 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.fernet_utils [None req-801ed90f-60f1-4b10-ab8c-54adb3c43788 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92057) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 12:41:19.157769 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.auth.core [None req-5f45f1a0-365d-48d0-a0ff-dc1bab83766a None None] MFA Rules not processed for user `0f7119f8fa5c4445977671140ded91a4`. Rule list: `[]` (Enabled: `True`). {{(pid=92056) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 12:41:19.158956 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.fernet_utils [None req-5f45f1a0-365d-48d0-a0ff-dc1bab83766a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92056) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 12:41:19.256223 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 249/483] 10.222.0.30 () {58 vars in 1023 bytes} [Mon May 23 12:41:19 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 129 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) May 23 12:41:19.266888 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 235/484] 10.222.0.30 () {58 vars in 1023 bytes} [Mon May 23 12:41:19 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 154 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) May 23 12:41:19.283366 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f45f1a0-365d-48d0-a0ff-dc1bab83766a None None] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:41:19.294630 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-801ed90f-60f1-4b10-ab8c-54adb3c43788 None None] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:41:19.357139 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=100 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:41:19.399439 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f45f1a0-365d-48d0-a0ff-dc1bab83766a None None] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:41:19.400364 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-801ed90f-60f1-4b10-ab8c-54adb3c43788 None None] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:41:19.419825 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=100 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:41:19.444910 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a03c870-a8d8-49df-9c8f-c160f5356bf5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:41:19.445990 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a03c870-a8d8-49df-9c8f-c160f5356bf5 None admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:41:19.446285 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a03c870-a8d8-49df-9c8f-c160f5356bf5 None admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:41:19.446564 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a03c870-a8d8-49df-9c8f-c160f5356bf5 None admin] PATH_INFO: `/v3/roles/service` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:41:19.450074 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-691725d2-73bc-4d7e-be93-dfa604d8c4d1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:41:19.450893 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-691725d2-73bc-4d7e-be93-dfa604d8c4d1 None admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:41:19.451167 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-691725d2-73bc-4d7e-be93-dfa604d8c4d1 None admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:41:19.451435 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-691725d2-73bc-4d7e-be93-dfa604d8c4d1 None admin] PATH_INFO: `/v3/roles/service` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:41:19.472039 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-691725d2-73bc-4d7e-be93-dfa604d8c4d1 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} May 23 12:41:19.474616 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a03c870-a8d8-49df-9c8f-c160f5356bf5 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} May 23 12:41:19.482973 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application [None req-7a03c870-a8d8-49df-9c8f-c160f5356bf5 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. May 23 12:41:19.482973 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application Traceback (most recent call last): May 23 12:41:19.482973 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request May 23 12:41:19.482973 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application rv = self.dispatch_request() May 23 12:41:19.482973 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request May 23 12:41:19.482973 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) May 23 12:41:19.482973 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper May 23 12:41:19.482973 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) May 23 12:41:19.482973 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view May 23 12:41:19.482973 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) May 23 12:41:19.482973 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request May 23 12:41:19.482973 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) May 23 12:41:19.482973 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get May 23 12:41:19.482973 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self._get_role(role_id) May 23 12:41:19.482973 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role May 23 12:41:19.482973 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') May 23 12:41:19.483886 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call May 23 12:41:19.483886 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( May 23 12:41:19.483886 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data May 23 12:41:19.483886 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) May 23 12:41:19.483886 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped May 23 12:41:19.483886 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) May 23 12:41:19.483886 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun May 23 12:41:19.483886 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) May 23 12:41:19.483886 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func May 23 12:41:19.483886 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self.get_or_create( May 23 12:41:19.483886 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create May 23 12:41:19.483886 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application with Lock( May 23 12:41:19.483886 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ May 23 12:41:19.483886 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self._enter() May 23 12:41:19.483886 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter May 23 12:41:19.483886 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) May 23 12:41:19.485114 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create May 23 12:41:19.485114 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self.creator() May 23 12:41:19.485114 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value May 23 12:41:19.485114 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application created_value = creator( May 23 12:41:19.485114 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role May 23 12:41:19.485114 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) May 23 12:41:19.485114 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role May 23 12:41:19.485114 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() May 23 12:41:19.485114 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role May 23 12:41:19.485114 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) May 23 12:41:19.485114 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. May 23 12:41:19.485114 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application May 23 12:41:19.485114 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 236/485] 10.222.0.30 () {56 vars in 1102 bytes} [Mon May 23 12:41:19 2022] GET /identity/v3/roles/service => generated 85 bytes in 190 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) May 23 12:41:19.490087 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application [None req-691725d2-73bc-4d7e-be93-dfa604d8c4d1 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. May 23 12:41:19.490087 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application Traceback (most recent call last): May 23 12:41:19.490087 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request May 23 12:41:19.490087 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application rv = self.dispatch_request() May 23 12:41:19.490087 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request May 23 12:41:19.490087 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) May 23 12:41:19.490087 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper May 23 12:41:19.490087 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) May 23 12:41:19.490087 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view May 23 12:41:19.490087 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) May 23 12:41:19.490087 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request May 23 12:41:19.490087 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) May 23 12:41:19.490087 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get May 23 12:41:19.490087 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self._get_role(role_id) May 23 12:41:19.490087 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role May 23 12:41:19.490087 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') May 23 12:41:19.490989 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call May 23 12:41:19.490989 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( May 23 12:41:19.490989 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data May 23 12:41:19.490989 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) May 23 12:41:19.490989 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped May 23 12:41:19.490989 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) May 23 12:41:19.490989 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun May 23 12:41:19.490989 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) May 23 12:41:19.490989 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func May 23 12:41:19.490989 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self.get_or_create( May 23 12:41:19.490989 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create May 23 12:41:19.490989 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application with Lock( May 23 12:41:19.490989 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ May 23 12:41:19.490989 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self._enter() May 23 12:41:19.490989 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter May 23 12:41:19.490989 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) May 23 12:41:19.493183 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create May 23 12:41:19.493183 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self.creator() May 23 12:41:19.493183 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value May 23 12:41:19.493183 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application created_value = creator( May 23 12:41:19.493183 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role May 23 12:41:19.493183 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) May 23 12:41:19.493183 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role May 23 12:41:19.493183 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() May 23 12:41:19.493183 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role May 23 12:41:19.493183 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) May 23 12:41:19.493183 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. May 23 12:41:19.493183 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application May 23 12:41:19.493183 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 250/486] 10.222.0.30 () {56 vars in 1102 bytes} [Mon May 23 12:41:19 2022] GET /identity/v3/roles/service => generated 85 bytes in 209 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) May 23 12:41:19.498149 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a03c870-a8d8-49df-9c8f-c160f5356bf5 None admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:41:19.499043 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-691725d2-73bc-4d7e-be93-dfa604d8c4d1 None admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:41:19.613684 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a03c870-a8d8-49df-9c8f-c160f5356bf5 None admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:41:19.635376 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=100 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:41:19.654656 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd95bc97-68fe-41d0-8b0b-d31ef1375bba None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:41:19.655314 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd95bc97-68fe-41d0-8b0b-d31ef1375bba None admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:41:19.655595 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd95bc97-68fe-41d0-8b0b-d31ef1375bba None admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:41:19.655873 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd95bc97-68fe-41d0-8b0b-d31ef1375bba None admin] PATH_INFO: `/v3/roles` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:41:19.656614 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd95bc97-68fe-41d0-8b0b-d31ef1375bba None admin] RBAC: Adding query filter params (name=service) {{(pid=92057) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} May 23 12:41:19.656917 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd95bc97-68fe-41d0-8b0b-d31ef1375bba None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:41:19.658338 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd95bc97-68fe-41d0-8b0b-d31ef1375bba None admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:41:19.675993 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 237/487] 10.222.0.30 () {56 vars in 1103 bytes} [Mon May 23 12:41:19 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 178 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) May 23 12:41:19.683442 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd95bc97-68fe-41d0-8b0b-d31ef1375bba None admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:41:19.716972 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-691725d2-73bc-4d7e-be93-dfa604d8c4d1 None admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:41:19.790657 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76bdfcdb-2b5f-476d-b40e-0432794247f1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:41:19.791327 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76bdfcdb-2b5f-476d-b40e-0432794247f1 None admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:41:19.791686 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76bdfcdb-2b5f-476d-b40e-0432794247f1 None admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:41:19.792005 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76bdfcdb-2b5f-476d-b40e-0432794247f1 None admin] PATH_INFO: `/v3/roles` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:41:19.792803 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76bdfcdb-2b5f-476d-b40e-0432794247f1 None admin] RBAC: Adding query filter params (name=service) {{(pid=92056) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} May 23 12:41:19.793117 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76bdfcdb-2b5f-476d-b40e-0432794247f1 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:41:19.794544 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76bdfcdb-2b5f-476d-b40e-0432794247f1 None admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:41:19.808475 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 251/488] 10.222.0.30 () {56 vars in 1103 bytes} [Mon May 23 12:41:19 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 309 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) May 23 12:41:19.813373 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76bdfcdb-2b5f-476d-b40e-0432794247f1 None admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:41:19.876521 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd95bc97-68fe-41d0-8b0b-d31ef1375bba None admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:41:19.891476 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=100 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:41:19.942696 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc6128bb-b7a6-4c99-9fd0-363fde40e9ac None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:41:19.943518 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc6128bb-b7a6-4c99-9fd0-363fde40e9ac None admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:41:19.943807 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc6128bb-b7a6-4c99-9fd0-363fde40e9ac None admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:41:19.944085 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc6128bb-b7a6-4c99-9fd0-363fde40e9ac None admin] PATH_INFO: `/v3/domains/Default` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:41:19.969598 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc6128bb-b7a6-4c99-9fd0-363fde40e9ac None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:41:19.971350 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc6128bb-b7a6-4c99-9fd0-363fde40e9ac None admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:41:20.005433 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application [None req-fc6128bb-b7a6-4c99-9fd0-363fde40e9ac None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. May 23 12:41:20.005433 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application Traceback (most recent call last): May 23 12:41:20.005433 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request May 23 12:41:20.005433 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application rv = self.dispatch_request() May 23 12:41:20.005433 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request May 23 12:41:20.005433 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) May 23 12:41:20.005433 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper May 23 12:41:20.005433 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) May 23 12:41:20.005433 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view May 23 12:41:20.005433 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) May 23 12:41:20.005433 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request May 23 12:41:20.005433 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) May 23 12:41:20.005433 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get May 23 12:41:20.005433 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self._get_domain(domain_id) May 23 12:41:20.005433 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain May 23 12:41:20.006582 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) May 23 12:41:20.006582 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped May 23 12:41:20.006582 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) May 23 12:41:20.006582 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun May 23 12:41:20.006582 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) May 23 12:41:20.006582 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func May 23 12:41:20.006582 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self.get_or_create( May 23 12:41:20.006582 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create May 23 12:41:20.006582 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application with Lock( May 23 12:41:20.006582 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ May 23 12:41:20.006582 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self._enter() May 23 12:41:20.006582 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter May 23 12:41:20.006582 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) May 23 12:41:20.006582 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create May 23 12:41:20.006582 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self.creator() May 23 12:41:20.006582 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value May 23 12:41:20.006582 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application created_value = creator( May 23 12:41:20.008293 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain May 23 12:41:20.008293 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) May 23 12:41:20.008293 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. May 23 12:41:20.008293 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application May 23 12:41:20.008606 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 238/489] 10.222.0.30 () {56 vars in 1108 bytes} [Mon May 23 12:41:19 2022] GET /identity/v3/domains/Default => generated 87 bytes in 325 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) May 23 12:41:20.014992 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76bdfcdb-2b5f-476d-b40e-0432794247f1 None admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:41:20.017255 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc6128bb-b7a6-4c99-9fd0-363fde40e9ac None admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:41:20.080437 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa8f88b3-6650-4f3b-bef4-9ad434736afd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:41:20.080970 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=100 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:41:20.081236 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa8f88b3-6650-4f3b-bef4-9ad434736afd None admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:41:20.081513 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa8f88b3-6650-4f3b-bef4-9ad434736afd None admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:41:20.081784 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa8f88b3-6650-4f3b-bef4-9ad434736afd None admin] PATH_INFO: `/v3/domains/Default` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:41:20.105570 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa8f88b3-6650-4f3b-bef4-9ad434736afd None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:41:20.108137 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa8f88b3-6650-4f3b-bef4-9ad434736afd None admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:41:20.138976 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application [None req-aa8f88b3-6650-4f3b-bef4-9ad434736afd None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. May 23 12:41:20.138976 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application Traceback (most recent call last): May 23 12:41:20.138976 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request May 23 12:41:20.138976 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application rv = self.dispatch_request() May 23 12:41:20.138976 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request May 23 12:41:20.138976 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) May 23 12:41:20.138976 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper May 23 12:41:20.138976 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) May 23 12:41:20.138976 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view May 23 12:41:20.138976 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) May 23 12:41:20.138976 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request May 23 12:41:20.138976 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) May 23 12:41:20.138976 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get May 23 12:41:20.138976 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self._get_domain(domain_id) May 23 12:41:20.138976 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain May 23 12:41:20.139900 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) May 23 12:41:20.139900 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped May 23 12:41:20.139900 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) May 23 12:41:20.139900 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun May 23 12:41:20.139900 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) May 23 12:41:20.139900 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func May 23 12:41:20.139900 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self.get_or_create( May 23 12:41:20.139900 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create May 23 12:41:20.139900 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application with Lock( May 23 12:41:20.139900 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ May 23 12:41:20.139900 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self._enter() May 23 12:41:20.139900 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter May 23 12:41:20.139900 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) May 23 12:41:20.139900 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create May 23 12:41:20.139900 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self.creator() May 23 12:41:20.139900 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value May 23 12:41:20.139900 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application created_value = creator( May 23 12:41:20.140785 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain May 23 12:41:20.140785 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) May 23 12:41:20.140785 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. May 23 12:41:20.140785 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application May 23 12:41:20.147265 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 252/490] 10.222.0.30 () {56 vars in 1108 bytes} [Mon May 23 12:41:19 2022] GET /identity/v3/domains/Default => generated 87 bytes in 334 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) May 23 12:41:20.149732 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ef3f313-bb45-4a76-bfcf-591459d6ab73 None None] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:41:20.150122 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ef3f313-bb45-4a76-bfcf-591459d6ab73 None None] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:41:20.150583 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ef3f313-bb45-4a76-bfcf-591459d6ab73 None None] PATH_INFO: `/` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:41:20.152277 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 253/491] 10.222.0.30 () {54 vars in 926 bytes} [Mon May 23 12:41:20 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) May 23 12:41:20.153573 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ef3f313-bb45-4a76-bfcf-591459d6ab73 None None] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:41:20.189918 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc6128bb-b7a6-4c99-9fd0-363fde40e9ac None admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:41:20.227526 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa0d2c67-beb6-4411-97af-0f8acf47be04 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:41:20.228556 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa0d2c67-beb6-4411-97af-0f8acf47be04 None admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:41:20.229144 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa0d2c67-beb6-4411-97af-0f8acf47be04 None admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:41:20.229631 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa0d2c67-beb6-4411-97af-0f8acf47be04 None admin] PATH_INFO: `/v3/domains` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:41:20.230616 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa0d2c67-beb6-4411-97af-0f8acf47be04 None admin] RBAC: Adding query filter params (name=Default) {{(pid=92057) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} May 23 12:41:20.231166 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa0d2c67-beb6-4411-97af-0f8acf47be04 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:41:20.232732 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa0d2c67-beb6-4411-97af-0f8acf47be04 None admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:41:20.246729 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 239/492] 10.222.0.30 () {56 vars in 1109 bytes} [Mon May 23 12:41:20 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 230 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) May 23 12:41:20.248383 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b8598a2-0bb8-4b22-bb39-3173c9b0ee72 None None] REQUEST_METHOD: `POST` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:41:20.248869 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b8598a2-0bb8-4b22-bb39-3173c9b0ee72 None None] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:41:20.249334 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b8598a2-0bb8-4b22-bb39-3173c9b0ee72 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:41:20.268370 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.auth.core [None req-9b8598a2-0bb8-4b22-bb39-3173c9b0ee72 None None] MFA Rules not processed for user `0f7119f8fa5c4445977671140ded91a4`. Rule list: `[]` (Enabled: `True`). {{(pid=92057) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 12:41:20.269922 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.fernet_utils [None req-9b8598a2-0bb8-4b22-bb39-3173c9b0ee72 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92057) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 12:41:20.270674 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=100 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:41:20.282247 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ef3f313-bb45-4a76-bfcf-591459d6ab73 None None] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:41:20.319735 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc354cda-9ea2-46ab-9d52-4c1b31fe3b0a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:41:20.321000 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc354cda-9ea2-46ab-9d52-4c1b31fe3b0a None admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:41:20.321273 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc354cda-9ea2-46ab-9d52-4c1b31fe3b0a None admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:41:20.321561 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc354cda-9ea2-46ab-9d52-4c1b31fe3b0a None admin] PATH_INFO: `/v3/domains` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:41:20.322305 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc354cda-9ea2-46ab-9d52-4c1b31fe3b0a None admin] RBAC: Adding query filter params (name=Default) {{(pid=92056) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} May 23 12:41:20.322617 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc354cda-9ea2-46ab-9d52-4c1b31fe3b0a None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:41:20.324006 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc354cda-9ea2-46ab-9d52-4c1b31fe3b0a None admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:41:20.337108 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 254/493] 10.222.0.30 () {56 vars in 1109 bytes} [Mon May 23 12:41:20 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 184 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) May 23 12:41:20.337995 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc354cda-9ea2-46ab-9d52-4c1b31fe3b0a None admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:41:20.348272 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 240/494] 10.222.0.30 () {58 vars in 1023 bytes} [Mon May 23 12:41:20 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 100 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) May 23 12:41:20.349428 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b8598a2-0bb8-4b22-bb39-3173c9b0ee72 None None] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:41:20.377692 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=100 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:41:20.453104 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc354cda-9ea2-46ab-9d52-4c1b31fe3b0a None admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:41:20.455618 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b8598a2-0bb8-4b22-bb39-3173c9b0ee72 None None] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:41:20.487736 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-550958f0-5cbe-47ef-85f6-00e27985d7dc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:41:20.488523 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-550958f0-5cbe-47ef-85f6-00e27985d7dc None admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:41:20.488801 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-550958f0-5cbe-47ef-85f6-00e27985d7dc None admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:41:20.489072 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-550958f0-5cbe-47ef-85f6-00e27985d7dc None admin] PATH_INFO: `/v3/users/glance` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:41:20.491259 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5325e02a-e54e-411d-8a13-244db375cdc5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:41:20.492661 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5325e02a-e54e-411d-8a13-244db375cdc5 None admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:41:20.493030 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5325e02a-e54e-411d-8a13-244db375cdc5 None admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:41:20.493348 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5325e02a-e54e-411d-8a13-244db375cdc5 None admin] PATH_INFO: `/v3/users/nova` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:41:20.497512 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-550958f0-5cbe-47ef-85f6-00e27985d7dc None admin] RBAC: Authorizing `identity:get_user(user_id=glance)` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:41:20.498984 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-550958f0-5cbe-47ef-85f6-00e27985d7dc None admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:41:20.500557 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5325e02a-e54e-411d-8a13-244db375cdc5 None admin] RBAC: Authorizing `identity:get_user(user_id=nova)` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:41:20.502045 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5325e02a-e54e-411d-8a13-244db375cdc5 None admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:41:20.514862 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application [None req-550958f0-5cbe-47ef-85f6-00e27985d7dc None admin] Could not find user: glance.: keystone.exception.UserNotFound: Could not find user: glance. May 23 12:41:20.514862 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application Traceback (most recent call last): May 23 12:41:20.514862 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request May 23 12:41:20.514862 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application rv = self.dispatch_request() May 23 12:41:20.514862 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request May 23 12:41:20.514862 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) May 23 12:41:20.514862 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper May 23 12:41:20.514862 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) May 23 12:41:20.514862 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view May 23 12:41:20.514862 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) May 23 12:41:20.514862 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request May 23 12:41:20.514862 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) May 23 12:41:20.514862 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get May 23 12:41:20.514862 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self._get_user(user_id) May 23 12:41:20.514862 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user May 23 12:41:20.514862 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) May 23 12:41:20.515893 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped May 23 12:41:20.515893 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) May 23 12:41:20.515893 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper May 23 12:41:20.515893 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) May 23 12:41:20.515893 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper May 23 12:41:20.515893 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) May 23 12:41:20.515893 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun May 23 12:41:20.515893 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) May 23 12:41:20.515893 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func May 23 12:41:20.515893 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self.get_or_create( May 23 12:41:20.515893 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create May 23 12:41:20.515893 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application with Lock( May 23 12:41:20.515893 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ May 23 12:41:20.515893 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self._enter() May 23 12:41:20.515893 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter May 23 12:41:20.515893 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) May 23 12:41:20.515893 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create May 23 12:41:20.515893 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self.creator() May 23 12:41:20.521971 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value May 23 12:41:20.521971 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application created_value = creator( May 23 12:41:20.521971 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user May 23 12:41:20.521971 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) May 23 12:41:20.521971 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user May 23 12:41:20.521971 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) May 23 12:41:20.521971 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user May 23 12:41:20.521971 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) May 23 12:41:20.521971 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: glance. May 23 12:41:20.521971 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application May 23 12:41:20.521971 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 255/495] 10.222.0.30 () {56 vars in 1099 bytes} [Mon May 23 12:41:20 2022] GET /identity/v3/users/glance => generated 84 bytes in 178 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) May 23 12:41:20.521971 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1a37d15-08eb-429d-be51-6b25a7bbe729 None None] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:41:20.521971 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1a37d15-08eb-429d-be51-6b25a7bbe729 None None] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:41:20.522670 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1a37d15-08eb-429d-be51-6b25a7bbe729 None None] PATH_INFO: `/` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:41:20.522737 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application [None req-5325e02a-e54e-411d-8a13-244db375cdc5 None admin] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova. May 23 12:41:20.522737 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application Traceback (most recent call last): May 23 12:41:20.522737 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request May 23 12:41:20.522737 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application rv = self.dispatch_request() May 23 12:41:20.522737 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request May 23 12:41:20.522737 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) May 23 12:41:20.522737 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper May 23 12:41:20.522737 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) May 23 12:41:20.522737 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view May 23 12:41:20.522737 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) May 23 12:41:20.522737 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request May 23 12:41:20.522737 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) May 23 12:41:20.522737 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get May 23 12:41:20.522737 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self._get_user(user_id) May 23 12:41:20.522737 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user May 23 12:41:20.522737 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) May 23 12:41:20.525417 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped May 23 12:41:20.525417 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) May 23 12:41:20.525417 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper May 23 12:41:20.525417 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) May 23 12:41:20.525417 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper May 23 12:41:20.525417 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) May 23 12:41:20.525417 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun May 23 12:41:20.525417 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) May 23 12:41:20.525417 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func May 23 12:41:20.525417 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self.get_or_create( May 23 12:41:20.525417 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create May 23 12:41:20.525417 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application with Lock( May 23 12:41:20.525417 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ May 23 12:41:20.525417 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self._enter() May 23 12:41:20.525417 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter May 23 12:41:20.525417 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) May 23 12:41:20.525417 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create May 23 12:41:20.525417 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self.creator() May 23 12:41:20.526360 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value May 23 12:41:20.526360 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application created_value = creator( May 23 12:41:20.526360 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user May 23 12:41:20.526360 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) May 23 12:41:20.526360 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user May 23 12:41:20.526360 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) May 23 12:41:20.526360 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user May 23 12:41:20.526360 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) May 23 12:41:20.526360 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: nova. May 23 12:41:20.526360 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application May 23 12:41:20.526891 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 256/496] 10.222.0.30 () {54 vars in 926 bytes} [Mon May 23 12:41:20 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) May 23 12:41:20.526891 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1a37d15-08eb-429d-be51-6b25a7bbe729 None None] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:41:20.527014 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 241/497] 10.222.0.30 () {56 vars in 1093 bytes} [Mon May 23 12:41:20 2022] GET /identity/v3/users/nova => generated 82 bytes in 173 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) May 23 12:41:20.527014 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5325e02a-e54e-411d-8a13-244db375cdc5 None admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:41:20.543785 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=100 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:41:20.620209 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=100 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:41:20.635395 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1a37d15-08eb-429d-be51-6b25a7bbe729 None None] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:41:20.639070 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5325e02a-e54e-411d-8a13-244db375cdc5 None admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:41:20.671666 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2b304ba-dfbe-45e2-9db4-a07ee02beca1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:41:20.672455 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2b304ba-dfbe-45e2-9db4-a07ee02beca1 None admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:41:20.672734 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2b304ba-dfbe-45e2-9db4-a07ee02beca1 None admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:41:20.673003 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2b304ba-dfbe-45e2-9db4-a07ee02beca1 None admin] PATH_INFO: `/v3/roles/service` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:41:20.675256 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9c9e896-830d-4aaa-99f4-8d7d212bc2a8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:41:20.675925 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9c9e896-830d-4aaa-99f4-8d7d212bc2a8 None admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:41:20.676208 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9c9e896-830d-4aaa-99f4-8d7d212bc2a8 None admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:41:20.676484 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9c9e896-830d-4aaa-99f4-8d7d212bc2a8 None admin] PATH_INFO: `/v3/users` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:41:20.677344 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9c9e896-830d-4aaa-99f4-8d7d212bc2a8 None admin] RBAC: Adding query filter params (domain_id=default, name=glance) {{(pid=92057) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} May 23 12:41:20.677656 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9c9e896-830d-4aaa-99f4-8d7d212bc2a8 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:41:20.680247 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9c9e896-830d-4aaa-99f4-8d7d212bc2a8 None admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:41:20.687492 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2b304ba-dfbe-45e2-9db4-a07ee02beca1 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} May 23 12:41:20.697107 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application [None req-b2b304ba-dfbe-45e2-9db4-a07ee02beca1 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. May 23 12:41:20.697107 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application Traceback (most recent call last): May 23 12:41:20.697107 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request May 23 12:41:20.697107 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application rv = self.dispatch_request() May 23 12:41:20.697107 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request May 23 12:41:20.697107 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) May 23 12:41:20.697107 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper May 23 12:41:20.697107 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) May 23 12:41:20.697107 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view May 23 12:41:20.697107 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) May 23 12:41:20.697107 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request May 23 12:41:20.697107 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) May 23 12:41:20.697107 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get May 23 12:41:20.697107 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self._get_role(role_id) May 23 12:41:20.697107 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role May 23 12:41:20.697107 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') May 23 12:41:20.698067 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call May 23 12:41:20.698067 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( May 23 12:41:20.698067 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data May 23 12:41:20.698067 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) May 23 12:41:20.698067 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped May 23 12:41:20.698067 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) May 23 12:41:20.698067 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun May 23 12:41:20.698067 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) May 23 12:41:20.698067 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func May 23 12:41:20.698067 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self.get_or_create( May 23 12:41:20.698067 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create May 23 12:41:20.698067 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application with Lock( May 23 12:41:20.698067 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ May 23 12:41:20.698067 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self._enter() May 23 12:41:20.698067 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter May 23 12:41:20.698067 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) May 23 12:41:20.698893 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create May 23 12:41:20.698893 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self.creator() May 23 12:41:20.698893 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value May 23 12:41:20.698893 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application created_value = creator( May 23 12:41:20.698893 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role May 23 12:41:20.698893 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) May 23 12:41:20.698893 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role May 23 12:41:20.698893 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() May 23 12:41:20.698893 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role May 23 12:41:20.698893 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) May 23 12:41:20.698893 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. May 23 12:41:20.698893 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application May 23 12:41:20.698893 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 257/498] 10.222.0.30 () {56 vars in 1102 bytes} [Mon May 23 12:41:20 2022] GET /identity/v3/roles/service => generated 85 bytes in 178 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) May 23 12:41:20.700056 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aee83a8f-1677-4218-8df6-8386c9020693 None None] REQUEST_METHOD: `POST` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:41:20.700056 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aee83a8f-1677-4218-8df6-8386c9020693 None None] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:41:20.700219 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aee83a8f-1677-4218-8df6-8386c9020693 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:41:20.717252 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.auth.core [None req-aee83a8f-1677-4218-8df6-8386c9020693 None None] MFA Rules not processed for user `0f7119f8fa5c4445977671140ded91a4`. Rule list: `[]` (Enabled: `True`). {{(pid=92056) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 12:41:20.718395 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.fernet_utils [None req-aee83a8f-1677-4218-8df6-8386c9020693 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92056) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 12:41:20.725244 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 242/499] 10.222.0.30 () {56 vars in 1137 bytes} [Mon May 23 12:41:20 2022] GET /identity/v3/users?domain_id=default&name=glance => generated 372 bytes in 202 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) May 23 12:41:20.726157 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9c9e896-830d-4aaa-99f4-8d7d212bc2a8 None admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:41:20.784770 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 258/500] 10.222.0.30 () {58 vars in 1023 bytes} [Mon May 23 12:41:20 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) May 23 12:41:20.785814 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aee83a8f-1677-4218-8df6-8386c9020693 None None] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:41:20.788177 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=100 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:41:20.835208 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9c9e896-830d-4aaa-99f4-8d7d212bc2a8 None admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:41:20.869427 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab349b20-b15a-4423-84ca-e87bc3a1892e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:41:20.870122 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab349b20-b15a-4423-84ca-e87bc3a1892e None admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:41:20.870409 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab349b20-b15a-4423-84ca-e87bc3a1892e None admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:41:20.870693 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab349b20-b15a-4423-84ca-e87bc3a1892e None admin] PATH_INFO: `/v3/users` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:41:20.871515 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab349b20-b15a-4423-84ca-e87bc3a1892e None admin] RBAC: Adding query filter params (domain_id=default, name=nova) {{(pid=92057) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} May 23 12:41:20.871825 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab349b20-b15a-4423-84ca-e87bc3a1892e None admin] RBAC: Authorizing `identity:list_users()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:41:20.873292 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab349b20-b15a-4423-84ca-e87bc3a1892e None admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:41:20.885346 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aee83a8f-1677-4218-8df6-8386c9020693 None None] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:41:20.886925 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=100 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:41:20.890508 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 243/501] 10.222.0.30 () {56 vars in 1133 bytes} [Mon May 23 12:41:20 2022] GET /identity/v3/users?domain_id=default&name=nova => generated 368 bytes in 165 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) May 23 12:41:20.891736 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2b14642-33cb-468f-8640-47572e913899 None None] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:41:20.892016 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2b14642-33cb-468f-8640-47572e913899 None None] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:41:20.892358 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2b14642-33cb-468f-8640-47572e913899 None None] PATH_INFO: `/` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:41:20.893744 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 244/502] 10.222.0.30 () {54 vars in 926 bytes} [Mon May 23 12:41:20 2022] GET /identity => generated 270 bytes in 2 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) May 23 12:41:20.894272 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2b14642-33cb-468f-8640-47572e913899 None None] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:41:20.924964 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0d4df7c-f0c9-4885-b047-8e524b7e3bb6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:41:20.925621 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0d4df7c-f0c9-4885-b047-8e524b7e3bb6 None admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:41:20.925926 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0d4df7c-f0c9-4885-b047-8e524b7e3bb6 None admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:41:20.926205 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0d4df7c-f0c9-4885-b047-8e524b7e3bb6 None admin] PATH_INFO: `/v3/roles` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:41:20.926938 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0d4df7c-f0c9-4885-b047-8e524b7e3bb6 None admin] RBAC: Adding query filter params (name=service) {{(pid=92056) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} May 23 12:41:20.927245 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0d4df7c-f0c9-4885-b047-8e524b7e3bb6 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:41:20.928619 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0d4df7c-f0c9-4885-b047-8e524b7e3bb6 None admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:41:20.936979 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 259/503] 10.222.0.30 () {56 vars in 1103 bytes} [Mon May 23 12:41:20 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 151 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) May 23 12:41:20.937827 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0d4df7c-f0c9-4885-b047-8e524b7e3bb6 None admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:41:20.997524 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=100 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:41:21.000375 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2b14642-33cb-468f-8640-47572e913899 None None] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:41:21.035123 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3fcc8aea-7dc8-4109-8f7c-77296240a189 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:41:21.035764 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fcc8aea-7dc8-4109-8f7c-77296240a189 None admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:41:21.036049 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fcc8aea-7dc8-4109-8f7c-77296240a189 None admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:41:21.036325 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fcc8aea-7dc8-4109-8f7c-77296240a189 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:41:21.037049 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0d4df7c-f0c9-4885-b047-8e524b7e3bb6 None admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:41:21.069624 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-773ef5d4-a980-4bb5-8102-c2e64ca96c1b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:41:21.071125 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-773ef5d4-a980-4bb5-8102-c2e64ca96c1b None admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:41:21.071431 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-773ef5d4-a980-4bb5-8102-c2e64ca96c1b None admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:41:21.071708 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-773ef5d4-a980-4bb5-8102-c2e64ca96c1b None admin] PATH_INFO: `/v3/roles/service` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:41:21.081781 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-773ef5d4-a980-4bb5-8102-c2e64ca96c1b None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} May 23 12:41:21.090566 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application [None req-773ef5d4-a980-4bb5-8102-c2e64ca96c1b None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. May 23 12:41:21.090566 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application Traceback (most recent call last): May 23 12:41:21.090566 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request May 23 12:41:21.090566 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application rv = self.dispatch_request() May 23 12:41:21.090566 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request May 23 12:41:21.090566 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) May 23 12:41:21.090566 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper May 23 12:41:21.090566 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) May 23 12:41:21.090566 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view May 23 12:41:21.090566 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) May 23 12:41:21.090566 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request May 23 12:41:21.090566 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) May 23 12:41:21.090566 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get May 23 12:41:21.090566 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self._get_role(role_id) May 23 12:41:21.090566 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role May 23 12:41:21.090566 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') May 23 12:41:21.091502 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call May 23 12:41:21.091502 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( May 23 12:41:21.091502 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data May 23 12:41:21.091502 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) May 23 12:41:21.091502 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped May 23 12:41:21.091502 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) May 23 12:41:21.091502 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun May 23 12:41:21.091502 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) May 23 12:41:21.091502 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func May 23 12:41:21.091502 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self.get_or_create( May 23 12:41:21.091502 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create May 23 12:41:21.091502 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application with Lock( May 23 12:41:21.091502 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ May 23 12:41:21.091502 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self._enter() May 23 12:41:21.091502 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter May 23 12:41:21.091502 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) May 23 12:41:21.092524 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create May 23 12:41:21.092524 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self.creator() May 23 12:41:21.092524 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value May 23 12:41:21.092524 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application created_value = creator( May 23 12:41:21.092524 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role May 23 12:41:21.092524 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) May 23 12:41:21.092524 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role May 23 12:41:21.092524 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() May 23 12:41:21.092524 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role May 23 12:41:21.092524 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) May 23 12:41:21.092524 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. May 23 12:41:21.092524 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application May 23 12:41:21.092524 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 260/504] 10.222.0.30 () {56 vars in 1102 bytes} [Mon May 23 12:41:20 2022] GET /identity/v3/roles/service => generated 85 bytes in 154 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) May 23 12:41:21.093646 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-773ef5d4-a980-4bb5-8102-c2e64ca96c1b None admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:41:21.106343 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=100 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:41:21.108610 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3fcc8aea-7dc8-4109-8f7c-77296240a189 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:41:21.110115 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3fcc8aea-7dc8-4109-8f7c-77296240a189 None admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:41:21.191760 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-773ef5d4-a980-4bb5-8102-c2e64ca96c1b None admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:41:21.209052 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 245/505] 10.222.0.30 () {58 vars in 1282 bytes} [Mon May 23 12:41:20 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 315 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) May 23 12:41:21.210361 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-202181f7-5080-48fb-af92-834b629c54a4 None None] REQUEST_METHOD: `POST` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:41:21.210777 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-202181f7-5080-48fb-af92-834b629c54a4 None None] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:41:21.211067 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-202181f7-5080-48fb-af92-834b629c54a4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:41:21.217175 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=100 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:41:21.227520 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.auth.core [None req-202181f7-5080-48fb-af92-834b629c54a4 None None] MFA Rules not processed for user `0f7119f8fa5c4445977671140ded91a4`. Rule list: `[]` (Enabled: `True`). {{(pid=92057) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 12:41:21.230389 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed43a91b-a9cb-4745-b964-e1bd364891af None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:41:21.230389 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed43a91b-a9cb-4745-b964-e1bd364891af None admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:41:21.230559 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.fernet_utils [None req-202181f7-5080-48fb-af92-834b629c54a4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92057) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 12:41:21.230645 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed43a91b-a9cb-4745-b964-e1bd364891af None admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:41:21.230645 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed43a91b-a9cb-4745-b964-e1bd364891af None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:41:21.295380 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 246/506] 10.222.0.30 () {58 vars in 1023 bytes} [Mon May 23 12:41:21 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) May 23 12:41:21.296121 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-202181f7-5080-48fb-af92-834b629c54a4 None None] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:41:21.302891 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed43a91b-a9cb-4745-b964-e1bd364891af None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:41:21.304431 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed43a91b-a9cb-4745-b964-e1bd364891af None admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:41:21.326418 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=100 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:41:21.396027 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-202181f7-5080-48fb-af92-834b629c54a4 None None] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:41:21.406793 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 261/507] 10.222.0.30 () {58 vars in 1282 bytes} [Mon May 23 12:41:21 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 314 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) May 23 12:41:21.407650 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed43a91b-a9cb-4745-b964-e1bd364891af None admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:41:21.431883 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=100 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:41:21.432276 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77a7b10b-5ba7-4f3a-ae28-c2481e6daf7d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:41:21.433068 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77a7b10b-5ba7-4f3a-ae28-c2481e6daf7d None admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:41:21.433349 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77a7b10b-5ba7-4f3a-ae28-c2481e6daf7d None admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:41:21.433623 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77a7b10b-5ba7-4f3a-ae28-c2481e6daf7d None admin] PATH_INFO: `/v3/domains/Default` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:41:21.443524 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77a7b10b-5ba7-4f3a-ae28-c2481e6daf7d None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:41:21.447845 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77a7b10b-5ba7-4f3a-ae28-c2481e6daf7d None admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:41:21.463814 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application [None req-77a7b10b-5ba7-4f3a-ae28-c2481e6daf7d None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. May 23 12:41:21.463814 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application Traceback (most recent call last): May 23 12:41:21.463814 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request May 23 12:41:21.463814 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application rv = self.dispatch_request() May 23 12:41:21.463814 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request May 23 12:41:21.463814 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) May 23 12:41:21.463814 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper May 23 12:41:21.463814 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) May 23 12:41:21.463814 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view May 23 12:41:21.463814 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) May 23 12:41:21.463814 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request May 23 12:41:21.463814 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) May 23 12:41:21.463814 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get May 23 12:41:21.463814 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self._get_domain(domain_id) May 23 12:41:21.463814 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain May 23 12:41:21.464682 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) May 23 12:41:21.464682 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped May 23 12:41:21.464682 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) May 23 12:41:21.464682 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun May 23 12:41:21.464682 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) May 23 12:41:21.464682 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func May 23 12:41:21.464682 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self.get_or_create( May 23 12:41:21.464682 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create May 23 12:41:21.464682 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application with Lock( May 23 12:41:21.464682 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ May 23 12:41:21.464682 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self._enter() May 23 12:41:21.464682 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter May 23 12:41:21.464682 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) May 23 12:41:21.464682 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create May 23 12:41:21.464682 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self.creator() May 23 12:41:21.464682 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value May 23 12:41:21.464682 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application created_value = creator( May 23 12:41:21.465560 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain May 23 12:41:21.465560 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) May 23 12:41:21.465560 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. May 23 12:41:21.465560 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application May 23 12:41:21.465560 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 247/508] 10.222.0.30 () {56 vars in 1108 bytes} [Mon May 23 12:41:21 2022] GET /identity/v3/domains/Default => generated 87 bytes in 169 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) May 23 12:41:21.465824 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77a7b10b-5ba7-4f3a-ae28-c2481e6daf7d None admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:41:21.510576 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed43a91b-a9cb-4745-b964-e1bd364891af None admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:41:21.544717 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ebee6dc-58ff-4718-8a1a-40199a03454e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:41:21.545374 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ebee6dc-58ff-4718-8a1a-40199a03454e None admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:41:21.545620 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=100 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:41:21.545733 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ebee6dc-58ff-4718-8a1a-40199a03454e None admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:41:21.545921 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ebee6dc-58ff-4718-8a1a-40199a03454e None admin] PATH_INFO: `/v3/roles` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:41:21.546660 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ebee6dc-58ff-4718-8a1a-40199a03454e None admin] RBAC: Adding query filter params (name=service) {{(pid=92056) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} May 23 12:41:21.546963 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ebee6dc-58ff-4718-8a1a-40199a03454e None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:41:21.548321 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ebee6dc-58ff-4718-8a1a-40199a03454e None admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:41:21.556322 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 262/509] 10.222.0.30 () {56 vars in 1103 bytes} [Mon May 23 12:41:21 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 149 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) May 23 12:41:21.565724 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ebee6dc-58ff-4718-8a1a-40199a03454e None admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:41:21.565853 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77a7b10b-5ba7-4f3a-ae28-c2481e6daf7d None admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:41:21.598210 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fbc7901f-7a2c-48fe-bf5e-f470eba4d718 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:41:21.598996 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbc7901f-7a2c-48fe-bf5e-f470eba4d718 None admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:41:21.599278 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbc7901f-7a2c-48fe-bf5e-f470eba4d718 None admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:41:21.599558 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbc7901f-7a2c-48fe-bf5e-f470eba4d718 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:41:21.610329 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fbc7901f-7a2c-48fe-bf5e-f470eba4d718 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:41:21.611772 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fbc7901f-7a2c-48fe-bf5e-f470eba4d718 None admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:41:21.628243 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application [None req-fbc7901f-7a2c-48fe-bf5e-f470eba4d718 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. May 23 12:41:21.628243 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application Traceback (most recent call last): May 23 12:41:21.628243 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request May 23 12:41:21.628243 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application rv = self.dispatch_request() May 23 12:41:21.628243 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request May 23 12:41:21.628243 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) May 23 12:41:21.628243 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper May 23 12:41:21.628243 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) May 23 12:41:21.628243 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view May 23 12:41:21.628243 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) May 23 12:41:21.628243 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request May 23 12:41:21.628243 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) May 23 12:41:21.628243 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get May 23 12:41:21.628243 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self._get_domain(domain_id) May 23 12:41:21.628243 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain May 23 12:41:21.629132 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) May 23 12:41:21.629132 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped May 23 12:41:21.629132 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) May 23 12:41:21.629132 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun May 23 12:41:21.629132 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) May 23 12:41:21.629132 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func May 23 12:41:21.629132 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self.get_or_create( May 23 12:41:21.629132 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create May 23 12:41:21.629132 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application with Lock( May 23 12:41:21.629132 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ May 23 12:41:21.629132 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self._enter() May 23 12:41:21.629132 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter May 23 12:41:21.629132 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) May 23 12:41:21.629132 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create May 23 12:41:21.629132 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self.creator() May 23 12:41:21.629132 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value May 23 12:41:21.629132 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application created_value = creator( May 23 12:41:21.630046 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain May 23 12:41:21.630046 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) May 23 12:41:21.630046 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. May 23 12:41:21.630046 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application May 23 12:41:21.630046 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 248/510] 10.222.0.30 () {56 vars in 1108 bytes} [Mon May 23 12:41:21 2022] GET /identity/v3/domains/Default => generated 87 bytes in 164 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) May 23 12:41:21.630335 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fbc7901f-7a2c-48fe-bf5e-f470eba4d718 None admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:41:21.641744 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=100 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:41:21.657452 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ebee6dc-58ff-4718-8a1a-40199a03454e None admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:41:21.690641 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5a5a401-c3f5-4c7f-b0a1-e59a2dcb17a4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:41:21.691498 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5a5a401-c3f5-4c7f-b0a1-e59a2dcb17a4 None admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:41:21.691764 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5a5a401-c3f5-4c7f-b0a1-e59a2dcb17a4 None admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:41:21.692057 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5a5a401-c3f5-4c7f-b0a1-e59a2dcb17a4 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:41:21.703100 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5a5a401-c3f5-4c7f-b0a1-e59a2dcb17a4 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:41:21.704531 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5a5a401-c3f5-4c7f-b0a1-e59a2dcb17a4 None admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:41:21.721728 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application [None req-b5a5a401-c3f5-4c7f-b0a1-e59a2dcb17a4 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. May 23 12:41:21.721728 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application Traceback (most recent call last): May 23 12:41:21.721728 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request May 23 12:41:21.721728 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application rv = self.dispatch_request() May 23 12:41:21.721728 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request May 23 12:41:21.721728 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) May 23 12:41:21.721728 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper May 23 12:41:21.721728 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) May 23 12:41:21.721728 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view May 23 12:41:21.721728 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) May 23 12:41:21.721728 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request May 23 12:41:21.721728 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) May 23 12:41:21.721728 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get May 23 12:41:21.721728 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self._get_domain(domain_id) May 23 12:41:21.721728 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain May 23 12:41:21.722800 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) May 23 12:41:21.722800 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped May 23 12:41:21.722800 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) May 23 12:41:21.722800 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun May 23 12:41:21.722800 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) May 23 12:41:21.722800 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func May 23 12:41:21.722800 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self.get_or_create( May 23 12:41:21.722800 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create May 23 12:41:21.722800 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application with Lock( May 23 12:41:21.722800 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ May 23 12:41:21.722800 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self._enter() May 23 12:41:21.722800 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter May 23 12:41:21.722800 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) May 23 12:41:21.722800 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create May 23 12:41:21.722800 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self.creator() May 23 12:41:21.722800 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value May 23 12:41:21.722800 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application created_value = creator( May 23 12:41:21.723754 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain May 23 12:41:21.723754 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) May 23 12:41:21.723754 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. May 23 12:41:21.723754 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application May 23 12:41:21.723754 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 263/511] 10.222.0.30 () {56 vars in 1108 bytes} [Mon May 23 12:41:21 2022] GET /identity/v3/domains/Default => generated 87 bytes in 166 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) May 23 12:41:21.724054 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5a5a401-c3f5-4c7f-b0a1-e59a2dcb17a4 None admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:41:21.734864 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fbc7901f-7a2c-48fe-bf5e-f470eba4d718 None admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:41:21.767741 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26e4dd4f-ca02-4f77-bd03-2d6ce75e9eb1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:41:21.768526 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26e4dd4f-ca02-4f77-bd03-2d6ce75e9eb1 None admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:41:21.768813 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26e4dd4f-ca02-4f77-bd03-2d6ce75e9eb1 None admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:41:21.769091 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26e4dd4f-ca02-4f77-bd03-2d6ce75e9eb1 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:41:21.771129 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=100 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:41:21.780965 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26e4dd4f-ca02-4f77-bd03-2d6ce75e9eb1 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:41:21.782463 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26e4dd4f-ca02-4f77-bd03-2d6ce75e9eb1 None admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:41:21.799374 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application [None req-26e4dd4f-ca02-4f77-bd03-2d6ce75e9eb1 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. May 23 12:41:21.799374 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application Traceback (most recent call last): May 23 12:41:21.799374 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request May 23 12:41:21.799374 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application rv = self.dispatch_request() May 23 12:41:21.799374 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request May 23 12:41:21.799374 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) May 23 12:41:21.799374 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper May 23 12:41:21.799374 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) May 23 12:41:21.799374 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view May 23 12:41:21.799374 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) May 23 12:41:21.799374 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request May 23 12:41:21.799374 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) May 23 12:41:21.799374 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get May 23 12:41:21.799374 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self._get_domain(domain_id) May 23 12:41:21.799374 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain May 23 12:41:21.800257 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) May 23 12:41:21.800257 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped May 23 12:41:21.800257 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) May 23 12:41:21.800257 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun May 23 12:41:21.800257 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) May 23 12:41:21.800257 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func May 23 12:41:21.800257 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self.get_or_create( May 23 12:41:21.800257 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create May 23 12:41:21.800257 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application with Lock( May 23 12:41:21.800257 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ May 23 12:41:21.800257 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self._enter() May 23 12:41:21.800257 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter May 23 12:41:21.800257 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) May 23 12:41:21.800257 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create May 23 12:41:21.800257 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self.creator() May 23 12:41:21.800257 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value May 23 12:41:21.800257 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application created_value = creator( May 23 12:41:21.801139 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain May 23 12:41:21.801139 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) May 23 12:41:21.801139 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. May 23 12:41:21.801139 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application May 23 12:41:21.801139 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 249/512] 10.222.0.30 () {56 vars in 1108 bytes} [Mon May 23 12:41:21 2022] GET /identity/v3/domains/Default => generated 87 bytes in 171 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) May 23 12:41:21.801823 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26e4dd4f-ca02-4f77-bd03-2d6ce75e9eb1 None admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:41:21.829526 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5a5a401-c3f5-4c7f-b0a1-e59a2dcb17a4 None admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:41:21.862488 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-276b6983-e2e9-4a89-b48b-21fd6f0028d8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:41:21.863147 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-276b6983-e2e9-4a89-b48b-21fd6f0028d8 None admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:41:21.863429 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-276b6983-e2e9-4a89-b48b-21fd6f0028d8 None admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:41:21.863699 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-276b6983-e2e9-4a89-b48b-21fd6f0028d8 None admin] PATH_INFO: `/v3/domains` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:41:21.864415 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-276b6983-e2e9-4a89-b48b-21fd6f0028d8 None admin] RBAC: Adding query filter params (name=Default) {{(pid=92056) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} May 23 12:41:21.864719 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-276b6983-e2e9-4a89-b48b-21fd6f0028d8 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:41:21.866197 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-276b6983-e2e9-4a89-b48b-21fd6f0028d8 None admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:41:21.873177 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=100 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:41:21.878583 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 264/513] 10.222.0.30 () {56 vars in 1109 bytes} [Mon May 23 12:41:21 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 155 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) May 23 12:41:21.879373 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-276b6983-e2e9-4a89-b48b-21fd6f0028d8 None admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:41:21.909646 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26e4dd4f-ca02-4f77-bd03-2d6ce75e9eb1 None admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:41:21.942973 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-932fe5a2-c41d-44be-abb6-d424375ae8db None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:41:21.943758 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-932fe5a2-c41d-44be-abb6-d424375ae8db None admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:41:21.944046 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-932fe5a2-c41d-44be-abb6-d424375ae8db None admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:41:21.944324 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-932fe5a2-c41d-44be-abb6-d424375ae8db None admin] PATH_INFO: `/v3/domains/Default` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:41:21.955244 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-932fe5a2-c41d-44be-abb6-d424375ae8db None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:41:21.956725 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-932fe5a2-c41d-44be-abb6-d424375ae8db None admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:41:21.973838 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application [None req-932fe5a2-c41d-44be-abb6-d424375ae8db None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. May 23 12:41:21.973838 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application Traceback (most recent call last): May 23 12:41:21.973838 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request May 23 12:41:21.973838 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application rv = self.dispatch_request() May 23 12:41:21.973838 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request May 23 12:41:21.973838 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) May 23 12:41:21.973838 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper May 23 12:41:21.973838 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) May 23 12:41:21.973838 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view May 23 12:41:21.973838 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) May 23 12:41:21.973838 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request May 23 12:41:21.973838 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) May 23 12:41:21.973838 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get May 23 12:41:21.973838 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self._get_domain(domain_id) May 23 12:41:21.973838 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain May 23 12:41:21.974709 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) May 23 12:41:21.974709 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped May 23 12:41:21.974709 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) May 23 12:41:21.974709 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun May 23 12:41:21.974709 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) May 23 12:41:21.974709 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func May 23 12:41:21.974709 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self.get_or_create( May 23 12:41:21.974709 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create May 23 12:41:21.974709 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application with Lock( May 23 12:41:21.974709 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ May 23 12:41:21.974709 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self._enter() May 23 12:41:21.974709 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter May 23 12:41:21.974709 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) May 23 12:41:21.974709 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create May 23 12:41:21.974709 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self.creator() May 23 12:41:21.974709 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value May 23 12:41:21.974709 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application created_value = creator( May 23 12:41:21.975589 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain May 23 12:41:21.975589 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) May 23 12:41:21.975589 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. May 23 12:41:21.975589 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application May 23 12:41:21.975589 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 250/514] 10.222.0.30 () {56 vars in 1108 bytes} [Mon May 23 12:41:21 2022] GET /identity/v3/domains/Default => generated 87 bytes in 174 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) May 23 12:41:21.975865 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-932fe5a2-c41d-44be-abb6-d424375ae8db None admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:41:21.983706 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-276b6983-e2e9-4a89-b48b-21fd6f0028d8 None admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:41:22.015813 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=100 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:41:22.018437 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a6531f6-2c17-467e-9539-e3acdbf3f0d4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:41:22.019095 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a6531f6-2c17-467e-9539-e3acdbf3f0d4 None admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:41:22.019372 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a6531f6-2c17-467e-9539-e3acdbf3f0d4 None admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:41:22.019640 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a6531f6-2c17-467e-9539-e3acdbf3f0d4 None admin] PATH_INFO: `/v3/domains` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:41:22.020350 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a6531f6-2c17-467e-9539-e3acdbf3f0d4 None admin] RBAC: Adding query filter params (name=Default) {{(pid=92056) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} May 23 12:41:22.020653 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a6531f6-2c17-467e-9539-e3acdbf3f0d4 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:41:22.022053 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a6531f6-2c17-467e-9539-e3acdbf3f0d4 None admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:41:22.033649 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 265/515] 10.222.0.30 () {56 vars in 1109 bytes} [Mon May 23 12:41:21 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 155 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) May 23 12:41:22.034510 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a6531f6-2c17-467e-9539-e3acdbf3f0d4 None admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:41:22.075650 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-932fe5a2-c41d-44be-abb6-d424375ae8db None admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:41:22.092519 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=100 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:41:22.108809 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-deb5aabd-1724-4f56-a247-3e8d42fcb026 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:41:22.109474 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-deb5aabd-1724-4f56-a247-3e8d42fcb026 None admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:41:22.109825 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-deb5aabd-1724-4f56-a247-3e8d42fcb026 None admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:41:22.110127 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-deb5aabd-1724-4f56-a247-3e8d42fcb026 None admin] PATH_INFO: `/v3/domains` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:41:22.110843 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-deb5aabd-1724-4f56-a247-3e8d42fcb026 None admin] RBAC: Adding query filter params (name=Default) {{(pid=92057) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} May 23 12:41:22.111149 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-deb5aabd-1724-4f56-a247-3e8d42fcb026 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:41:22.112518 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-deb5aabd-1724-4f56-a247-3e8d42fcb026 None admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:41:22.124939 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 251/516] 10.222.0.30 () {56 vars in 1109 bytes} [Mon May 23 12:41:21 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 149 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) May 23 12:41:22.125550 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-deb5aabd-1724-4f56-a247-3e8d42fcb026 None admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:41:22.139702 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a6531f6-2c17-467e-9539-e3acdbf3f0d4 None admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:41:22.173153 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b725353c-92cd-44d1-bd52-08a38d15a3ff None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:41:22.173801 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b725353c-92cd-44d1-bd52-08a38d15a3ff None admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:41:22.174115 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b725353c-92cd-44d1-bd52-08a38d15a3ff None admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:41:22.174383 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b725353c-92cd-44d1-bd52-08a38d15a3ff None admin] PATH_INFO: `/v3/domains` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:41:22.175086 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b725353c-92cd-44d1-bd52-08a38d15a3ff None admin] RBAC: Adding query filter params (name=Default) {{(pid=92056) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} May 23 12:41:22.175387 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b725353c-92cd-44d1-bd52-08a38d15a3ff None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:41:22.176757 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b725353c-92cd-44d1-bd52-08a38d15a3ff None admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:41:22.188750 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 266/517] 10.222.0.30 () {56 vars in 1109 bytes} [Mon May 23 12:41:22 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 155 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) May 23 12:41:22.189522 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b725353c-92cd-44d1-bd52-08a38d15a3ff None admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:41:22.222568 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=100 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:41:22.227407 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-deb5aabd-1724-4f56-a247-3e8d42fcb026 None admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:41:22.264454 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b64841e9-020d-432e-bad8-61db59ecc4a9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:41:22.265245 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b64841e9-020d-432e-bad8-61db59ecc4a9 None admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:41:22.265532 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b64841e9-020d-432e-bad8-61db59ecc4a9 None admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:41:22.265810 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b64841e9-020d-432e-bad8-61db59ecc4a9 None admin] PATH_INFO: `/v3/users/cinder` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:41:22.273176 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b64841e9-020d-432e-bad8-61db59ecc4a9 None admin] RBAC: Authorizing `identity:get_user(user_id=cinder)` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:41:22.275425 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b64841e9-020d-432e-bad8-61db59ecc4a9 None admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:41:22.286674 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b725353c-92cd-44d1-bd52-08a38d15a3ff None admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:41:22.289826 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application [None req-b64841e9-020d-432e-bad8-61db59ecc4a9 None admin] Could not find user: cinder.: keystone.exception.UserNotFound: Could not find user: cinder. May 23 12:41:22.289826 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application Traceback (most recent call last): May 23 12:41:22.289826 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request May 23 12:41:22.289826 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application rv = self.dispatch_request() May 23 12:41:22.289826 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request May 23 12:41:22.289826 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) May 23 12:41:22.289826 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper May 23 12:41:22.289826 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) May 23 12:41:22.289826 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view May 23 12:41:22.289826 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) May 23 12:41:22.289826 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request May 23 12:41:22.289826 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) May 23 12:41:22.289826 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get May 23 12:41:22.289826 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self._get_user(user_id) May 23 12:41:22.289826 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user May 23 12:41:22.289826 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) May 23 12:41:22.290817 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped May 23 12:41:22.290817 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) May 23 12:41:22.290817 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper May 23 12:41:22.290817 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) May 23 12:41:22.290817 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper May 23 12:41:22.290817 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) May 23 12:41:22.290817 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun May 23 12:41:22.290817 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) May 23 12:41:22.290817 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func May 23 12:41:22.290817 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self.get_or_create( May 23 12:41:22.290817 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create May 23 12:41:22.290817 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application with Lock( May 23 12:41:22.290817 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ May 23 12:41:22.290817 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self._enter() May 23 12:41:22.290817 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter May 23 12:41:22.290817 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) May 23 12:41:22.290817 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create May 23 12:41:22.290817 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self.creator() May 23 12:41:22.291751 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value May 23 12:41:22.291751 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application created_value = creator( May 23 12:41:22.291751 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user May 23 12:41:22.291751 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) May 23 12:41:22.291751 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user May 23 12:41:22.291751 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) May 23 12:41:22.291751 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user May 23 12:41:22.291751 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) May 23 12:41:22.291751 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: cinder. May 23 12:41:22.291751 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application May 23 12:41:22.291751 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 252/518] 10.222.0.30 () {56 vars in 1099 bytes} [Mon May 23 12:41:22 2022] GET /identity/v3/users/cinder => generated 84 bytes in 166 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) May 23 12:41:22.292798 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b64841e9-020d-432e-bad8-61db59ecc4a9 None admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:41:22.306409 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=100 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:41:22.329344 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d607c850-89a2-45f8-9414-a458dc47d703 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:41:22.330258 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d607c850-89a2-45f8-9414-a458dc47d703 None admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:41:22.330544 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d607c850-89a2-45f8-9414-a458dc47d703 None admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:41:22.330822 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d607c850-89a2-45f8-9414-a458dc47d703 None admin] PATH_INFO: `/v3/domains` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:41:22.331540 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d607c850-89a2-45f8-9414-a458dc47d703 None admin] RBAC: Adding query filter params (name=Default) {{(pid=92056) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} May 23 12:41:22.331847 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d607c850-89a2-45f8-9414-a458dc47d703 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:41:22.333224 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d607c850-89a2-45f8-9414-a458dc47d703 None admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:41:22.344553 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 267/519] 10.222.0.30 () {56 vars in 1109 bytes} [Mon May 23 12:41:22 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 155 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) May 23 12:41:22.345700 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d607c850-89a2-45f8-9414-a458dc47d703 None admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:41:22.396338 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b64841e9-020d-432e-bad8-61db59ecc4a9 None admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:41:22.428982 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6371c1c5-06a0-43ac-90a9-31b263149a17 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:41:22.429765 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6371c1c5-06a0-43ac-90a9-31b263149a17 None admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:41:22.430069 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6371c1c5-06a0-43ac-90a9-31b263149a17 None admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:41:22.430341 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6371c1c5-06a0-43ac-90a9-31b263149a17 None admin] PATH_INFO: `/v3/projects/service` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:41:22.436873 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6371c1c5-06a0-43ac-90a9-31b263149a17 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:41:22.438348 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6371c1c5-06a0-43ac-90a9-31b263149a17 None admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:41:22.443474 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d607c850-89a2-45f8-9414-a458dc47d703 None admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:41:22.453380 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application [None req-6371c1c5-06a0-43ac-90a9-31b263149a17 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. May 23 12:41:22.453380 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application Traceback (most recent call last): May 23 12:41:22.453380 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request May 23 12:41:22.453380 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application rv = self.dispatch_request() May 23 12:41:22.453380 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request May 23 12:41:22.453380 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) May 23 12:41:22.453380 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper May 23 12:41:22.453380 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) May 23 12:41:22.453380 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view May 23 12:41:22.453380 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) May 23 12:41:22.453380 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request May 23 12:41:22.453380 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) May 23 12:41:22.453380 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get May 23 12:41:22.453380 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self._get_project(project_id) May 23 12:41:22.453380 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project May 23 12:41:22.454308 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) May 23 12:41:22.454308 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped May 23 12:41:22.454308 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) May 23 12:41:22.454308 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun May 23 12:41:22.454308 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) May 23 12:41:22.454308 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func May 23 12:41:22.454308 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self.get_or_create( May 23 12:41:22.454308 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create May 23 12:41:22.454308 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application with Lock( May 23 12:41:22.454308 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ May 23 12:41:22.454308 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self._enter() May 23 12:41:22.454308 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter May 23 12:41:22.454308 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) May 23 12:41:22.454308 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create May 23 12:41:22.454308 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self.creator() May 23 12:41:22.454308 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value May 23 12:41:22.454308 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application created_value = creator( May 23 12:41:22.455338 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project May 23 12:41:22.455338 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) May 23 12:41:22.455338 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project May 23 12:41:22.455338 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() May 23 12:41:22.455338 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project May 23 12:41:22.455338 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) May 23 12:41:22.455338 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. May 23 12:41:22.455338 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application May 23 12:41:22.455338 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 253/520] 10.222.0.30 () {56 vars in 1111 bytes} [Mon May 23 12:41:22 2022] GET /identity/v3/projects/service => generated 88 bytes in 163 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) May 23 12:41:22.455338 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6371c1c5-06a0-43ac-90a9-31b263149a17 None admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:41:22.477218 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=100 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:41:22.483413 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93307995-fd77-4b96-a9b9-2e4b7c22cb73 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:41:22.486400 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93307995-fd77-4b96-a9b9-2e4b7c22cb73 None admin] REQUEST_METHOD: `POST` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:41:22.486400 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93307995-fd77-4b96-a9b9-2e4b7c22cb73 None admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:41:22.486400 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93307995-fd77-4b96-a9b9-2e4b7c22cb73 None admin] PATH_INFO: `/v3/users` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:41:22.486400 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93307995-fd77-4b96-a9b9-2e4b7c22cb73 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:41:22.486901 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93307995-fd77-4b96-a9b9-2e4b7c22cb73 None admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:41:22.555752 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6371c1c5-06a0-43ac-90a9-31b263149a17 None admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:41:22.572157 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 268/521] 10.222.0.30 () {60 vars in 1132 bytes} [Mon May 23 12:41:22 2022] POST /identity/v3/users => generated 247 bytes in 227 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) May 23 12:41:22.572858 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93307995-fd77-4b96-a9b9-2e4b7c22cb73 None admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:41:22.593842 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79e53d16-e92b-4248-9de9-59da185bd927 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:41:22.594601 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79e53d16-e92b-4248-9de9-59da185bd927 None admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:41:22.594885 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79e53d16-e92b-4248-9de9-59da185bd927 None admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:41:22.595166 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79e53d16-e92b-4248-9de9-59da185bd927 None admin] PATH_INFO: `/v3/projects/service` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:41:22.603958 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79e53d16-e92b-4248-9de9-59da185bd927 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:41:22.605606 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79e53d16-e92b-4248-9de9-59da185bd927 None admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:41:22.610980 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:41:22.623256 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application [None req-79e53d16-e92b-4248-9de9-59da185bd927 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. May 23 12:41:22.623256 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application Traceback (most recent call last): May 23 12:41:22.623256 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request May 23 12:41:22.623256 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application rv = self.dispatch_request() May 23 12:41:22.623256 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request May 23 12:41:22.623256 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) May 23 12:41:22.623256 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper May 23 12:41:22.623256 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) May 23 12:41:22.623256 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view May 23 12:41:22.623256 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) May 23 12:41:22.623256 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request May 23 12:41:22.623256 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) May 23 12:41:22.623256 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get May 23 12:41:22.623256 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self._get_project(project_id) May 23 12:41:22.623256 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project May 23 12:41:22.624251 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) May 23 12:41:22.624251 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped May 23 12:41:22.624251 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) May 23 12:41:22.624251 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun May 23 12:41:22.624251 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) May 23 12:41:22.624251 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func May 23 12:41:22.624251 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self.get_or_create( May 23 12:41:22.624251 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create May 23 12:41:22.624251 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application with Lock( May 23 12:41:22.624251 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ May 23 12:41:22.624251 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self._enter() May 23 12:41:22.624251 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter May 23 12:41:22.624251 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) May 23 12:41:22.624251 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create May 23 12:41:22.624251 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self.creator() May 23 12:41:22.624251 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value May 23 12:41:22.624251 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application created_value = creator( May 23 12:41:22.625282 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project May 23 12:41:22.625282 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) May 23 12:41:22.625282 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project May 23 12:41:22.625282 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() May 23 12:41:22.625282 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project May 23 12:41:22.625282 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) May 23 12:41:22.625282 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. May 23 12:41:22.625282 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application May 23 12:41:22.625282 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 254/522] 10.222.0.30 () {56 vars in 1111 bytes} [Mon May 23 12:41:22 2022] GET /identity/v3/projects/service => generated 88 bytes in 169 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) May 23 12:41:22.626833 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79e53d16-e92b-4248-9de9-59da185bd927 None admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:41:22.697747 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93307995-fd77-4b96-a9b9-2e4b7c22cb73 None admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:41:22.721821 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=100 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:41:22.738843 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3012bce6-50a1-4d6a-8db4-1b6314164fa6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:41:22.739525 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3012bce6-50a1-4d6a-8db4-1b6314164fa6 None admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:41:22.739802 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3012bce6-50a1-4d6a-8db4-1b6314164fa6 None admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:41:22.740072 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3012bce6-50a1-4d6a-8db4-1b6314164fa6 None admin] PATH_INFO: `/v3/users` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:41:22.740918 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3012bce6-50a1-4d6a-8db4-1b6314164fa6 None admin] RBAC: Adding query filter params (domain_id=default, name=cinder) {{(pid=92056) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} May 23 12:41:22.741229 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3012bce6-50a1-4d6a-8db4-1b6314164fa6 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:41:22.743476 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3012bce6-50a1-4d6a-8db4-1b6314164fa6 None admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:41:22.747188 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79e53d16-e92b-4248-9de9-59da185bd927 None admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:41:22.782903 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 269/523] 10.222.0.30 () {56 vars in 1137 bytes} [Mon May 23 12:41:22 2022] GET /identity/v3/users?domain_id=default&name=cinder => generated 372 bytes in 210 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) May 23 12:41:22.784138 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3012bce6-50a1-4d6a-8db4-1b6314164fa6 None admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:41:22.787415 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c784bf90-bff0-4504-bdcd-bd73a8e8ae0c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:41:22.788438 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c784bf90-bff0-4504-bdcd-bd73a8e8ae0c None admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:41:22.788580 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c784bf90-bff0-4504-bdcd-bd73a8e8ae0c None admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:41:22.789318 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c784bf90-bff0-4504-bdcd-bd73a8e8ae0c None admin] PATH_INFO: `/v3/users/neutron` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:41:22.799379 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c784bf90-bff0-4504-bdcd-bd73a8e8ae0c None admin] RBAC: Authorizing `identity:get_user(user_id=neutron)` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:41:22.800873 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c784bf90-bff0-4504-bdcd-bd73a8e8ae0c None admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:41:22.817573 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application [None req-c784bf90-bff0-4504-bdcd-bd73a8e8ae0c None admin] Could not find user: neutron.: keystone.exception.UserNotFound: Could not find user: neutron. May 23 12:41:22.817573 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application Traceback (most recent call last): May 23 12:41:22.817573 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request May 23 12:41:22.817573 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application rv = self.dispatch_request() May 23 12:41:22.817573 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request May 23 12:41:22.817573 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) May 23 12:41:22.817573 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper May 23 12:41:22.817573 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) May 23 12:41:22.817573 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view May 23 12:41:22.817573 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) May 23 12:41:22.817573 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request May 23 12:41:22.817573 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) May 23 12:41:22.817573 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get May 23 12:41:22.817573 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self._get_user(user_id) May 23 12:41:22.817573 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user May 23 12:41:22.817573 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) May 23 12:41:22.818709 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped May 23 12:41:22.818709 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) May 23 12:41:22.818709 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper May 23 12:41:22.818709 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) May 23 12:41:22.818709 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper May 23 12:41:22.818709 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) May 23 12:41:22.818709 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun May 23 12:41:22.818709 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) May 23 12:41:22.818709 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func May 23 12:41:22.818709 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self.get_or_create( May 23 12:41:22.818709 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create May 23 12:41:22.818709 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application with Lock( May 23 12:41:22.818709 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ May 23 12:41:22.818709 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self._enter() May 23 12:41:22.818709 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter May 23 12:41:22.818709 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) May 23 12:41:22.818709 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create May 23 12:41:22.818709 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self.creator() May 23 12:41:22.820312 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value May 23 12:41:22.820312 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application created_value = creator( May 23 12:41:22.820312 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user May 23 12:41:22.820312 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) May 23 12:41:22.820312 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user May 23 12:41:22.820312 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) May 23 12:41:22.820312 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user May 23 12:41:22.820312 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) May 23 12:41:22.820312 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: neutron. May 23 12:41:22.820312 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application May 23 12:41:22.820312 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 255/524] 10.222.0.30 () {56 vars in 1102 bytes} [Mon May 23 12:41:22 2022] GET /identity/v3/users/neutron => generated 85 bytes in 194 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) May 23 12:41:22.821001 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c784bf90-bff0-4504-bdcd-bd73a8e8ae0c None admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:41:22.890456 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=100 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:41:22.900562 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3012bce6-50a1-4d6a-8db4-1b6314164fa6 None admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:41:22.924467 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c784bf90-bff0-4504-bdcd-bd73a8e8ae0c None admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:41:22.936944 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-449f1798-2f94-4ada-b659-4fff2b55c276 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:41:22.937639 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-449f1798-2f94-4ada-b659-4fff2b55c276 None admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:41:22.938232 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-449f1798-2f94-4ada-b659-4fff2b55c276 None admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:41:22.938498 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-449f1798-2f94-4ada-b659-4fff2b55c276 None admin] PATH_INFO: `/v3/projects` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:41:22.939460 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-449f1798-2f94-4ada-b659-4fff2b55c276 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=92056) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} May 23 12:41:22.939785 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-449f1798-2f94-4ada-b659-4fff2b55c276 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:41:22.941717 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-449f1798-2f94-4ada-b659-4fff2b55c276 None admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:41:22.959105 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4b9e9c2-58f9-46c8-a281-61e7dd6a313d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:41:22.959774 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4b9e9c2-58f9-46c8-a281-61e7dd6a313d None admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:41:22.960062 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4b9e9c2-58f9-46c8-a281-61e7dd6a313d None admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:41:22.960869 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4b9e9c2-58f9-46c8-a281-61e7dd6a313d None admin] PATH_INFO: `/v3/projects` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:41:22.961920 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c4b9e9c2-58f9-46c8-a281-61e7dd6a313d None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=92057) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} May 23 12:41:22.962225 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c4b9e9c2-58f9-46c8-a281-61e7dd6a313d None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:41:22.968153 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 270/525] 10.222.0.30 () {56 vars in 1148 bytes} [Mon May 23 12:41:22 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 180 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) May 23 12:41:22.968289 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c4b9e9c2-58f9-46c8-a281-61e7dd6a313d None admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:41:22.968377 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-449f1798-2f94-4ada-b659-4fff2b55c276 None admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:41:22.979043 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=100 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:41:22.991036 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 256/526] 10.222.0.30 () {56 vars in 1148 bytes} [Mon May 23 12:41:22 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 167 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) May 23 12:41:22.991036 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4b9e9c2-58f9-46c8-a281-61e7dd6a313d None admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:41:23.084280 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-449f1798-2f94-4ada-b659-4fff2b55c276 None admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:41:23.097080 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4b9e9c2-58f9-46c8-a281-61e7dd6a313d None admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:41:23.122978 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9d5efc3-5b6e-4a68-b442-5c8bd23d310c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:41:23.123629 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9d5efc3-5b6e-4a68-b442-5c8bd23d310c None admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:41:23.123912 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9d5efc3-5b6e-4a68-b442-5c8bd23d310c None admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:41:23.124188 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9d5efc3-5b6e-4a68-b442-5c8bd23d310c None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:41:23.140303 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43dde76f-4186-4ed5-bda6-e202268a3660 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:41:23.141042 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43dde76f-4186-4ed5-bda6-e202268a3660 None admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:41:23.141263 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43dde76f-4186-4ed5-bda6-e202268a3660 None admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:41:23.141544 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43dde76f-4186-4ed5-bda6-e202268a3660 None admin] PATH_INFO: `/v3/users` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:41:23.142411 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43dde76f-4186-4ed5-bda6-e202268a3660 None admin] RBAC: Adding query filter params (domain_id=default, name=neutron) {{(pid=92057) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} May 23 12:41:23.142724 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43dde76f-4186-4ed5-bda6-e202268a3660 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:41:23.144161 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43dde76f-4186-4ed5-bda6-e202268a3660 None admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:41:23.148836 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=100 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:41:23.169916 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 257/527] 10.222.0.30 () {56 vars in 1139 bytes} [Mon May 23 12:41:22 2022] GET /identity/v3/users?domain_id=default&name=neutron => generated 374 bytes in 179 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) May 23 12:41:23.169916 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43dde76f-4186-4ed5-bda6-e202268a3660 None admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:41:23.201196 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9d5efc3-5b6e-4a68-b442-5c8bd23d310c None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:41:23.203001 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9d5efc3-5b6e-4a68-b442-5c8bd23d310c None admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:41:23.231204 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=100 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:41:23.278259 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43dde76f-4186-4ed5-bda6-e202268a3660 None admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:41:23.314509 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 271/528] 10.222.0.30 () {58 vars in 1282 bytes} [Mon May 23 12:41:22 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 351 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) May 23 12:41:23.315350 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9d5efc3-5b6e-4a68-b442-5c8bd23d310c None admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:41:23.319128 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-edc29e65-93a8-43a4-93a2-6ac145747efc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:41:23.320872 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edc29e65-93a8-43a4-93a2-6ac145747efc None admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:41:23.321257 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edc29e65-93a8-43a4-93a2-6ac145747efc None admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:41:23.321547 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edc29e65-93a8-43a4-93a2-6ac145747efc None admin] PATH_INFO: `/v3/role_assignments` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:41:23.322415 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-edc29e65-93a8-43a4-93a2-6ac145747efc None admin] RBAC: Adding query filter params (role.id=41ab28c1ae3d4941bd09fb5d35f1dc54, scope.project.id=9c389ae65aaa4542bb5a5c4fd0bacf5b, user.id=5cea0feed4254ec085854986f99acf30) {{(pid=92057) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} May 23 12:41:23.322735 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-edc29e65-93a8-43a4-93a2-6ac145747efc None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:41:23.324201 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-edc29e65-93a8-43a4-93a2-6ac145747efc None admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:41:23.330252 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 258/529] 10.222.0.30 () {56 vars in 1374 bytes} [Mon May 23 12:41:23 2022] GET /identity/v3/role_assignments?user.id=5cea0feed4254ec085854986f99acf30&scope.project.id=9c389ae65aaa4542bb5a5c4fd0bacf5b&role.id=41ab28c1ae3d4941bd09fb5d35f1dc54 => generated 259 bytes in 164 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) May 23 12:41:23.339612 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-edc29e65-93a8-43a4-93a2-6ac145747efc None admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:41:23.381464 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=100 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:41:23.439286 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9d5efc3-5b6e-4a68-b442-5c8bd23d310c None admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:41:23.481526 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea9c6d0f-98fa-4c83-9003-dfbe74217fdb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:41:23.482262 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea9c6d0f-98fa-4c83-9003-dfbe74217fdb None admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:41:23.482609 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea9c6d0f-98fa-4c83-9003-dfbe74217fdb None admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:41:23.482951 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea9c6d0f-98fa-4c83-9003-dfbe74217fdb None admin] PATH_INFO: `/v3/role_assignments` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:41:23.483923 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea9c6d0f-98fa-4c83-9003-dfbe74217fdb None admin] RBAC: Adding query filter params (role.id=41ab28c1ae3d4941bd09fb5d35f1dc54, scope.project.id=9c389ae65aaa4542bb5a5c4fd0bacf5b, user.id=cbe9931d94d44cc7ace0e83a7e5015fc) {{(pid=92056) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} May 23 12:41:23.484243 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea9c6d0f-98fa-4c83-9003-dfbe74217fdb None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:41:23.485819 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea9c6d0f-98fa-4c83-9003-dfbe74217fdb None admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:41:23.493674 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 272/530] 10.222.0.30 () {56 vars in 1374 bytes} [Mon May 23 12:41:23 2022] GET /identity/v3/role_assignments?user.id=cbe9931d94d44cc7ace0e83a7e5015fc&scope.project.id=9c389ae65aaa4542bb5a5c4fd0bacf5b&role.id=41ab28c1ae3d4941bd09fb5d35f1dc54 => generated 259 bytes in 178 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) May 23 12:41:23.494681 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea9c6d0f-98fa-4c83-9003-dfbe74217fdb None admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:41:23.496353 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-edc29e65-93a8-43a4-93a2-6ac145747efc None admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:41:23.519890 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=100 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:41:23.536891 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63cdf7c2-5e78-485b-84f2-dedd2bb28aa7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:41:23.544209 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63cdf7c2-5e78-485b-84f2-dedd2bb28aa7 None admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:41:23.544209 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63cdf7c2-5e78-485b-84f2-dedd2bb28aa7 None admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:41:23.544209 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63cdf7c2-5e78-485b-84f2-dedd2bb28aa7 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:41:23.613000 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea9c6d0f-98fa-4c83-9003-dfbe74217fdb None admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:41:23.618103 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63cdf7c2-5e78-485b-84f2-dedd2bb28aa7 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:41:23.619973 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63cdf7c2-5e78-485b-84f2-dedd2bb28aa7 None admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:41:23.632800 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=100 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:41:23.653264 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-119254e3-6221-4330-a039-22e9790dc566 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:41:23.654474 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-119254e3-6221-4330-a039-22e9790dc566 None admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:41:23.654606 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-119254e3-6221-4330-a039-22e9790dc566 None admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:41:23.654967 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-119254e3-6221-4330-a039-22e9790dc566 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:41:23.667378 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-119254e3-6221-4330-a039-22e9790dc566 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:41:23.669586 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-119254e3-6221-4330-a039-22e9790dc566 None admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:41:23.689014 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application [None req-119254e3-6221-4330-a039-22e9790dc566 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. May 23 12:41:23.689014 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application Traceback (most recent call last): May 23 12:41:23.689014 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request May 23 12:41:23.689014 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application rv = self.dispatch_request() May 23 12:41:23.689014 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request May 23 12:41:23.689014 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) May 23 12:41:23.689014 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper May 23 12:41:23.689014 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) May 23 12:41:23.689014 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view May 23 12:41:23.689014 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) May 23 12:41:23.689014 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request May 23 12:41:23.689014 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) May 23 12:41:23.689014 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get May 23 12:41:23.689014 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self._get_domain(domain_id) May 23 12:41:23.689014 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain May 23 12:41:23.690055 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) May 23 12:41:23.690055 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped May 23 12:41:23.690055 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) May 23 12:41:23.690055 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun May 23 12:41:23.690055 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) May 23 12:41:23.690055 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func May 23 12:41:23.690055 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self.get_or_create( May 23 12:41:23.690055 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create May 23 12:41:23.690055 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application with Lock( May 23 12:41:23.690055 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ May 23 12:41:23.690055 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self._enter() May 23 12:41:23.690055 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter May 23 12:41:23.690055 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) May 23 12:41:23.690055 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create May 23 12:41:23.690055 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self.creator() May 23 12:41:23.690055 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value May 23 12:41:23.690055 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application created_value = creator( May 23 12:41:23.690943 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain May 23 12:41:23.690943 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) May 23 12:41:23.690943 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. May 23 12:41:23.690943 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application May 23 12:41:23.690943 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 273/531] 10.222.0.30 () {56 vars in 1108 bytes} [Mon May 23 12:41:23 2022] GET /identity/v3/domains/Default => generated 87 bytes in 196 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) May 23 12:41:23.702668 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-119254e3-6221-4330-a039-22e9790dc566 None admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:41:23.775974 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 259/532] 10.222.0.30 () {58 vars in 1282 bytes} [Mon May 23 12:41:23 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 437 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) May 23 12:41:23.782884 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63cdf7c2-5e78-485b-84f2-dedd2bb28aa7 None admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:41:23.823767 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=100 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:41:23.844784 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-119254e3-6221-4330-a039-22e9790dc566 None admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:41:23.910882 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4a1b28b-638c-4cb4-bd1c-a4ff26f30c35 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:41:23.911698 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4a1b28b-638c-4cb4-bd1c-a4ff26f30c35 None admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:41:23.912032 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4a1b28b-638c-4cb4-bd1c-a4ff26f30c35 None admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:41:23.912354 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4a1b28b-638c-4cb4-bd1c-a4ff26f30c35 None admin] PATH_INFO: `/v3/domains` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:41:23.913076 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4a1b28b-638c-4cb4-bd1c-a4ff26f30c35 None admin] RBAC: Adding query filter params (name=Default) {{(pid=92056) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} May 23 12:41:23.913911 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4a1b28b-638c-4cb4-bd1c-a4ff26f30c35 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:41:23.915302 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4a1b28b-638c-4cb4-bd1c-a4ff26f30c35 None admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:41:23.938274 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 274/533] 10.222.0.30 () {56 vars in 1109 bytes} [Mon May 23 12:41:23 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 239 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) May 23 12:41:23.943422 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4a1b28b-638c-4cb4-bd1c-a4ff26f30c35 None admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:41:23.955817 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63cdf7c2-5e78-485b-84f2-dedd2bb28aa7 None admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:41:23.974827 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=100 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:41:24.021046 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-950fd66b-9a07-4f29-8f4c-1705a7daf948 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:41:24.021848 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-950fd66b-9a07-4f29-8f4c-1705a7daf948 None admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:41:24.022199 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-950fd66b-9a07-4f29-8f4c-1705a7daf948 None admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:41:24.022536 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-950fd66b-9a07-4f29-8f4c-1705a7daf948 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:41:24.052627 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4a1b28b-638c-4cb4-bd1c-a4ff26f30c35 None admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:41:24.054000 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-950fd66b-9a07-4f29-8f4c-1705a7daf948 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:41:24.056619 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-950fd66b-9a07-4f29-8f4c-1705a7daf948 None admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:41:24.084196 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application [None req-950fd66b-9a07-4f29-8f4c-1705a7daf948 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. May 23 12:41:24.084196 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application Traceback (most recent call last): May 23 12:41:24.084196 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request May 23 12:41:24.084196 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application rv = self.dispatch_request() May 23 12:41:24.084196 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request May 23 12:41:24.084196 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) May 23 12:41:24.084196 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper May 23 12:41:24.084196 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) May 23 12:41:24.084196 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view May 23 12:41:24.084196 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) May 23 12:41:24.084196 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request May 23 12:41:24.084196 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) May 23 12:41:24.084196 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get May 23 12:41:24.084196 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self._get_domain(domain_id) May 23 12:41:24.084196 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain May 23 12:41:24.085378 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) May 23 12:41:24.085378 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped May 23 12:41:24.085378 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) May 23 12:41:24.085378 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun May 23 12:41:24.085378 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) May 23 12:41:24.085378 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func May 23 12:41:24.085378 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self.get_or_create( May 23 12:41:24.085378 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create May 23 12:41:24.085378 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application with Lock( May 23 12:41:24.085378 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ May 23 12:41:24.085378 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self._enter() May 23 12:41:24.085378 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter May 23 12:41:24.085378 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) May 23 12:41:24.085378 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create May 23 12:41:24.085378 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self.creator() May 23 12:41:24.085378 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value May 23 12:41:24.085378 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application created_value = creator( May 23 12:41:24.087017 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain May 23 12:41:24.087017 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) May 23 12:41:24.087017 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. May 23 12:41:24.087017 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application May 23 12:41:24.092455 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64f184c7-e3bb-43a0-829f-6c7f785b2bec None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:41:24.093438 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64f184c7-e3bb-43a0-829f-6c7f785b2bec None admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:41:24.093862 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64f184c7-e3bb-43a0-829f-6c7f785b2bec None admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:41:24.094277 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 260/534] 10.222.0.30 () {56 vars in 1108 bytes} [Mon May 23 12:41:23 2022] GET /identity/v3/domains/Default => generated 87 bytes in 312 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) May 23 12:41:24.094447 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64f184c7-e3bb-43a0-829f-6c7f785b2bec None admin] PATH_INFO: `/v3/projects/service` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:41:24.099827 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-950fd66b-9a07-4f29-8f4c-1705a7daf948 None admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:41:24.102088 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-64f184c7-e3bb-43a0-829f-6c7f785b2bec None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:41:24.103815 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-64f184c7-e3bb-43a0-829f-6c7f785b2bec None admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:41:24.118906 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application [None req-64f184c7-e3bb-43a0-829f-6c7f785b2bec None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. May 23 12:41:24.118906 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application Traceback (most recent call last): May 23 12:41:24.118906 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request May 23 12:41:24.118906 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application rv = self.dispatch_request() May 23 12:41:24.118906 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request May 23 12:41:24.118906 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) May 23 12:41:24.118906 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper May 23 12:41:24.118906 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) May 23 12:41:24.118906 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view May 23 12:41:24.118906 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) May 23 12:41:24.118906 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request May 23 12:41:24.118906 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) May 23 12:41:24.118906 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get May 23 12:41:24.118906 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self._get_project(project_id) May 23 12:41:24.118906 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project May 23 12:41:24.119982 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) May 23 12:41:24.119982 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped May 23 12:41:24.119982 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) May 23 12:41:24.119982 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun May 23 12:41:24.119982 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) May 23 12:41:24.119982 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func May 23 12:41:24.119982 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self.get_or_create( May 23 12:41:24.119982 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create May 23 12:41:24.119982 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application with Lock( May 23 12:41:24.119982 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ May 23 12:41:24.119982 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self._enter() May 23 12:41:24.119982 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter May 23 12:41:24.119982 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) May 23 12:41:24.119982 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create May 23 12:41:24.119982 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self.creator() May 23 12:41:24.119982 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value May 23 12:41:24.119982 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application created_value = creator( May 23 12:41:24.121053 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project May 23 12:41:24.121053 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) May 23 12:41:24.121053 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project May 23 12:41:24.121053 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() May 23 12:41:24.121053 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project May 23 12:41:24.121053 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) May 23 12:41:24.121053 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. May 23 12:41:24.121053 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application May 23 12:41:24.123324 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 275/535] 10.222.0.30 () {56 vars in 1111 bytes} [Mon May 23 12:41:23 2022] GET /identity/v3/projects/service => generated 88 bytes in 180 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) May 23 12:41:24.134852 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64f184c7-e3bb-43a0-829f-6c7f785b2bec None admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:41:24.219454 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=100 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:41:24.249946 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-950fd66b-9a07-4f29-8f4c-1705a7daf948 None admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:41:24.293158 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2a5b5f4-1edf-49ed-987d-476be4e937ee None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:41:24.294259 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2a5b5f4-1edf-49ed-987d-476be4e937ee None admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:41:24.294546 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2a5b5f4-1edf-49ed-987d-476be4e937ee None admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:41:24.294975 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2a5b5f4-1edf-49ed-987d-476be4e937ee None admin] PATH_INFO: `/v3/domains` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:41:24.295941 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2a5b5f4-1edf-49ed-987d-476be4e937ee None admin] RBAC: Adding query filter params (name=Default) {{(pid=92057) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} May 23 12:41:24.296249 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2a5b5f4-1edf-49ed-987d-476be4e937ee None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:41:24.298018 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2a5b5f4-1edf-49ed-987d-476be4e937ee None admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:41:24.310667 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=100 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:41:24.317925 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 261/536] 10.222.0.30 () {56 vars in 1109 bytes} [Mon May 23 12:41:24 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 214 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) May 23 12:41:24.319391 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2a5b5f4-1edf-49ed-987d-476be4e937ee None admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:41:24.342409 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64f184c7-e3bb-43a0-829f-6c7f785b2bec None admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:41:24.419049 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d39302eb-5903-404b-ab2e-2c6261ec7141 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:41:24.419765 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d39302eb-5903-404b-ab2e-2c6261ec7141 None admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:41:24.420052 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d39302eb-5903-404b-ab2e-2c6261ec7141 None admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:41:24.420329 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d39302eb-5903-404b-ab2e-2c6261ec7141 None admin] PATH_INFO: `/v3/projects` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:41:24.421121 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d39302eb-5903-404b-ab2e-2c6261ec7141 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=92056) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} May 23 12:41:24.421432 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d39302eb-5903-404b-ab2e-2c6261ec7141 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:41:24.424685 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d39302eb-5903-404b-ab2e-2c6261ec7141 None admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:41:24.456920 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 276/537] 10.222.0.30 () {56 vars in 1148 bytes} [Mon May 23 12:41:24 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 316 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) May 23 12:41:24.459018 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d39302eb-5903-404b-ab2e-2c6261ec7141 None admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:41:24.475782 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2a5b5f4-1edf-49ed-987d-476be4e937ee None admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:41:24.528095 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=100 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:41:24.533609 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf654f70-4870-4739-b75e-de0dd1c121bd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:41:24.534962 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf654f70-4870-4739-b75e-de0dd1c121bd None admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:41:24.535696 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf654f70-4870-4739-b75e-de0dd1c121bd None admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:41:24.536196 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf654f70-4870-4739-b75e-de0dd1c121bd None admin] PATH_INFO: `/v3/projects/service` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:41:24.546024 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf654f70-4870-4739-b75e-de0dd1c121bd None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:41:24.548653 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf654f70-4870-4739-b75e-de0dd1c121bd None admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:41:24.566361 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application [None req-bf654f70-4870-4739-b75e-de0dd1c121bd None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. May 23 12:41:24.566361 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application Traceback (most recent call last): May 23 12:41:24.566361 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request May 23 12:41:24.566361 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application rv = self.dispatch_request() May 23 12:41:24.566361 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request May 23 12:41:24.566361 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) May 23 12:41:24.566361 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper May 23 12:41:24.566361 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) May 23 12:41:24.566361 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view May 23 12:41:24.566361 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) May 23 12:41:24.566361 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request May 23 12:41:24.566361 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) May 23 12:41:24.566361 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get May 23 12:41:24.566361 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self._get_project(project_id) May 23 12:41:24.566361 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project May 23 12:41:24.567284 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) May 23 12:41:24.567284 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped May 23 12:41:24.567284 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) May 23 12:41:24.567284 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun May 23 12:41:24.567284 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) May 23 12:41:24.567284 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func May 23 12:41:24.567284 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self.get_or_create( May 23 12:41:24.567284 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create May 23 12:41:24.567284 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application with Lock( May 23 12:41:24.567284 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ May 23 12:41:24.567284 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self._enter() May 23 12:41:24.567284 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter May 23 12:41:24.567284 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) May 23 12:41:24.567284 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create May 23 12:41:24.567284 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self.creator() May 23 12:41:24.567284 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value May 23 12:41:24.567284 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application created_value = creator( May 23 12:41:24.568148 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project May 23 12:41:24.568148 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) May 23 12:41:24.568148 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project May 23 12:41:24.568148 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() May 23 12:41:24.568148 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project May 23 12:41:24.568148 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) May 23 12:41:24.568148 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. May 23 12:41:24.568148 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application May 23 12:41:24.572348 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 262/538] 10.222.0.30 () {56 vars in 1111 bytes} [Mon May 23 12:41:24 2022] GET /identity/v3/projects/service => generated 88 bytes in 253 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) May 23 12:41:24.578939 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf654f70-4870-4739-b75e-de0dd1c121bd None admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:41:24.618631 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d39302eb-5903-404b-ab2e-2c6261ec7141 None admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:41:24.660926 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c75bfbe2-293e-4ff7-83ce-08c13bf394bf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:41:24.661584 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c75bfbe2-293e-4ff7-83ce-08c13bf394bf None admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:41:24.661863 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c75bfbe2-293e-4ff7-83ce-08c13bf394bf None admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:41:24.662167 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c75bfbe2-293e-4ff7-83ce-08c13bf394bf None admin] PATH_INFO: `/v3/role_assignments` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:41:24.664826 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c75bfbe2-293e-4ff7-83ce-08c13bf394bf None admin] RBAC: Adding query filter params (role.id=41ab28c1ae3d4941bd09fb5d35f1dc54, scope.project.id=9c389ae65aaa4542bb5a5c4fd0bacf5b, user.id=1666097f8ed545c6b0e4809c93e4775a) {{(pid=92056) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} May 23 12:41:24.665850 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c75bfbe2-293e-4ff7-83ce-08c13bf394bf None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:41:24.667317 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c75bfbe2-293e-4ff7-83ce-08c13bf394bf None admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:41:24.686207 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 277/539] 10.222.0.30 () {56 vars in 1374 bytes} [Mon May 23 12:41:24 2022] GET /identity/v3/role_assignments?user.id=1666097f8ed545c6b0e4809c93e4775a&scope.project.id=9c389ae65aaa4542bb5a5c4fd0bacf5b&role.id=41ab28c1ae3d4941bd09fb5d35f1dc54 => generated 259 bytes in 224 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) May 23 12:41:24.740511 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf654f70-4870-4739-b75e-de0dd1c121bd None admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:41:24.777125 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-efcb299a-bc1f-46b7-95bc-1a09b412a587 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:41:24.778079 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efcb299a-bc1f-46b7-95bc-1a09b412a587 None admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:41:24.778364 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efcb299a-bc1f-46b7-95bc-1a09b412a587 None admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:41:24.778636 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efcb299a-bc1f-46b7-95bc-1a09b412a587 None admin] PATH_INFO: `/v3/projects` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:41:24.779425 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-efcb299a-bc1f-46b7-95bc-1a09b412a587 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=92057) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} May 23 12:41:24.779733 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-efcb299a-bc1f-46b7-95bc-1a09b412a587 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:41:24.781249 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-efcb299a-bc1f-46b7-95bc-1a09b412a587 None admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:41:24.802133 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 263/540] 10.222.0.30 () {56 vars in 1148 bytes} [Mon May 23 12:41:24 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 221 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) May 23 12:41:24.808521 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c75bfbe2-293e-4ff7-83ce-08c13bf394bf None admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:41:24.836631 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=100 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:41:25.017519 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c75bfbe2-293e-4ff7-83ce-08c13bf394bf None admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:41:25.060758 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2657b9c7-4cca-4b65-a8f0-02736864cd2b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:41:25.061459 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2657b9c7-4cca-4b65-a8f0-02736864cd2b None admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:41:25.061740 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2657b9c7-4cca-4b65-a8f0-02736864cd2b None admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:41:25.062041 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2657b9c7-4cca-4b65-a8f0-02736864cd2b None admin] PATH_INFO: `/v3/role_assignments` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:41:25.062939 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2657b9c7-4cca-4b65-a8f0-02736864cd2b None admin] RBAC: Adding query filter params (role.id=41ab28c1ae3d4941bd09fb5d35f1dc54, scope.project.id=9c389ae65aaa4542bb5a5c4fd0bacf5b, user.id=2eb83ca38e79449095f3a2f947bbb06a) {{(pid=92056) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} May 23 12:41:25.063282 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2657b9c7-4cca-4b65-a8f0-02736864cd2b None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:41:25.064798 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2657b9c7-4cca-4b65-a8f0-02736864cd2b None admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:41:25.078395 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 278/541] 10.222.0.30 () {56 vars in 1374 bytes} [Mon May 23 12:41:24 2022] GET /identity/v3/role_assignments?user.id=2eb83ca38e79449095f3a2f947bbb06a&scope.project.id=9c389ae65aaa4542bb5a5c4fd0bacf5b&role.id=41ab28c1ae3d4941bd09fb5d35f1dc54 => generated 259 bytes in 271 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) May 23 12:41:25.165626 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35fa5086-8b5c-4936-a658-08013e987365 None None] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:41:25.165626 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35fa5086-8b5c-4936-a658-08013e987365 None None] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:41:25.165626 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35fa5086-8b5c-4936-a658-08013e987365 None None] PATH_INFO: `/` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:41:25.166372 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 264/542] 10.222.0.30 () {54 vars in 926 bytes} [Mon May 23 12:41:25 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) May 23 12:41:25.172128 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3187f0e-0762-4de9-8a6d-89b14ddd68f2 None None] REQUEST_METHOD: `POST` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:41:25.172680 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3187f0e-0762-4de9-8a6d-89b14ddd68f2 None None] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:41:25.173151 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3187f0e-0762-4de9-8a6d-89b14ddd68f2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:41:25.199002 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.auth.core [None req-d3187f0e-0762-4de9-8a6d-89b14ddd68f2 None None] MFA Rules not processed for user `0f7119f8fa5c4445977671140ded91a4`. Rule list: `[]` (Enabled: `True`). {{(pid=92056) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 12:41:25.200562 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.fernet_utils [None req-d3187f0e-0762-4de9-8a6d-89b14ddd68f2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92056) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 12:41:25.300847 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 279/543] 10.222.0.30 () {58 vars in 1023 bytes} [Mon May 23 12:41:25 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 129 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) May 23 12:41:25.328791 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35fa5086-8b5c-4936-a658-08013e987365 None None] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:41:25.421599 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=100 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:41:25.484673 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35fa5086-8b5c-4936-a658-08013e987365 None None] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:41:25.532572 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f00b7a5-e7ff-48ac-bbbd-a74541bc9046 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:41:25.533374 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f00b7a5-e7ff-48ac-bbbd-a74541bc9046 None admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:41:25.533654 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f00b7a5-e7ff-48ac-bbbd-a74541bc9046 None admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:41:25.533949 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f00b7a5-e7ff-48ac-bbbd-a74541bc9046 None admin] PATH_INFO: `/v3/roles/service` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:41:25.548068 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f00b7a5-e7ff-48ac-bbbd-a74541bc9046 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} May 23 12:41:25.556152 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application [None req-5f00b7a5-e7ff-48ac-bbbd-a74541bc9046 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. May 23 12:41:25.556152 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application Traceback (most recent call last): May 23 12:41:25.556152 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request May 23 12:41:25.556152 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application rv = self.dispatch_request() May 23 12:41:25.556152 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request May 23 12:41:25.556152 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) May 23 12:41:25.556152 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper May 23 12:41:25.556152 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) May 23 12:41:25.556152 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view May 23 12:41:25.556152 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) May 23 12:41:25.556152 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request May 23 12:41:25.556152 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) May 23 12:41:25.556152 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get May 23 12:41:25.556152 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self._get_role(role_id) May 23 12:41:25.556152 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role May 23 12:41:25.556152 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') May 23 12:41:25.557057 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call May 23 12:41:25.557057 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( May 23 12:41:25.557057 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data May 23 12:41:25.557057 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) May 23 12:41:25.557057 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped May 23 12:41:25.557057 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) May 23 12:41:25.557057 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun May 23 12:41:25.557057 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) May 23 12:41:25.557057 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func May 23 12:41:25.557057 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self.get_or_create( May 23 12:41:25.557057 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create May 23 12:41:25.557057 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application with Lock( May 23 12:41:25.557057 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ May 23 12:41:25.557057 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self._enter() May 23 12:41:25.557057 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter May 23 12:41:25.557057 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) May 23 12:41:25.557856 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create May 23 12:41:25.557856 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self.creator() May 23 12:41:25.557856 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value May 23 12:41:25.557856 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application created_value = creator( May 23 12:41:25.557856 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role May 23 12:41:25.557856 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) May 23 12:41:25.557856 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role May 23 12:41:25.557856 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() May 23 12:41:25.557856 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role May 23 12:41:25.557856 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) May 23 12:41:25.557856 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. May 23 12:41:25.557856 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application May 23 12:41:25.557856 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 265/544] 10.222.0.30 () {56 vars in 1102 bytes} [Mon May 23 12:41:25 2022] GET /identity/v3/roles/service => generated 85 bytes in 229 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) May 23 12:41:25.563350 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3187f0e-0762-4de9-8a6d-89b14ddd68f2 None None] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:41:25.572953 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=100 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:41:25.685914 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3187f0e-0762-4de9-8a6d-89b14ddd68f2 None None] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:41:25.725399 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-460394b1-61b8-4df5-919f-5d2170221e39 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:41:25.726627 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-460394b1-61b8-4df5-919f-5d2170221e39 None admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:41:25.726909 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-460394b1-61b8-4df5-919f-5d2170221e39 None admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:41:25.727181 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-460394b1-61b8-4df5-919f-5d2170221e39 None admin] PATH_INFO: `/v3/roles` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:41:25.727942 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-460394b1-61b8-4df5-919f-5d2170221e39 None admin] RBAC: Adding query filter params (name=service) {{(pid=92056) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} May 23 12:41:25.728244 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-460394b1-61b8-4df5-919f-5d2170221e39 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:41:25.729621 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-460394b1-61b8-4df5-919f-5d2170221e39 None admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:41:25.738408 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 280/545] 10.222.0.30 () {56 vars in 1103 bytes} [Mon May 23 12:41:25 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 176 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) May 23 12:41:25.744459 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f00b7a5-e7ff-48ac-bbbd-a74541bc9046 None admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:41:25.874830 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f00b7a5-e7ff-48ac-bbbd-a74541bc9046 None admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:41:25.896737 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=100 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:41:25.922794 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0137787-d588-4bc1-8b4e-6551ed6d6261 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:41:25.923611 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0137787-d588-4bc1-8b4e-6551ed6d6261 None admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:41:25.923903 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0137787-d588-4bc1-8b4e-6551ed6d6261 None admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:41:25.924173 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0137787-d588-4bc1-8b4e-6551ed6d6261 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:41:25.940643 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0137787-d588-4bc1-8b4e-6551ed6d6261 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:41:25.942203 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0137787-d588-4bc1-8b4e-6551ed6d6261 None admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:41:25.969709 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application [None req-c0137787-d588-4bc1-8b4e-6551ed6d6261 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. May 23 12:41:25.969709 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application Traceback (most recent call last): May 23 12:41:25.969709 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request May 23 12:41:25.969709 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application rv = self.dispatch_request() May 23 12:41:25.969709 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request May 23 12:41:25.969709 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) May 23 12:41:25.969709 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper May 23 12:41:25.969709 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) May 23 12:41:25.969709 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view May 23 12:41:25.969709 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) May 23 12:41:25.969709 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request May 23 12:41:25.969709 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) May 23 12:41:25.969709 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get May 23 12:41:25.969709 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self._get_domain(domain_id) May 23 12:41:25.969709 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain May 23 12:41:25.970660 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) May 23 12:41:25.970660 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped May 23 12:41:25.970660 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) May 23 12:41:25.970660 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun May 23 12:41:25.970660 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) May 23 12:41:25.970660 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func May 23 12:41:25.970660 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self.get_or_create( May 23 12:41:25.970660 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create May 23 12:41:25.970660 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application with Lock( May 23 12:41:25.970660 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ May 23 12:41:25.970660 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self._enter() May 23 12:41:25.970660 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter May 23 12:41:25.970660 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) May 23 12:41:25.970660 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create May 23 12:41:25.970660 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self.creator() May 23 12:41:25.970660 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value May 23 12:41:25.970660 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application created_value = creator( May 23 12:41:25.971534 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain May 23 12:41:25.971534 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) May 23 12:41:25.971534 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. May 23 12:41:25.971534 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application May 23 12:41:25.971534 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 266/546] 10.222.0.30 () {56 vars in 1108 bytes} [Mon May 23 12:41:25 2022] GET /identity/v3/domains/Default => generated 87 bytes in 227 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) May 23 12:41:25.976844 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-460394b1-61b8-4df5-919f-5d2170221e39 None admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:41:26.057043 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=100 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:41:26.115665 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-460394b1-61b8-4df5-919f-5d2170221e39 None admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:41:26.125490 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6040c56-3101-427c-90fa-1d6aae2ea45e None None] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:41:26.125733 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6040c56-3101-427c-90fa-1d6aae2ea45e None None] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:41:26.126009 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6040c56-3101-427c-90fa-1d6aae2ea45e None None] PATH_INFO: `/` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:41:26.127386 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 267/547] 10.222.0.30 () {54 vars in 926 bytes} [Mon May 23 12:41:26 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) May 23 12:41:26.133247 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9782469-19b2-443d-b90c-cb54c8551f2e None None] REQUEST_METHOD: `POST` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:41:26.133530 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9782469-19b2-443d-b90c-cb54c8551f2e None None] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:41:26.133805 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9782469-19b2-443d-b90c-cb54c8551f2e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:41:26.164804 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.auth.core [None req-b9782469-19b2-443d-b90c-cb54c8551f2e None None] MFA Rules not processed for user `0f7119f8fa5c4445977671140ded91a4`. Rule list: `[]` (Enabled: `True`). {{(pid=92057) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 12:41:26.166304 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.fernet_utils [None req-b9782469-19b2-443d-b90c-cb54c8551f2e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92057) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 12:41:26.169971 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4aac952f-30d5-4f6e-8f8d-e8900ead1bd7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:41:26.170639 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4aac952f-30d5-4f6e-8f8d-e8900ead1bd7 None admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:41:26.170916 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4aac952f-30d5-4f6e-8f8d-e8900ead1bd7 None admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:41:26.171185 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4aac952f-30d5-4f6e-8f8d-e8900ead1bd7 None admin] PATH_INFO: `/v3/domains` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:41:26.171904 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4aac952f-30d5-4f6e-8f8d-e8900ead1bd7 None admin] RBAC: Adding query filter params (name=Default) {{(pid=92056) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} May 23 12:41:26.172204 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4aac952f-30d5-4f6e-8f8d-e8900ead1bd7 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:41:26.173568 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4aac952f-30d5-4f6e-8f8d-e8900ead1bd7 None admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:41:26.195940 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 281/548] 10.222.0.30 () {56 vars in 1109 bytes} [Mon May 23 12:41:25 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 218 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) May 23 12:41:26.199260 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4aac952f-30d5-4f6e-8f8d-e8900ead1bd7 None admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:41:26.292783 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 268/549] 10.222.0.30 () {58 vars in 1023 bytes} [Mon May 23 12:41:26 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 161 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) May 23 12:41:26.294724 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d37e2e72-8a14-4e63-a72d-1e635028d206 None None] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:41:26.295024 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d37e2e72-8a14-4e63-a72d-1e635028d206 None None] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:41:26.295304 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d37e2e72-8a14-4e63-a72d-1e635028d206 None None] PATH_INFO: `/` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:41:26.296818 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 269/550] 10.222.0.30 () {54 vars in 926 bytes} [Mon May 23 12:41:26 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) May 23 12:41:26.302172 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c954e4df-45a3-4028-9f29-26d28df643f4 None None] REQUEST_METHOD: `POST` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:41:26.302627 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c954e4df-45a3-4028-9f29-26d28df643f4 None None] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:41:26.302967 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c954e4df-45a3-4028-9f29-26d28df643f4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:41:26.325758 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.auth.core [None req-c954e4df-45a3-4028-9f29-26d28df643f4 None None] MFA Rules not processed for user `0f7119f8fa5c4445977671140ded91a4`. Rule list: `[]` (Enabled: `True`). {{(pid=92057) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 12:41:26.330659 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.fernet_utils [None req-c954e4df-45a3-4028-9f29-26d28df643f4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92057) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 12:41:26.374372 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4aac952f-30d5-4f6e-8f8d-e8900ead1bd7 None admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:41:26.403816 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=100 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:41:26.407718 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 270/551] 10.222.0.30 () {58 vars in 1023 bytes} [Mon May 23 12:41:26 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 106 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) May 23 12:41:26.408619 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c954e4df-45a3-4028-9f29-26d28df643f4 None None] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:41:26.419763 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d8c4725-2817-4592-9daf-4b6242506546 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:41:26.420696 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d8c4725-2817-4592-9daf-4b6242506546 None admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:41:26.420988 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d8c4725-2817-4592-9daf-4b6242506546 None admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:41:26.421264 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d8c4725-2817-4592-9daf-4b6242506546 None admin] PATH_INFO: `/v3/users/swift` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:41:26.431473 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d8c4725-2817-4592-9daf-4b6242506546 None admin] RBAC: Authorizing `identity:get_user(user_id=swift)` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:41:26.432980 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d8c4725-2817-4592-9daf-4b6242506546 None admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:41:26.445954 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=100 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:41:26.463567 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application [None req-2d8c4725-2817-4592-9daf-4b6242506546 None admin] Could not find user: swift.: keystone.exception.UserNotFound: Could not find user: swift. May 23 12:41:26.463567 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application Traceback (most recent call last): May 23 12:41:26.463567 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request May 23 12:41:26.463567 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application rv = self.dispatch_request() May 23 12:41:26.463567 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request May 23 12:41:26.463567 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) May 23 12:41:26.463567 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper May 23 12:41:26.463567 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) May 23 12:41:26.463567 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view May 23 12:41:26.463567 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) May 23 12:41:26.463567 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request May 23 12:41:26.463567 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) May 23 12:41:26.463567 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get May 23 12:41:26.463567 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self._get_user(user_id) May 23 12:41:26.463567 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user May 23 12:41:26.463567 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) May 23 12:41:26.466021 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped May 23 12:41:26.466021 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) May 23 12:41:26.466021 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper May 23 12:41:26.466021 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) May 23 12:41:26.466021 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper May 23 12:41:26.466021 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) May 23 12:41:26.466021 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun May 23 12:41:26.466021 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) May 23 12:41:26.466021 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func May 23 12:41:26.466021 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self.get_or_create( May 23 12:41:26.466021 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create May 23 12:41:26.466021 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application with Lock( May 23 12:41:26.466021 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ May 23 12:41:26.466021 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self._enter() May 23 12:41:26.466021 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter May 23 12:41:26.466021 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) May 23 12:41:26.466021 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create May 23 12:41:26.466021 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self.creator() May 23 12:41:26.467022 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value May 23 12:41:26.467022 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application created_value = creator( May 23 12:41:26.467022 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user May 23 12:41:26.467022 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) May 23 12:41:26.467022 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user May 23 12:41:26.467022 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) May 23 12:41:26.467022 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user May 23 12:41:26.467022 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) May 23 12:41:26.467022 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: swift. May 23 12:41:26.467022 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application May 23 12:41:26.472523 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 282/552] 10.222.0.30 () {56 vars in 1096 bytes} [Mon May 23 12:41:26 2022] GET /identity/v3/users/swift => generated 83 bytes in 269 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) May 23 12:41:26.473553 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d8c4725-2817-4592-9daf-4b6242506546 None admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:41:26.515386 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c954e4df-45a3-4028-9f29-26d28df643f4 None None] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:41:26.553707 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b3368a3-f1f2-49a0-a692-b5ed0240ae17 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:41:26.555680 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b3368a3-f1f2-49a0-a692-b5ed0240ae17 None admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:41:26.555993 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b3368a3-f1f2-49a0-a692-b5ed0240ae17 None admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:41:26.556269 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b3368a3-f1f2-49a0-a692-b5ed0240ae17 None admin] PATH_INFO: `/v3/roles/service` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:41:26.569098 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b3368a3-f1f2-49a0-a692-b5ed0240ae17 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} May 23 12:41:26.579508 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application [None req-1b3368a3-f1f2-49a0-a692-b5ed0240ae17 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. May 23 12:41:26.579508 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application Traceback (most recent call last): May 23 12:41:26.579508 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request May 23 12:41:26.579508 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application rv = self.dispatch_request() May 23 12:41:26.579508 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request May 23 12:41:26.579508 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) May 23 12:41:26.579508 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper May 23 12:41:26.579508 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) May 23 12:41:26.579508 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view May 23 12:41:26.579508 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) May 23 12:41:26.579508 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request May 23 12:41:26.579508 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) May 23 12:41:26.579508 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get May 23 12:41:26.579508 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self._get_role(role_id) May 23 12:41:26.579508 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role May 23 12:41:26.579508 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') May 23 12:41:26.580527 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call May 23 12:41:26.580527 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( May 23 12:41:26.580527 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data May 23 12:41:26.580527 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) May 23 12:41:26.580527 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped May 23 12:41:26.580527 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) May 23 12:41:26.580527 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun May 23 12:41:26.580527 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) May 23 12:41:26.580527 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func May 23 12:41:26.580527 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self.get_or_create( May 23 12:41:26.580527 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create May 23 12:41:26.580527 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application with Lock( May 23 12:41:26.580527 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ May 23 12:41:26.580527 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self._enter() May 23 12:41:26.580527 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter May 23 12:41:26.580527 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) May 23 12:41:26.581424 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create May 23 12:41:26.581424 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self.creator() May 23 12:41:26.581424 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value May 23 12:41:26.581424 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application created_value = creator( May 23 12:41:26.581424 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role May 23 12:41:26.581424 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) May 23 12:41:26.581424 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role May 23 12:41:26.581424 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() May 23 12:41:26.581424 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role May 23 12:41:26.581424 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) May 23 12:41:26.581424 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. May 23 12:41:26.581424 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application May 23 12:41:26.581424 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 271/553] 10.222.0.30 () {56 vars in 1102 bytes} [Mon May 23 12:41:26 2022] GET /identity/v3/roles/service => generated 85 bytes in 172 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) May 23 12:41:26.588116 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b3368a3-f1f2-49a0-a692-b5ed0240ae17 None admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:41:26.594188 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d8c4725-2817-4592-9daf-4b6242506546 None admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:41:26.637242 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d43e2a70-ea21-4534-937d-7aed1b5b013d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:41:26.638121 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d43e2a70-ea21-4534-937d-7aed1b5b013d None admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:41:26.638426 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d43e2a70-ea21-4534-937d-7aed1b5b013d None admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:41:26.638697 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d43e2a70-ea21-4534-937d-7aed1b5b013d None admin] PATH_INFO: `/v3/roles/service` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:41:26.651107 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d43e2a70-ea21-4534-937d-7aed1b5b013d None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} May 23 12:41:26.659980 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application [None req-d43e2a70-ea21-4534-937d-7aed1b5b013d None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. May 23 12:41:26.659980 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application Traceback (most recent call last): May 23 12:41:26.659980 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request May 23 12:41:26.659980 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application rv = self.dispatch_request() May 23 12:41:26.659980 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request May 23 12:41:26.659980 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) May 23 12:41:26.659980 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper May 23 12:41:26.659980 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) May 23 12:41:26.659980 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view May 23 12:41:26.659980 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) May 23 12:41:26.659980 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request May 23 12:41:26.659980 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) May 23 12:41:26.659980 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get May 23 12:41:26.659980 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self._get_role(role_id) May 23 12:41:26.659980 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role May 23 12:41:26.659980 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') May 23 12:41:26.660904 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call May 23 12:41:26.660904 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( May 23 12:41:26.660904 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data May 23 12:41:26.660904 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) May 23 12:41:26.660904 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped May 23 12:41:26.660904 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) May 23 12:41:26.660904 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun May 23 12:41:26.660904 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) May 23 12:41:26.660904 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func May 23 12:41:26.660904 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self.get_or_create( May 23 12:41:26.660904 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create May 23 12:41:26.660904 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application with Lock( May 23 12:41:26.660904 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ May 23 12:41:26.660904 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self._enter() May 23 12:41:26.660904 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter May 23 12:41:26.660904 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) May 23 12:41:26.662186 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create May 23 12:41:26.662186 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self.creator() May 23 12:41:26.662186 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value May 23 12:41:26.662186 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application created_value = creator( May 23 12:41:26.662186 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role May 23 12:41:26.662186 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) May 23 12:41:26.662186 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role May 23 12:41:26.662186 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() May 23 12:41:26.662186 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role May 23 12:41:26.662186 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) May 23 12:41:26.662186 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. May 23 12:41:26.662186 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application May 23 12:41:26.666027 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 283/554] 10.222.0.30 () {56 vars in 1102 bytes} [Mon May 23 12:41:26 2022] GET /identity/v3/roles/service => generated 85 bytes in 188 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) May 23 12:41:26.667020 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d43e2a70-ea21-4534-937d-7aed1b5b013d None admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:41:26.699836 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=100 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:41:26.723707 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b3368a3-f1f2-49a0-a692-b5ed0240ae17 None admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:41:26.732399 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=100 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:41:26.767767 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45b8af6f-32c2-4447-98cb-03901d1b8ecf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:41:26.768811 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45b8af6f-32c2-4447-98cb-03901d1b8ecf None admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:41:26.769259 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45b8af6f-32c2-4447-98cb-03901d1b8ecf None admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:41:26.769822 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45b8af6f-32c2-4447-98cb-03901d1b8ecf None admin] PATH_INFO: `/v3/users` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:41:26.771207 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45b8af6f-32c2-4447-98cb-03901d1b8ecf None admin] RBAC: Adding query filter params (domain_id=default, name=swift) {{(pid=92057) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} May 23 12:41:26.771674 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45b8af6f-32c2-4447-98cb-03901d1b8ecf None admin] RBAC: Authorizing `identity:list_users()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:41:26.774318 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45b8af6f-32c2-4447-98cb-03901d1b8ecf None admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:41:26.793438 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d43e2a70-ea21-4534-937d-7aed1b5b013d None admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:41:26.798138 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 272/555] 10.222.0.30 () {56 vars in 1135 bytes} [Mon May 23 12:41:26 2022] GET /identity/v3/users?domain_id=default&name=swift => generated 370 bytes in 217 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) May 23 12:41:26.799214 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45b8af6f-32c2-4447-98cb-03901d1b8ecf None admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:41:26.836511 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f5b41a9-fb75-42aa-8afd-7cc417167781 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:41:26.837621 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f5b41a9-fb75-42aa-8afd-7cc417167781 None admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:41:26.837840 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f5b41a9-fb75-42aa-8afd-7cc417167781 None admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:41:26.838189 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f5b41a9-fb75-42aa-8afd-7cc417167781 None admin] PATH_INFO: `/v3/roles` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:41:26.839061 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f5b41a9-fb75-42aa-8afd-7cc417167781 None admin] RBAC: Adding query filter params (name=service) {{(pid=92056) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} May 23 12:41:26.839517 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f5b41a9-fb75-42aa-8afd-7cc417167781 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:41:26.840995 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f5b41a9-fb75-42aa-8afd-7cc417167781 None admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:41:26.855457 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 284/556] 10.222.0.30 () {56 vars in 1103 bytes} [Mon May 23 12:41:26 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 184 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) May 23 12:41:26.855457 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f5b41a9-fb75-42aa-8afd-7cc417167781 None admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:41:26.937819 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45b8af6f-32c2-4447-98cb-03901d1b8ecf None admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:41:26.968453 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f5b41a9-fb75-42aa-8afd-7cc417167781 None admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:41:26.980526 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=100 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:41:26.984686 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18750eb9-d8e9-45df-8b55-e187b4245a5b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:41:26.985712 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18750eb9-d8e9-45df-8b55-e187b4245a5b None admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:41:26.986560 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18750eb9-d8e9-45df-8b55-e187b4245a5b None admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:41:26.992037 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18750eb9-d8e9-45df-8b55-e187b4245a5b None admin] PATH_INFO: `/v3/roles` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:41:26.992037 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18750eb9-d8e9-45df-8b55-e187b4245a5b None admin] RBAC: Adding query filter params (name=service) {{(pid=92057) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} May 23 12:41:26.992037 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18750eb9-d8e9-45df-8b55-e187b4245a5b None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:41:26.992037 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18750eb9-d8e9-45df-8b55-e187b4245a5b None admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:41:27.002878 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 273/557] 10.222.0.30 () {56 vars in 1103 bytes} [Mon May 23 12:41:26 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 204 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) May 23 12:41:27.004005 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18750eb9-d8e9-45df-8b55-e187b4245a5b None admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:41:27.008557 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=100 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:41:27.020368 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6297f916-8f61-401e-a9c0-e1a25bb92092 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:41:27.023400 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6297f916-8f61-401e-a9c0-e1a25bb92092 None admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:41:27.023400 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6297f916-8f61-401e-a9c0-e1a25bb92092 None admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:41:27.023400 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6297f916-8f61-401e-a9c0-e1a25bb92092 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:41:27.116145 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18750eb9-d8e9-45df-8b55-e187b4245a5b None admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:41:27.119531 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6297f916-8f61-401e-a9c0-e1a25bb92092 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:41:27.121230 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6297f916-8f61-401e-a9c0-e1a25bb92092 None admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:41:27.160541 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a0d0cb4-f51a-40f3-a701-2e3954855b5a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:41:27.161855 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a0d0cb4-f51a-40f3-a701-2e3954855b5a None admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:41:27.162372 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a0d0cb4-f51a-40f3-a701-2e3954855b5a None admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:41:27.163520 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a0d0cb4-f51a-40f3-a701-2e3954855b5a None admin] PATH_INFO: `/v3/domains/Default` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:41:27.178041 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a0d0cb4-f51a-40f3-a701-2e3954855b5a None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:41:27.180964 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a0d0cb4-f51a-40f3-a701-2e3954855b5a None admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:41:27.205636 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application [None req-6a0d0cb4-f51a-40f3-a701-2e3954855b5a None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. May 23 12:41:27.205636 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application Traceback (most recent call last): May 23 12:41:27.205636 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request May 23 12:41:27.205636 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application rv = self.dispatch_request() May 23 12:41:27.205636 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request May 23 12:41:27.205636 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) May 23 12:41:27.205636 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper May 23 12:41:27.205636 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) May 23 12:41:27.205636 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view May 23 12:41:27.205636 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) May 23 12:41:27.205636 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request May 23 12:41:27.205636 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) May 23 12:41:27.205636 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get May 23 12:41:27.205636 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self._get_domain(domain_id) May 23 12:41:27.205636 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain May 23 12:41:27.206810 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) May 23 12:41:27.206810 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped May 23 12:41:27.206810 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) May 23 12:41:27.206810 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun May 23 12:41:27.206810 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) May 23 12:41:27.206810 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func May 23 12:41:27.206810 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self.get_or_create( May 23 12:41:27.206810 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create May 23 12:41:27.206810 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application with Lock( May 23 12:41:27.206810 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ May 23 12:41:27.206810 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self._enter() May 23 12:41:27.206810 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter May 23 12:41:27.206810 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) May 23 12:41:27.206810 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create May 23 12:41:27.206810 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self.creator() May 23 12:41:27.206810 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value May 23 12:41:27.206810 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application created_value = creator( May 23 12:41:27.207896 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain May 23 12:41:27.207896 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) May 23 12:41:27.207896 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. May 23 12:41:27.207896 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application May 23 12:41:27.211610 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 274/558] 10.222.0.30 () {56 vars in 1108 bytes} [Mon May 23 12:41:27 2022] GET /identity/v3/domains/Default => generated 87 bytes in 208 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) May 23 12:41:27.212582 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a0d0cb4-f51a-40f3-a701-2e3954855b5a None admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:41:27.252250 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 285/559] 10.222.0.30 () {58 vars in 1282 bytes} [Mon May 23 12:41:26 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 400 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) May 23 12:41:27.253427 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6297f916-8f61-401e-a9c0-e1a25bb92092 None admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:41:27.267653 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=100 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:41:27.294072 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=100 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:41:27.356350 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a0d0cb4-f51a-40f3-a701-2e3954855b5a None admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:41:27.367908 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6297f916-8f61-401e-a9c0-e1a25bb92092 None admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:41:27.407048 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe380c7a-d12d-4c4f-bfcd-8c33a5f6fce2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:41:27.407387 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0ac2efe-187f-47ef-9d2b-ccc0971f7501 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:41:27.407963 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe380c7a-d12d-4c4f-bfcd-8c33a5f6fce2 None admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:41:27.408371 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe380c7a-d12d-4c4f-bfcd-8c33a5f6fce2 None admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:41:27.408683 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0ac2efe-187f-47ef-9d2b-ccc0971f7501 None admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:41:27.408796 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe380c7a-d12d-4c4f-bfcd-8c33a5f6fce2 None admin] PATH_INFO: `/v3/domains` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:41:27.409211 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0ac2efe-187f-47ef-9d2b-ccc0971f7501 None admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:41:27.409633 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0ac2efe-187f-47ef-9d2b-ccc0971f7501 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:41:27.409991 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe380c7a-d12d-4c4f-bfcd-8c33a5f6fce2 None admin] RBAC: Adding query filter params (name=Default) {{(pid=92056) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} May 23 12:41:27.410404 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe380c7a-d12d-4c4f-bfcd-8c33a5f6fce2 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:41:27.411870 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe380c7a-d12d-4c4f-bfcd-8c33a5f6fce2 None admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:41:27.423662 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0ac2efe-187f-47ef-9d2b-ccc0971f7501 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:41:27.424480 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 286/560] 10.222.0.30 () {56 vars in 1109 bytes} [Mon May 23 12:41:27 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 172 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) May 23 12:41:27.425653 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0ac2efe-187f-47ef-9d2b-ccc0971f7501 None admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:41:27.430717 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe380c7a-d12d-4c4f-bfcd-8c33a5f6fce2 None admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:41:27.448884 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application [None req-a0ac2efe-187f-47ef-9d2b-ccc0971f7501 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. May 23 12:41:27.448884 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application Traceback (most recent call last): May 23 12:41:27.448884 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request May 23 12:41:27.448884 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application rv = self.dispatch_request() May 23 12:41:27.448884 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request May 23 12:41:27.448884 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) May 23 12:41:27.448884 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper May 23 12:41:27.448884 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) May 23 12:41:27.448884 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view May 23 12:41:27.448884 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) May 23 12:41:27.448884 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request May 23 12:41:27.448884 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) May 23 12:41:27.448884 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get May 23 12:41:27.448884 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self._get_domain(domain_id) May 23 12:41:27.448884 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain May 23 12:41:27.450035 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) May 23 12:41:27.450035 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped May 23 12:41:27.450035 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) May 23 12:41:27.450035 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun May 23 12:41:27.450035 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) May 23 12:41:27.450035 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func May 23 12:41:27.450035 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self.get_or_create( May 23 12:41:27.450035 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create May 23 12:41:27.450035 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application with Lock( May 23 12:41:27.450035 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ May 23 12:41:27.450035 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self._enter() May 23 12:41:27.450035 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter May 23 12:41:27.450035 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) May 23 12:41:27.450035 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create May 23 12:41:27.450035 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self.creator() May 23 12:41:27.450035 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value May 23 12:41:27.450035 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application created_value = creator( May 23 12:41:27.451148 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain May 23 12:41:27.451148 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) May 23 12:41:27.451148 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. May 23 12:41:27.451148 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application May 23 12:41:27.453896 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 275/561] 10.222.0.30 () {56 vars in 1108 bytes} [Mon May 23 12:41:27 2022] GET /identity/v3/domains/Default => generated 87 bytes in 241 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) May 23 12:41:27.454894 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0ac2efe-187f-47ef-9d2b-ccc0971f7501 None admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:41:27.515587 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=100 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:41:27.549975 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe380c7a-d12d-4c4f-bfcd-8c33a5f6fce2 None admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:41:27.570667 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0ac2efe-187f-47ef-9d2b-ccc0971f7501 None admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:41:27.581768 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=100 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:41:27.588275 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff313693-9de9-409c-b7b2-f05ef92fb71a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:41:27.589134 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff313693-9de9-409c-b7b2-f05ef92fb71a None admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:41:27.589419 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff313693-9de9-409c-b7b2-f05ef92fb71a None admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:41:27.589695 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff313693-9de9-409c-b7b2-f05ef92fb71a None admin] PATH_INFO: `/v3/domains/Default` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:41:27.600422 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff313693-9de9-409c-b7b2-f05ef92fb71a None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:41:27.603178 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff313693-9de9-409c-b7b2-f05ef92fb71a None admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:41:27.611892 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ac1614f-026f-46c6-9dfb-568963839fce None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:41:27.613156 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ac1614f-026f-46c6-9dfb-568963839fce None admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:41:27.613640 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ac1614f-026f-46c6-9dfb-568963839fce None admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:41:27.614128 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ac1614f-026f-46c6-9dfb-568963839fce None admin] PATH_INFO: `/v3/users/nova` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:41:27.621188 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application [None req-ff313693-9de9-409c-b7b2-f05ef92fb71a None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. May 23 12:41:27.621188 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application Traceback (most recent call last): May 23 12:41:27.621188 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request May 23 12:41:27.621188 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application rv = self.dispatch_request() May 23 12:41:27.621188 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request May 23 12:41:27.621188 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) May 23 12:41:27.621188 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper May 23 12:41:27.621188 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) May 23 12:41:27.621188 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view May 23 12:41:27.621188 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) May 23 12:41:27.621188 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request May 23 12:41:27.621188 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) May 23 12:41:27.621188 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get May 23 12:41:27.621188 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self._get_domain(domain_id) May 23 12:41:27.621188 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain May 23 12:41:27.622247 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) May 23 12:41:27.622247 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped May 23 12:41:27.622247 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) May 23 12:41:27.622247 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun May 23 12:41:27.622247 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) May 23 12:41:27.622247 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func May 23 12:41:27.622247 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self.get_or_create( May 23 12:41:27.622247 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create May 23 12:41:27.622247 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application with Lock( May 23 12:41:27.622247 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ May 23 12:41:27.622247 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self._enter() May 23 12:41:27.622247 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter May 23 12:41:27.622247 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) May 23 12:41:27.622247 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create May 23 12:41:27.622247 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self.creator() May 23 12:41:27.622247 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value May 23 12:41:27.622247 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application created_value = creator( May 23 12:41:27.623173 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain May 23 12:41:27.623173 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) May 23 12:41:27.623173 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. May 23 12:41:27.623173 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application May 23 12:41:27.623173 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 287/562] 10.222.0.30 () {56 vars in 1108 bytes} [Mon May 23 12:41:27 2022] GET /identity/v3/domains/Default => generated 87 bytes in 197 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) May 23 12:41:27.623488 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff313693-9de9-409c-b7b2-f05ef92fb71a None admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:41:27.629098 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ac1614f-026f-46c6-9dfb-568963839fce None admin] RBAC: Authorizing `identity:get_user(user_id=nova)` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:41:27.630811 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ac1614f-026f-46c6-9dfb-568963839fce None admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:41:27.650895 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application [None req-7ac1614f-026f-46c6-9dfb-568963839fce None admin] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova. May 23 12:41:27.650895 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application Traceback (most recent call last): May 23 12:41:27.650895 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request May 23 12:41:27.650895 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application rv = self.dispatch_request() May 23 12:41:27.650895 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request May 23 12:41:27.650895 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) May 23 12:41:27.650895 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper May 23 12:41:27.650895 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) May 23 12:41:27.650895 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view May 23 12:41:27.650895 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) May 23 12:41:27.650895 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request May 23 12:41:27.650895 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) May 23 12:41:27.650895 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get May 23 12:41:27.650895 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self._get_user(user_id) May 23 12:41:27.650895 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user May 23 12:41:27.650895 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) May 23 12:41:27.651981 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped May 23 12:41:27.651981 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) May 23 12:41:27.651981 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper May 23 12:41:27.651981 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) May 23 12:41:27.651981 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper May 23 12:41:27.651981 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) May 23 12:41:27.651981 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun May 23 12:41:27.651981 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) May 23 12:41:27.651981 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func May 23 12:41:27.651981 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self.get_or_create( May 23 12:41:27.651981 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create May 23 12:41:27.651981 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application with Lock( May 23 12:41:27.651981 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ May 23 12:41:27.651981 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self._enter() May 23 12:41:27.651981 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter May 23 12:41:27.651981 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) May 23 12:41:27.651981 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create May 23 12:41:27.651981 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self.creator() May 23 12:41:27.653227 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value May 23 12:41:27.653227 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application created_value = creator( May 23 12:41:27.653227 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user May 23 12:41:27.653227 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) May 23 12:41:27.653227 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user May 23 12:41:27.653227 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) May 23 12:41:27.653227 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user May 23 12:41:27.653227 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) May 23 12:41:27.653227 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: nova. May 23 12:41:27.653227 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application May 23 12:41:27.653227 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 276/563] 10.222.0.30 () {56 vars in 1093 bytes} [Mon May 23 12:41:27 2022] GET /identity/v3/users/nova => generated 82 bytes in 199 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) May 23 12:41:27.655703 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a93b3d94-3c6c-42a5-b80e-f1bea8ed86ab None None] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:41:27.655703 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a93b3d94-3c6c-42a5-b80e-f1bea8ed86ab None None] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:41:27.655703 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a93b3d94-3c6c-42a5-b80e-f1bea8ed86ab None None] PATH_INFO: `/` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:41:27.661189 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 277/564] 10.222.0.30 () {54 vars in 926 bytes} [Mon May 23 12:41:27 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) May 23 12:41:27.661189 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a93b3d94-3c6c-42a5-b80e-f1bea8ed86ab None None] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:41:27.744766 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff313693-9de9-409c-b7b2-f05ef92fb71a None admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:41:27.767762 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=100 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:41:27.776405 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a93b3d94-3c6c-42a5-b80e-f1bea8ed86ab None None] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:41:27.780633 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5d9daa7-b124-42d7-a69d-502c2d8c53ad None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:41:27.781319 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5d9daa7-b124-42d7-a69d-502c2d8c53ad None admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:41:27.781604 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5d9daa7-b124-42d7-a69d-502c2d8c53ad None admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:41:27.781900 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5d9daa7-b124-42d7-a69d-502c2d8c53ad None admin] PATH_INFO: `/v3/domains` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:41:27.782627 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5d9daa7-b124-42d7-a69d-502c2d8c53ad None admin] RBAC: Adding query filter params (name=Default) {{(pid=92056) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} May 23 12:41:27.782933 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5d9daa7-b124-42d7-a69d-502c2d8c53ad None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:41:27.784343 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5d9daa7-b124-42d7-a69d-502c2d8c53ad None admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:41:27.797380 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 288/565] 10.222.0.30 () {56 vars in 1109 bytes} [Mon May 23 12:41:27 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 174 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) May 23 12:41:27.799883 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5d9daa7-b124-42d7-a69d-502c2d8c53ad None admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:41:27.814005 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3c26660-b23a-4043-83e7-2ed938eebaf0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:41:27.814675 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3c26660-b23a-4043-83e7-2ed938eebaf0 None admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:41:27.814959 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3c26660-b23a-4043-83e7-2ed938eebaf0 None admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:41:27.815238 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3c26660-b23a-4043-83e7-2ed938eebaf0 None admin] PATH_INFO: `/v3/domains` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:41:27.815958 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3c26660-b23a-4043-83e7-2ed938eebaf0 None admin] RBAC: Adding query filter params (name=Default) {{(pid=92057) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} May 23 12:41:27.816266 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3c26660-b23a-4043-83e7-2ed938eebaf0 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:41:27.817639 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3c26660-b23a-4043-83e7-2ed938eebaf0 None admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:41:27.831530 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 278/566] 10.222.0.30 () {56 vars in 1109 bytes} [Mon May 23 12:41:27 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 174 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) May 23 12:41:27.832828 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44aa2f79-a845-4c7b-80fd-6200a933f2a4 None None] REQUEST_METHOD: `POST` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:41:27.833122 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44aa2f79-a845-4c7b-80fd-6200a933f2a4 None None] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:41:27.833399 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44aa2f79-a845-4c7b-80fd-6200a933f2a4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:41:27.851691 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.auth.core [None req-44aa2f79-a845-4c7b-80fd-6200a933f2a4 None None] MFA Rules not processed for user `0f7119f8fa5c4445977671140ded91a4`. Rule list: `[]` (Enabled: `True`). {{(pid=92057) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 12:41:27.852996 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.fernet_utils [None req-44aa2f79-a845-4c7b-80fd-6200a933f2a4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92057) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 12:41:27.876424 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=100 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:41:27.910268 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5d9daa7-b124-42d7-a69d-502c2d8c53ad None admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:41:27.919787 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 279/567] 10.222.0.30 () {58 vars in 1023 bytes} [Mon May 23 12:41:27 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 88 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) May 23 12:41:27.920610 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44aa2f79-a845-4c7b-80fd-6200a933f2a4 None None] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:41:27.945780 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c919707-4dfb-420c-a83e-706456238263 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:41:27.946478 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c919707-4dfb-420c-a83e-706456238263 None admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:41:27.946759 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c919707-4dfb-420c-a83e-706456238263 None admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:41:27.947034 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c919707-4dfb-420c-a83e-706456238263 None admin] PATH_INFO: `/v3/users` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:41:27.947870 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c919707-4dfb-420c-a83e-706456238263 None admin] RBAC: Adding query filter params (domain_id=default, name=nova) {{(pid=92056) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} May 23 12:41:27.948176 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c919707-4dfb-420c-a83e-706456238263 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:41:27.949658 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c919707-4dfb-420c-a83e-706456238263 None admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:41:27.966857 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 289/568] 10.222.0.30 () {56 vars in 1133 bytes} [Mon May 23 12:41:27 2022] GET /identity/v3/users?domain_id=default&name=nova => generated 368 bytes in 168 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) May 23 12:41:27.967658 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c919707-4dfb-420c-a83e-706456238263 None admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:41:28.011754 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=100 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:41:28.032426 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44aa2f79-a845-4c7b-80fd-6200a933f2a4 None None] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:41:28.066206 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54933be6-f8f9-4585-a5be-941772935578 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:41:28.066993 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54933be6-f8f9-4585-a5be-941772935578 None admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:41:28.067279 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54933be6-f8f9-4585-a5be-941772935578 None admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:41:28.067553 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54933be6-f8f9-4585-a5be-941772935578 None admin] PATH_INFO: `/v3/users/glance` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:41:28.069210 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c919707-4dfb-420c-a83e-706456238263 None admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:41:28.074743 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54933be6-f8f9-4585-a5be-941772935578 None admin] RBAC: Authorizing `identity:get_user(user_id=glance)` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:41:28.076198 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54933be6-f8f9-4585-a5be-941772935578 None admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:41:28.091123 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application [None req-54933be6-f8f9-4585-a5be-941772935578 None admin] Could not find user: glance.: keystone.exception.UserNotFound: Could not find user: glance. May 23 12:41:28.091123 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application Traceback (most recent call last): May 23 12:41:28.091123 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request May 23 12:41:28.091123 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application rv = self.dispatch_request() May 23 12:41:28.091123 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request May 23 12:41:28.091123 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) May 23 12:41:28.091123 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper May 23 12:41:28.091123 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) May 23 12:41:28.091123 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view May 23 12:41:28.091123 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) May 23 12:41:28.091123 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request May 23 12:41:28.091123 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) May 23 12:41:28.091123 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get May 23 12:41:28.091123 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self._get_user(user_id) May 23 12:41:28.091123 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user May 23 12:41:28.091123 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) May 23 12:41:28.092653 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped May 23 12:41:28.092653 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) May 23 12:41:28.092653 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper May 23 12:41:28.092653 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) May 23 12:41:28.092653 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper May 23 12:41:28.092653 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) May 23 12:41:28.092653 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun May 23 12:41:28.092653 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) May 23 12:41:28.092653 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func May 23 12:41:28.092653 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self.get_or_create( May 23 12:41:28.092653 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create May 23 12:41:28.092653 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application with Lock( May 23 12:41:28.092653 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ May 23 12:41:28.092653 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self._enter() May 23 12:41:28.092653 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter May 23 12:41:28.092653 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) May 23 12:41:28.092653 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create May 23 12:41:28.092653 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self.creator() May 23 12:41:28.093690 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value May 23 12:41:28.093690 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application created_value = creator( May 23 12:41:28.093690 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user May 23 12:41:28.093690 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) May 23 12:41:28.093690 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user May 23 12:41:28.093690 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) May 23 12:41:28.093690 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user May 23 12:41:28.093690 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) May 23 12:41:28.093690 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: glance. May 23 12:41:28.093690 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application May 23 12:41:28.093690 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 280/569] 10.222.0.30 () {56 vars in 1099 bytes} [Mon May 23 12:41:27 2022] GET /identity/v3/users/glance => generated 84 bytes in 172 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) May 23 12:41:28.098030 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-770b0df4-e84d-47da-80ea-6453b44ce62f None None] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:41:28.098322 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-770b0df4-e84d-47da-80ea-6453b44ce62f None None] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:41:28.098604 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-770b0df4-e84d-47da-80ea-6453b44ce62f None None] PATH_INFO: `/` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:41:28.099880 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 281/570] 10.222.0.30 () {54 vars in 926 bytes} [Mon May 23 12:41:28 2022] GET /identity => generated 270 bytes in 7 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) May 23 12:41:28.100614 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-770b0df4-e84d-47da-80ea-6453b44ce62f None None] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:41:28.104956 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b28690dc-18c4-4552-8c5c-c42cc7cb913a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:41:28.105748 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b28690dc-18c4-4552-8c5c-c42cc7cb913a None admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:41:28.106064 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b28690dc-18c4-4552-8c5c-c42cc7cb913a None admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:41:28.106342 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b28690dc-18c4-4552-8c5c-c42cc7cb913a None admin] PATH_INFO: `/v3/projects/service` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:41:28.113255 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b28690dc-18c4-4552-8c5c-c42cc7cb913a None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:41:28.114734 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b28690dc-18c4-4552-8c5c-c42cc7cb913a None admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:41:28.121383 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=100 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:41:28.128285 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application [None req-b28690dc-18c4-4552-8c5c-c42cc7cb913a None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. May 23 12:41:28.128285 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application Traceback (most recent call last): May 23 12:41:28.128285 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request May 23 12:41:28.128285 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application rv = self.dispatch_request() May 23 12:41:28.128285 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request May 23 12:41:28.128285 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) May 23 12:41:28.128285 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper May 23 12:41:28.128285 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) May 23 12:41:28.128285 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view May 23 12:41:28.128285 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) May 23 12:41:28.128285 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request May 23 12:41:28.128285 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) May 23 12:41:28.128285 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get May 23 12:41:28.128285 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self._get_project(project_id) May 23 12:41:28.128285 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project May 23 12:41:28.132648 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) May 23 12:41:28.132648 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped May 23 12:41:28.132648 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) May 23 12:41:28.132648 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun May 23 12:41:28.132648 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) May 23 12:41:28.132648 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func May 23 12:41:28.132648 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self.get_or_create( May 23 12:41:28.132648 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create May 23 12:41:28.132648 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application with Lock( May 23 12:41:28.132648 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ May 23 12:41:28.132648 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self._enter() May 23 12:41:28.132648 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter May 23 12:41:28.132648 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) May 23 12:41:28.132648 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create May 23 12:41:28.132648 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self.creator() May 23 12:41:28.132648 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value May 23 12:41:28.132648 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application created_value = creator( May 23 12:41:28.133766 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project May 23 12:41:28.133766 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) May 23 12:41:28.133766 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project May 23 12:41:28.133766 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() May 23 12:41:28.133766 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project May 23 12:41:28.133766 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) May 23 12:41:28.133766 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. May 23 12:41:28.133766 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application May 23 12:41:28.133766 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 290/571] 10.222.0.30 () {56 vars in 1111 bytes} [Mon May 23 12:41:27 2022] GET /identity/v3/projects/service => generated 88 bytes in 162 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) May 23 12:41:28.133766 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b28690dc-18c4-4552-8c5c-c42cc7cb913a None admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:41:28.203867 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-770b0df4-e84d-47da-80ea-6453b44ce62f None None] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:41:28.231017 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=100 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:41:28.234166 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b28690dc-18c4-4552-8c5c-c42cc7cb913a None admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:41:28.238558 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0573600-e863-4518-93bd-4a0a2c8c475a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:41:28.239351 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0573600-e863-4518-93bd-4a0a2c8c475a None admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:41:28.240514 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0573600-e863-4518-93bd-4a0a2c8c475a None admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:41:28.240514 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0573600-e863-4518-93bd-4a0a2c8c475a None admin] PATH_INFO: `/v3/roles/service` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:41:28.250830 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0573600-e863-4518-93bd-4a0a2c8c475a None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} May 23 12:41:28.258543 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application [None req-d0573600-e863-4518-93bd-4a0a2c8c475a None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. May 23 12:41:28.258543 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application Traceback (most recent call last): May 23 12:41:28.258543 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request May 23 12:41:28.258543 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application rv = self.dispatch_request() May 23 12:41:28.258543 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request May 23 12:41:28.258543 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) May 23 12:41:28.258543 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper May 23 12:41:28.258543 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) May 23 12:41:28.258543 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view May 23 12:41:28.258543 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) May 23 12:41:28.258543 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request May 23 12:41:28.258543 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) May 23 12:41:28.258543 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get May 23 12:41:28.258543 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self._get_role(role_id) May 23 12:41:28.258543 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role May 23 12:41:28.258543 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') May 23 12:41:28.259577 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call May 23 12:41:28.259577 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( May 23 12:41:28.259577 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data May 23 12:41:28.259577 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) May 23 12:41:28.259577 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped May 23 12:41:28.259577 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) May 23 12:41:28.259577 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun May 23 12:41:28.259577 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) May 23 12:41:28.259577 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func May 23 12:41:28.259577 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self.get_or_create( May 23 12:41:28.259577 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create May 23 12:41:28.259577 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application with Lock( May 23 12:41:28.259577 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ May 23 12:41:28.259577 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self._enter() May 23 12:41:28.259577 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter May 23 12:41:28.259577 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) May 23 12:41:28.260461 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create May 23 12:41:28.260461 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self.creator() May 23 12:41:28.260461 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value May 23 12:41:28.260461 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application created_value = creator( May 23 12:41:28.260461 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role May 23 12:41:28.260461 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) May 23 12:41:28.260461 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role May 23 12:41:28.260461 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() May 23 12:41:28.260461 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role May 23 12:41:28.260461 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) May 23 12:41:28.260461 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. May 23 12:41:28.260461 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application May 23 12:41:28.260461 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 282/572] 10.222.0.30 () {56 vars in 1102 bytes} [Mon May 23 12:41:28 2022] GET /identity/v3/roles/service => generated 85 bytes in 159 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) May 23 12:41:28.261188 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0573600-e863-4518-93bd-4a0a2c8c475a None admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:41:28.271445 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-914e6098-a048-47d1-8b39-7fca72cb2bc4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:41:28.272240 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-914e6098-a048-47d1-8b39-7fca72cb2bc4 None admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:41:28.273115 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-914e6098-a048-47d1-8b39-7fca72cb2bc4 None admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:41:28.273392 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-914e6098-a048-47d1-8b39-7fca72cb2bc4 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:41:28.284308 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-914e6098-a048-47d1-8b39-7fca72cb2bc4 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:41:28.285689 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-914e6098-a048-47d1-8b39-7fca72cb2bc4 None admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:41:28.303480 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application [None req-914e6098-a048-47d1-8b39-7fca72cb2bc4 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. May 23 12:41:28.303480 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application Traceback (most recent call last): May 23 12:41:28.303480 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request May 23 12:41:28.303480 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application rv = self.dispatch_request() May 23 12:41:28.303480 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request May 23 12:41:28.303480 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) May 23 12:41:28.303480 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper May 23 12:41:28.303480 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) May 23 12:41:28.303480 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view May 23 12:41:28.303480 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) May 23 12:41:28.303480 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request May 23 12:41:28.303480 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) May 23 12:41:28.303480 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get May 23 12:41:28.303480 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self._get_domain(domain_id) May 23 12:41:28.303480 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain May 23 12:41:28.304464 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) May 23 12:41:28.304464 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped May 23 12:41:28.304464 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) May 23 12:41:28.304464 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun May 23 12:41:28.304464 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) May 23 12:41:28.304464 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func May 23 12:41:28.304464 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self.get_or_create( May 23 12:41:28.304464 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create May 23 12:41:28.304464 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application with Lock( May 23 12:41:28.304464 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ May 23 12:41:28.304464 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self._enter() May 23 12:41:28.304464 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter May 23 12:41:28.304464 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) May 23 12:41:28.304464 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create May 23 12:41:28.304464 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self.creator() May 23 12:41:28.304464 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value May 23 12:41:28.304464 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application created_value = creator( May 23 12:41:28.305394 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain May 23 12:41:28.305394 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) May 23 12:41:28.305394 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. May 23 12:41:28.305394 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application May 23 12:41:28.305394 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 291/573] 10.222.0.30 () {56 vars in 1108 bytes} [Mon May 23 12:41:28 2022] GET /identity/v3/domains/Default => generated 87 bytes in 175 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) May 23 12:41:28.306291 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9b562ee-8784-4fe5-98e9-086b15930c76 None None] REQUEST_METHOD: `POST` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:41:28.306572 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9b562ee-8784-4fe5-98e9-086b15930c76 None None] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:41:28.306844 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9b562ee-8784-4fe5-98e9-086b15930c76 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:41:28.323153 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.auth.core [None req-c9b562ee-8784-4fe5-98e9-086b15930c76 None None] MFA Rules not processed for user `0f7119f8fa5c4445977671140ded91a4`. Rule list: `[]` (Enabled: `True`). {{(pid=92056) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 12:41:28.324367 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.fernet_utils [None req-c9b562ee-8784-4fe5-98e9-086b15930c76 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92056) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 12:41:28.352873 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=100 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:41:28.374012 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0573600-e863-4518-93bd-4a0a2c8c475a None admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:41:28.391807 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 292/574] 10.222.0.30 () {58 vars in 1023 bytes} [Mon May 23 12:41:28 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) May 23 12:41:28.392653 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9b562ee-8784-4fe5-98e9-086b15930c76 None None] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:41:28.420664 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17e4b8fc-c682-4377-a0df-8c1cc78d715b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:41:28.421353 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17e4b8fc-c682-4377-a0df-8c1cc78d715b None admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:41:28.421628 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17e4b8fc-c682-4377-a0df-8c1cc78d715b None admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:41:28.421926 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17e4b8fc-c682-4377-a0df-8c1cc78d715b None admin] PATH_INFO: `/v3/users` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:41:28.422761 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17e4b8fc-c682-4377-a0df-8c1cc78d715b None admin] RBAC: Adding query filter params (domain_id=default, name=glance) {{(pid=92057) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} May 23 12:41:28.423077 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17e4b8fc-c682-4377-a0df-8c1cc78d715b None admin] RBAC: Authorizing `identity:list_users()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:41:28.424611 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17e4b8fc-c682-4377-a0df-8c1cc78d715b None admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:41:28.442133 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 283/575] 10.222.0.30 () {56 vars in 1137 bytes} [Mon May 23 12:41:28 2022] GET /identity/v3/users?domain_id=default&name=glance => generated 372 bytes in 182 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) May 23 12:41:28.442871 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17e4b8fc-c682-4377-a0df-8c1cc78d715b None admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:41:28.481339 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=100 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:41:28.499417 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9b562ee-8784-4fe5-98e9-086b15930c76 None None] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:41:28.533681 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-361aef70-375d-4ff8-b1f2-4c7e6958333c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:41:28.534426 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-361aef70-375d-4ff8-b1f2-4c7e6958333c None admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:41:28.534651 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-361aef70-375d-4ff8-b1f2-4c7e6958333c None admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:41:28.534921 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-361aef70-375d-4ff8-b1f2-4c7e6958333c None admin] PATH_INFO: `/v3/projects` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:41:28.535695 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-361aef70-375d-4ff8-b1f2-4c7e6958333c None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=92056) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} May 23 12:41:28.536092 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-361aef70-375d-4ff8-b1f2-4c7e6958333c None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:41:28.537535 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-361aef70-375d-4ff8-b1f2-4c7e6958333c None admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:41:28.544110 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17e4b8fc-c682-4377-a0df-8c1cc78d715b None admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:41:28.548575 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 293/576] 10.222.0.30 () {56 vars in 1148 bytes} [Mon May 23 12:41:28 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 156 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) May 23 12:41:28.549413 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-361aef70-375d-4ff8-b1f2-4c7e6958333c None admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:41:28.581581 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4780fc9-3af5-4e0d-ae45-2830c149234f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:41:28.582290 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4780fc9-3af5-4e0d-ae45-2830c149234f None admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:41:28.582581 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4780fc9-3af5-4e0d-ae45-2830c149234f None admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:41:28.582860 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4780fc9-3af5-4e0d-ae45-2830c149234f None admin] PATH_INFO: `/v3/roles` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:41:28.583600 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f4780fc9-3af5-4e0d-ae45-2830c149234f None admin] RBAC: Adding query filter params (name=service) {{(pid=92057) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} May 23 12:41:28.583908 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f4780fc9-3af5-4e0d-ae45-2830c149234f None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:41:28.585840 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f4780fc9-3af5-4e0d-ae45-2830c149234f None admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:41:28.588003 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=100 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:41:28.594056 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 284/577] 10.222.0.30 () {56 vars in 1103 bytes} [Mon May 23 12:41:28 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 151 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) May 23 12:41:28.594846 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4780fc9-3af5-4e0d-ae45-2830c149234f None admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:41:28.650364 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-361aef70-375d-4ff8-b1f2-4c7e6958333c None admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:41:28.682898 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e1f959e-1e91-494b-a610-caac2ad18b1b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:41:28.683555 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e1f959e-1e91-494b-a610-caac2ad18b1b None admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:41:28.683838 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e1f959e-1e91-494b-a610-caac2ad18b1b None admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:41:28.684108 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e1f959e-1e91-494b-a610-caac2ad18b1b None admin] PATH_INFO: `/v3/domains` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:41:28.684820 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e1f959e-1e91-494b-a610-caac2ad18b1b None admin] RBAC: Adding query filter params (name=Default) {{(pid=92056) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} May 23 12:41:28.685121 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e1f959e-1e91-494b-a610-caac2ad18b1b None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:41:28.686525 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e1f959e-1e91-494b-a610-caac2ad18b1b None admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:41:28.699009 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4780fc9-3af5-4e0d-ae45-2830c149234f None admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:41:28.699421 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 294/578] 10.222.0.30 () {56 vars in 1109 bytes} [Mon May 23 12:41:28 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 150 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) May 23 12:41:28.700230 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e1f959e-1e91-494b-a610-caac2ad18b1b None admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:41:28.702861 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=100 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:41:28.738093 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-077cbe3b-e496-4918-a42f-7a0f709f64ca None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:41:28.738884 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-077cbe3b-e496-4918-a42f-7a0f709f64ca None admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:41:28.739167 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-077cbe3b-e496-4918-a42f-7a0f709f64ca None admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:41:28.739438 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-077cbe3b-e496-4918-a42f-7a0f709f64ca None admin] PATH_INFO: `/v3/roles/service` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:41:28.750058 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-077cbe3b-e496-4918-a42f-7a0f709f64ca None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} May 23 12:41:28.758424 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application [None req-077cbe3b-e496-4918-a42f-7a0f709f64ca None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. May 23 12:41:28.758424 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application Traceback (most recent call last): May 23 12:41:28.758424 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request May 23 12:41:28.758424 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application rv = self.dispatch_request() May 23 12:41:28.758424 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request May 23 12:41:28.758424 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) May 23 12:41:28.758424 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper May 23 12:41:28.758424 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) May 23 12:41:28.758424 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view May 23 12:41:28.758424 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) May 23 12:41:28.758424 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request May 23 12:41:28.758424 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) May 23 12:41:28.758424 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get May 23 12:41:28.758424 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self._get_role(role_id) May 23 12:41:28.758424 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role May 23 12:41:28.758424 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') May 23 12:41:28.759420 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call May 23 12:41:28.759420 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( May 23 12:41:28.759420 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data May 23 12:41:28.759420 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) May 23 12:41:28.759420 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped May 23 12:41:28.759420 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) May 23 12:41:28.759420 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun May 23 12:41:28.759420 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) May 23 12:41:28.759420 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func May 23 12:41:28.759420 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self.get_or_create( May 23 12:41:28.759420 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create May 23 12:41:28.759420 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application with Lock( May 23 12:41:28.759420 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ May 23 12:41:28.759420 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self._enter() May 23 12:41:28.759420 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter May 23 12:41:28.759420 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) May 23 12:41:28.760281 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create May 23 12:41:28.760281 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self.creator() May 23 12:41:28.760281 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value May 23 12:41:28.760281 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application created_value = creator( May 23 12:41:28.760281 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role May 23 12:41:28.760281 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) May 23 12:41:28.760281 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role May 23 12:41:28.760281 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() May 23 12:41:28.760281 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role May 23 12:41:28.760281 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) May 23 12:41:28.760281 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. May 23 12:41:28.760281 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application May 23 12:41:28.761238 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 285/579] 10.222.0.30 () {56 vars in 1102 bytes} [Mon May 23 12:41:28 2022] GET /identity/v3/roles/service => generated 85 bytes in 167 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) May 23 12:41:28.762221 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-077cbe3b-e496-4918-a42f-7a0f709f64ca None admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:41:28.806066 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e1f959e-1e91-494b-a610-caac2ad18b1b None admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:41:28.822523 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=100 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:41:28.839973 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04737bd9-2ebe-46d8-8fdd-b02bbfaa01fc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:41:28.840759 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04737bd9-2ebe-46d8-8fdd-b02bbfaa01fc None admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:41:28.841045 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04737bd9-2ebe-46d8-8fdd-b02bbfaa01fc None admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:41:28.841323 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04737bd9-2ebe-46d8-8fdd-b02bbfaa01fc None admin] PATH_INFO: `/v3/domains/Default` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:41:28.851131 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04737bd9-2ebe-46d8-8fdd-b02bbfaa01fc None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:41:28.852569 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04737bd9-2ebe-46d8-8fdd-b02bbfaa01fc None admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:41:28.863150 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-077cbe3b-e496-4918-a42f-7a0f709f64ca None admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:41:28.870368 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application [None req-04737bd9-2ebe-46d8-8fdd-b02bbfaa01fc None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. May 23 12:41:28.870368 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application Traceback (most recent call last): May 23 12:41:28.870368 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request May 23 12:41:28.870368 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application rv = self.dispatch_request() May 23 12:41:28.870368 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request May 23 12:41:28.870368 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) May 23 12:41:28.870368 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper May 23 12:41:28.870368 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) May 23 12:41:28.870368 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view May 23 12:41:28.870368 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) May 23 12:41:28.870368 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request May 23 12:41:28.870368 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) May 23 12:41:28.870368 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get May 23 12:41:28.870368 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self._get_domain(domain_id) May 23 12:41:28.870368 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain May 23 12:41:28.871270 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) May 23 12:41:28.871270 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped May 23 12:41:28.871270 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) May 23 12:41:28.871270 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun May 23 12:41:28.871270 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) May 23 12:41:28.871270 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func May 23 12:41:28.871270 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self.get_or_create( May 23 12:41:28.871270 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create May 23 12:41:28.871270 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application with Lock( May 23 12:41:28.871270 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ May 23 12:41:28.871270 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self._enter() May 23 12:41:28.871270 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter May 23 12:41:28.871270 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) May 23 12:41:28.871270 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create May 23 12:41:28.871270 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self.creator() May 23 12:41:28.871270 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value May 23 12:41:28.871270 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application created_value = creator( May 23 12:41:28.872198 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain May 23 12:41:28.872198 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) May 23 12:41:28.872198 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. May 23 12:41:28.872198 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application May 23 12:41:28.872198 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 295/580] 10.222.0.30 () {56 vars in 1108 bytes} [Mon May 23 12:41:28 2022] GET /identity/v3/domains/Default => generated 87 bytes in 172 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) May 23 12:41:28.872491 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04737bd9-2ebe-46d8-8fdd-b02bbfaa01fc None admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:41:28.901505 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bfaf5f71-cef9-4f90-8889-d66dd4c448fc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:41:28.902188 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfaf5f71-cef9-4f90-8889-d66dd4c448fc None admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:41:28.902476 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfaf5f71-cef9-4f90-8889-d66dd4c448fc None admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:41:28.903002 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfaf5f71-cef9-4f90-8889-d66dd4c448fc None admin] PATH_INFO: `/v3/role_assignments` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:41:28.903827 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bfaf5f71-cef9-4f90-8889-d66dd4c448fc None admin] RBAC: Adding query filter params (role.id=41ab28c1ae3d4941bd09fb5d35f1dc54, scope.project.id=9c389ae65aaa4542bb5a5c4fd0bacf5b, user.id=138d91edb99f450e846765f9eef00d62) {{(pid=92057) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} May 23 12:41:28.904140 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bfaf5f71-cef9-4f90-8889-d66dd4c448fc None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:41:28.905571 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bfaf5f71-cef9-4f90-8889-d66dd4c448fc None admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:41:28.911375 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 286/581] 10.222.0.30 () {56 vars in 1374 bytes} [Mon May 23 12:41:28 2022] GET /identity/v3/role_assignments?user.id=138d91edb99f450e846765f9eef00d62&scope.project.id=9c389ae65aaa4542bb5a5c4fd0bacf5b&role.id=41ab28c1ae3d4941bd09fb5d35f1dc54 => generated 259 bytes in 149 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) May 23 12:41:28.912149 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bfaf5f71-cef9-4f90-8889-d66dd4c448fc None admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:41:28.934339 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=100 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:41:28.976276 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04737bd9-2ebe-46d8-8fdd-b02bbfaa01fc None admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:41:29.019989 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e06aebb-1396-4641-bc8b-4a7e4755c7fd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:41:29.020845 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e06aebb-1396-4641-bc8b-4a7e4755c7fd None admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:41:29.021132 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e06aebb-1396-4641-bc8b-4a7e4755c7fd None admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:41:29.021633 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e06aebb-1396-4641-bc8b-4a7e4755c7fd None admin] PATH_INFO: `/v3/domains/Default` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:41:29.024489 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bfaf5f71-cef9-4f90-8889-d66dd4c448fc None admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:41:29.033309 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e06aebb-1396-4641-bc8b-4a7e4755c7fd None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:41:29.034865 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e06aebb-1396-4641-bc8b-4a7e4755c7fd None admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:41:29.051027 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=100 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:41:29.053430 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application [None req-7e06aebb-1396-4641-bc8b-4a7e4755c7fd None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. May 23 12:41:29.053430 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application Traceback (most recent call last): May 23 12:41:29.053430 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request May 23 12:41:29.053430 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application rv = self.dispatch_request() May 23 12:41:29.053430 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request May 23 12:41:29.053430 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) May 23 12:41:29.053430 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper May 23 12:41:29.053430 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) May 23 12:41:29.053430 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view May 23 12:41:29.053430 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) May 23 12:41:29.053430 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request May 23 12:41:29.053430 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) May 23 12:41:29.053430 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get May 23 12:41:29.053430 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self._get_domain(domain_id) May 23 12:41:29.053430 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain May 23 12:41:29.055322 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) May 23 12:41:29.055322 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped May 23 12:41:29.055322 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) May 23 12:41:29.055322 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun May 23 12:41:29.055322 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) May 23 12:41:29.055322 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func May 23 12:41:29.055322 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self.get_or_create( May 23 12:41:29.055322 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create May 23 12:41:29.055322 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application with Lock( May 23 12:41:29.055322 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ May 23 12:41:29.055322 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self._enter() May 23 12:41:29.055322 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter May 23 12:41:29.055322 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) May 23 12:41:29.055322 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create May 23 12:41:29.055322 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self.creator() May 23 12:41:29.055322 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value May 23 12:41:29.055322 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application created_value = creator( May 23 12:41:29.056331 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain May 23 12:41:29.056331 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) May 23 12:41:29.056331 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. May 23 12:41:29.056331 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application May 23 12:41:29.062543 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 296/582] 10.222.0.30 () {56 vars in 1108 bytes} [Mon May 23 12:41:28 2022] GET /identity/v3/domains/Default => generated 87 bytes in 185 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) May 23 12:41:29.062543 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e06aebb-1396-4641-bc8b-4a7e4755c7fd None admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:41:29.076428 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c0ad1ba-c68b-45fb-b182-8a85d1833f9b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:41:29.077311 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c0ad1ba-c68b-45fb-b182-8a85d1833f9b None admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:41:29.077598 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c0ad1ba-c68b-45fb-b182-8a85d1833f9b None admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:41:29.077900 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c0ad1ba-c68b-45fb-b182-8a85d1833f9b None admin] PATH_INFO: `/v3/projects/service` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:41:29.086313 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c0ad1ba-c68b-45fb-b182-8a85d1833f9b None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:41:29.087800 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c0ad1ba-c68b-45fb-b182-8a85d1833f9b None admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:41:29.105632 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application [None req-0c0ad1ba-c68b-45fb-b182-8a85d1833f9b None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. May 23 12:41:29.105632 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application Traceback (most recent call last): May 23 12:41:29.105632 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request May 23 12:41:29.105632 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application rv = self.dispatch_request() May 23 12:41:29.105632 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request May 23 12:41:29.105632 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) May 23 12:41:29.105632 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper May 23 12:41:29.105632 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) May 23 12:41:29.105632 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view May 23 12:41:29.105632 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) May 23 12:41:29.105632 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request May 23 12:41:29.105632 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) May 23 12:41:29.105632 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get May 23 12:41:29.105632 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self._get_project(project_id) May 23 12:41:29.105632 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project May 23 12:41:29.106639 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) May 23 12:41:29.106639 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped May 23 12:41:29.106639 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) May 23 12:41:29.106639 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun May 23 12:41:29.106639 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) May 23 12:41:29.106639 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func May 23 12:41:29.106639 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self.get_or_create( May 23 12:41:29.106639 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create May 23 12:41:29.106639 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application with Lock( May 23 12:41:29.106639 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ May 23 12:41:29.106639 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self._enter() May 23 12:41:29.106639 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter May 23 12:41:29.106639 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) May 23 12:41:29.106639 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create May 23 12:41:29.106639 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self.creator() May 23 12:41:29.106639 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value May 23 12:41:29.106639 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application created_value = creator( May 23 12:41:29.107574 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project May 23 12:41:29.107574 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) May 23 12:41:29.107574 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project May 23 12:41:29.107574 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() May 23 12:41:29.107574 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project May 23 12:41:29.107574 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) May 23 12:41:29.107574 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. May 23 12:41:29.107574 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application May 23 12:41:29.107574 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 287/583] 10.222.0.30 () {56 vars in 1111 bytes} [Mon May 23 12:41:28 2022] GET /identity/v3/projects/service => generated 88 bytes in 195 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) May 23 12:41:29.108109 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c0ad1ba-c68b-45fb-b182-8a85d1833f9b None admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:41:29.177250 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e06aebb-1396-4641-bc8b-4a7e4755c7fd None admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:41:29.186361 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=100 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:41:29.213085 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c0ad1ba-c68b-45fb-b182-8a85d1833f9b None admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:41:29.213341 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c05b05dd-e547-4ceb-b74c-713564fc5b5c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:41:29.214169 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c05b05dd-e547-4ceb-b74c-713564fc5b5c None admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:41:29.214499 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c05b05dd-e547-4ceb-b74c-713564fc5b5c None admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:41:29.214797 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c05b05dd-e547-4ceb-b74c-713564fc5b5c None admin] PATH_INFO: `/v3/roles` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:41:29.215711 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c05b05dd-e547-4ceb-b74c-713564fc5b5c None admin] RBAC: Adding query filter params (name=service) {{(pid=92056) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} May 23 12:41:29.216036 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c05b05dd-e547-4ceb-b74c-713564fc5b5c None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:41:29.217903 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c05b05dd-e547-4ceb-b74c-713564fc5b5c None admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:41:29.231700 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 297/584] 10.222.0.30 () {56 vars in 1103 bytes} [Mon May 23 12:41:29 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 169 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) May 23 12:41:29.231700 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c05b05dd-e547-4ceb-b74c-713564fc5b5c None admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:41:29.257918 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f903259-285a-4f22-b9ee-b7c43dfe6c35 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:41:29.258846 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f903259-285a-4f22-b9ee-b7c43dfe6c35 None admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:41:29.259128 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f903259-285a-4f22-b9ee-b7c43dfe6c35 None admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:41:29.259402 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f903259-285a-4f22-b9ee-b7c43dfe6c35 None admin] PATH_INFO: `/v3/domains` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:41:29.260152 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f903259-285a-4f22-b9ee-b7c43dfe6c35 None admin] RBAC: Adding query filter params (name=Default) {{(pid=92057) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} May 23 12:41:29.260484 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f903259-285a-4f22-b9ee-b7c43dfe6c35 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:41:29.261864 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f903259-285a-4f22-b9ee-b7c43dfe6c35 None admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:41:29.279216 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 288/585] 10.222.0.30 () {56 vars in 1109 bytes} [Mon May 23 12:41:29 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 167 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) May 23 12:41:29.279216 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f903259-285a-4f22-b9ee-b7c43dfe6c35 None admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:41:29.323592 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=100 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:41:29.344309 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c05b05dd-e547-4ceb-b74c-713564fc5b5c None admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:41:29.384033 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eeffaac3-c935-402a-904f-8061cd21fa90 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:41:29.384726 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eeffaac3-c935-402a-904f-8061cd21fa90 None admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:41:29.385013 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eeffaac3-c935-402a-904f-8061cd21fa90 None admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:41:29.385296 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eeffaac3-c935-402a-904f-8061cd21fa90 None admin] PATH_INFO: `/v3/domains` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:41:29.386063 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eeffaac3-c935-402a-904f-8061cd21fa90 None admin] RBAC: Adding query filter params (name=Default) {{(pid=92056) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} May 23 12:41:29.386376 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eeffaac3-c935-402a-904f-8061cd21fa90 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:41:29.387757 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eeffaac3-c935-402a-904f-8061cd21fa90 None admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:41:29.388129 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f903259-285a-4f22-b9ee-b7c43dfe6c35 None admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:41:29.401510 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 298/586] 10.222.0.30 () {56 vars in 1109 bytes} [Mon May 23 12:41:29 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 174 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) May 23 12:41:29.402380 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eeffaac3-c935-402a-904f-8061cd21fa90 None admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:41:29.430385 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b43b2604-58bf-463e-8acc-79bd9dbf2a0e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:41:29.431488 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b43b2604-58bf-463e-8acc-79bd9dbf2a0e None admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:41:29.432016 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=100 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:41:29.432476 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b43b2604-58bf-463e-8acc-79bd9dbf2a0e None admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:41:29.433334 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b43b2604-58bf-463e-8acc-79bd9dbf2a0e None admin] PATH_INFO: `/v3/projects` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:41:29.434400 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b43b2604-58bf-463e-8acc-79bd9dbf2a0e None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=92057) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} May 23 12:41:29.434809 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b43b2604-58bf-463e-8acc-79bd9dbf2a0e None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:41:29.436253 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b43b2604-58bf-463e-8acc-79bd9dbf2a0e None admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:41:29.448844 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 289/587] 10.222.0.30 () {56 vars in 1148 bytes} [Mon May 23 12:41:29 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 174 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) May 23 12:41:29.449729 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b43b2604-58bf-463e-8acc-79bd9dbf2a0e None admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:41:29.513258 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eeffaac3-c935-402a-904f-8061cd21fa90 None admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:41:29.549733 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b646395d-3a21-4894-a57a-474d10a9049b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:41:29.550558 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b646395d-3a21-4894-a57a-474d10a9049b None admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:41:29.550843 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b646395d-3a21-4894-a57a-474d10a9049b None admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:41:29.551123 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b646395d-3a21-4894-a57a-474d10a9049b None admin] PATH_INFO: `/v3/domains/Default` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:41:29.561031 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b646395d-3a21-4894-a57a-474d10a9049b None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:41:29.562523 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b646395d-3a21-4894-a57a-474d10a9049b None admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:41:29.564408 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b43b2604-58bf-463e-8acc-79bd9dbf2a0e None admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:41:29.566096 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=100 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:41:29.580437 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application [None req-b646395d-3a21-4894-a57a-474d10a9049b None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. May 23 12:41:29.580437 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application Traceback (most recent call last): May 23 12:41:29.580437 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request May 23 12:41:29.580437 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application rv = self.dispatch_request() May 23 12:41:29.580437 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request May 23 12:41:29.580437 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) May 23 12:41:29.580437 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper May 23 12:41:29.580437 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) May 23 12:41:29.580437 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view May 23 12:41:29.580437 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) May 23 12:41:29.580437 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request May 23 12:41:29.580437 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) May 23 12:41:29.580437 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get May 23 12:41:29.580437 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self._get_domain(domain_id) May 23 12:41:29.580437 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain May 23 12:41:29.581369 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) May 23 12:41:29.581369 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped May 23 12:41:29.581369 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) May 23 12:41:29.581369 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun May 23 12:41:29.581369 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) May 23 12:41:29.581369 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func May 23 12:41:29.581369 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self.get_or_create( May 23 12:41:29.581369 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create May 23 12:41:29.581369 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application with Lock( May 23 12:41:29.581369 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ May 23 12:41:29.581369 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self._enter() May 23 12:41:29.581369 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter May 23 12:41:29.581369 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) May 23 12:41:29.581369 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create May 23 12:41:29.581369 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self.creator() May 23 12:41:29.581369 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value May 23 12:41:29.581369 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application created_value = creator( May 23 12:41:29.582323 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain May 23 12:41:29.582323 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) May 23 12:41:29.582323 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. May 23 12:41:29.582323 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application May 23 12:41:29.582323 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 299/588] 10.222.0.30 () {56 vars in 1108 bytes} [Mon May 23 12:41:29 2022] GET /identity/v3/domains/Default => generated 87 bytes in 180 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) May 23 12:41:29.582608 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b646395d-3a21-4894-a57a-474d10a9049b None admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:41:29.609230 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd770897-70c3-4c6f-bec7-9213c6023372 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:41:29.610488 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd770897-70c3-4c6f-bec7-9213c6023372 None admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:41:29.610965 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd770897-70c3-4c6f-bec7-9213c6023372 None admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:41:29.611419 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd770897-70c3-4c6f-bec7-9213c6023372 None admin] PATH_INFO: `/v3/projects/service` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:41:29.619168 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd770897-70c3-4c6f-bec7-9213c6023372 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:41:29.620890 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd770897-70c3-4c6f-bec7-9213c6023372 None admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:41:29.635719 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application [None req-dd770897-70c3-4c6f-bec7-9213c6023372 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. May 23 12:41:29.635719 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application Traceback (most recent call last): May 23 12:41:29.635719 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request May 23 12:41:29.635719 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application rv = self.dispatch_request() May 23 12:41:29.635719 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request May 23 12:41:29.635719 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) May 23 12:41:29.635719 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper May 23 12:41:29.635719 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) May 23 12:41:29.635719 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view May 23 12:41:29.635719 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) May 23 12:41:29.635719 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request May 23 12:41:29.635719 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) May 23 12:41:29.635719 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get May 23 12:41:29.635719 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self._get_project(project_id) May 23 12:41:29.635719 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project May 23 12:41:29.636658 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) May 23 12:41:29.636658 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped May 23 12:41:29.636658 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) May 23 12:41:29.636658 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun May 23 12:41:29.636658 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) May 23 12:41:29.636658 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func May 23 12:41:29.636658 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self.get_or_create( May 23 12:41:29.636658 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create May 23 12:41:29.636658 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application with Lock( May 23 12:41:29.636658 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ May 23 12:41:29.636658 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self._enter() May 23 12:41:29.636658 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter May 23 12:41:29.636658 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) May 23 12:41:29.636658 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create May 23 12:41:29.636658 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self.creator() May 23 12:41:29.636658 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value May 23 12:41:29.636658 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application created_value = creator( May 23 12:41:29.637693 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project May 23 12:41:29.637693 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) May 23 12:41:29.637693 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project May 23 12:41:29.637693 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() May 23 12:41:29.637693 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project May 23 12:41:29.637693 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) May 23 12:41:29.637693 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. May 23 12:41:29.637693 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application May 23 12:41:29.644654 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 290/589] 10.222.0.30 () {56 vars in 1111 bytes} [Mon May 23 12:41:29 2022] GET /identity/v3/projects/service => generated 88 bytes in 190 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) May 23 12:41:29.644654 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd770897-70c3-4c6f-bec7-9213c6023372 None admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:41:29.673313 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=100 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:41:29.702012 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b646395d-3a21-4894-a57a-474d10a9049b None admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:41:29.738783 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c84676e-079a-4639-af86-1daa43a9536c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:41:29.739585 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c84676e-079a-4639-af86-1daa43a9536c None admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:41:29.739872 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c84676e-079a-4639-af86-1daa43a9536c None admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:41:29.740151 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c84676e-079a-4639-af86-1daa43a9536c None admin] PATH_INFO: `/v3/users/cinder` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:41:29.748817 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c84676e-079a-4639-af86-1daa43a9536c None admin] RBAC: Authorizing `identity:get_user(user_id=cinder)` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:41:29.750369 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c84676e-079a-4639-af86-1daa43a9536c None admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:41:29.751276 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd770897-70c3-4c6f-bec7-9213c6023372 None admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:41:29.766072 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application [None req-7c84676e-079a-4639-af86-1daa43a9536c None admin] Could not find user: cinder.: keystone.exception.UserNotFound: Could not find user: cinder. May 23 12:41:29.766072 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application Traceback (most recent call last): May 23 12:41:29.766072 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request May 23 12:41:29.766072 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application rv = self.dispatch_request() May 23 12:41:29.766072 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request May 23 12:41:29.766072 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) May 23 12:41:29.766072 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper May 23 12:41:29.766072 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) May 23 12:41:29.766072 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view May 23 12:41:29.766072 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) May 23 12:41:29.766072 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request May 23 12:41:29.766072 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) May 23 12:41:29.766072 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get May 23 12:41:29.766072 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self._get_user(user_id) May 23 12:41:29.766072 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user May 23 12:41:29.766072 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) May 23 12:41:29.767247 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped May 23 12:41:29.767247 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) May 23 12:41:29.767247 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper May 23 12:41:29.767247 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) May 23 12:41:29.767247 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper May 23 12:41:29.767247 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) May 23 12:41:29.767247 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun May 23 12:41:29.767247 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) May 23 12:41:29.767247 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func May 23 12:41:29.767247 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self.get_or_create( May 23 12:41:29.767247 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create May 23 12:41:29.767247 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application with Lock( May 23 12:41:29.767247 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ May 23 12:41:29.767247 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self._enter() May 23 12:41:29.767247 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter May 23 12:41:29.767247 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) May 23 12:41:29.767247 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create May 23 12:41:29.767247 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self.creator() May 23 12:41:29.768232 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value May 23 12:41:29.768232 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application created_value = creator( May 23 12:41:29.768232 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user May 23 12:41:29.768232 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) May 23 12:41:29.768232 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user May 23 12:41:29.768232 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) May 23 12:41:29.768232 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user May 23 12:41:29.768232 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) May 23 12:41:29.768232 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: cinder. May 23 12:41:29.768232 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application May 23 12:41:29.768232 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 300/590] 10.222.0.30 () {56 vars in 1099 bytes} [Mon May 23 12:41:29 2022] GET /identity/v3/users/cinder => generated 84 bytes in 185 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) May 23 12:41:29.768232 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c84676e-079a-4639-af86-1daa43a9536c None admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:41:29.791831 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2847813-c185-4738-b432-9e70f937e944 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:41:29.792838 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2847813-c185-4738-b432-9e70f937e944 None admin] REQUEST_METHOD: `PUT` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:41:29.793123 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2847813-c185-4738-b432-9e70f937e944 None admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:41:29.793399 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2847813-c185-4738-b432-9e70f937e944 None admin] PATH_INFO: `/v3/projects/9c389ae65aaa4542bb5a5c4fd0bacf5b/users/cbe9931d94d44cc7ace0e83a7e5015fc/roles/41ab28c1ae3d4941bd09fb5d35f1dc54` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:41:29.809477 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2847813-c185-4738-b432-9e70f937e944 None admin] RBAC: Authorizing `identity:create_grant(project_id=9c389ae65aaa4542bb5a5c4fd0bacf5b, user_id=cbe9931d94d44cc7ace0e83a7e5015fc, role_id=41ab28c1ae3d4941bd09fb5d35f1dc54)` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:41:29.811212 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2847813-c185-4738-b432-9e70f937e944 None admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:41:29.870560 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c84676e-079a-4639-af86-1daa43a9536c None admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:41:29.878363 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 291/591] 10.222.0.30 () {58 vars in 1439 bytes} [Mon May 23 12:41:29 2022] PUT /identity/v3/projects/9c389ae65aaa4542bb5a5c4fd0bacf5b/users/cbe9931d94d44cc7ace0e83a7e5015fc/roles/41ab28c1ae3d4941bd09fb5d35f1dc54 => generated 0 bytes in 238 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) May 23 12:41:29.879356 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2847813-c185-4738-b432-9e70f937e944 None admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:41:29.899759 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:41:29.920651 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6251d062-3840-4fbd-9d22-d4c5e0117cfe None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:41:29.921746 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6251d062-3840-4fbd-9d22-d4c5e0117cfe None admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:41:29.922277 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6251d062-3840-4fbd-9d22-d4c5e0117cfe None admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:41:29.922724 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6251d062-3840-4fbd-9d22-d4c5e0117cfe None admin] PATH_INFO: `/v3/domains` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:41:29.924195 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6251d062-3840-4fbd-9d22-d4c5e0117cfe None admin] RBAC: Adding query filter params (name=Default) {{(pid=92056) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} May 23 12:41:29.924195 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6251d062-3840-4fbd-9d22-d4c5e0117cfe None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:41:29.925992 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6251d062-3840-4fbd-9d22-d4c5e0117cfe None admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:41:29.937306 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=100 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:41:29.941140 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 301/592] 10.222.0.30 () {56 vars in 1109 bytes} [Mon May 23 12:41:29 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 173 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) May 23 12:41:29.947028 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6251d062-3840-4fbd-9d22-d4c5e0117cfe None admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:41:30.015890 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2847813-c185-4738-b432-9e70f937e944 None admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:41:30.065902 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f54e215-d8f4-4999-bfdc-b3c2637f774b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:41:30.066881 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f54e215-d8f4-4999-bfdc-b3c2637f774b None admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:41:30.067315 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f54e215-d8f4-4999-bfdc-b3c2637f774b None admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:41:30.067813 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f54e215-d8f4-4999-bfdc-b3c2637f774b None admin] PATH_INFO: `/v3/projects` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:41:30.068928 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f54e215-d8f4-4999-bfdc-b3c2637f774b None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=92057) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} May 23 12:41:30.070080 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f54e215-d8f4-4999-bfdc-b3c2637f774b None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:41:30.071805 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f54e215-d8f4-4999-bfdc-b3c2637f774b None admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:41:30.092775 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6251d062-3840-4fbd-9d22-d4c5e0117cfe None admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:41:30.093034 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 292/593] 10.222.0.30 () {56 vars in 1148 bytes} [Mon May 23 12:41:29 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 209 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) May 23 12:41:30.094671 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f54e215-d8f4-4999-bfdc-b3c2637f774b None admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:41:30.134623 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e09226c2-50d8-4f4e-a8a9-05ef46c22668 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:41:30.135689 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e09226c2-50d8-4f4e-a8a9-05ef46c22668 None admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:41:30.135969 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e09226c2-50d8-4f4e-a8a9-05ef46c22668 None admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:41:30.136242 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e09226c2-50d8-4f4e-a8a9-05ef46c22668 None admin] PATH_INFO: `/v3/users` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:41:30.137076 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e09226c2-50d8-4f4e-a8a9-05ef46c22668 None admin] RBAC: Adding query filter params (domain_id=default, name=cinder) {{(pid=92056) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} May 23 12:41:30.137383 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e09226c2-50d8-4f4e-a8a9-05ef46c22668 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:41:30.138973 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e09226c2-50d8-4f4e-a8a9-05ef46c22668 None admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:41:30.158173 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 302/594] 10.222.0.30 () {56 vars in 1137 bytes} [Mon May 23 12:41:29 2022] GET /identity/v3/users?domain_id=default&name=cinder => generated 372 bytes in 216 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) May 23 12:41:30.159066 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e09226c2-50d8-4f4e-a8a9-05ef46c22668 None admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:41:30.210623 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=100 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:41:30.213823 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f54e215-d8f4-4999-bfdc-b3c2637f774b None admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:41:30.218527 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=100 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:41:30.250642 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7eb25499-e46e-4525-935d-d050345bf26d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:41:30.251488 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7eb25499-e46e-4525-935d-d050345bf26d None admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:41:30.251777 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7eb25499-e46e-4525-935d-d050345bf26d None admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:41:30.252055 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7eb25499-e46e-4525-935d-d050345bf26d None admin] PATH_INFO: `/v3/users/neutron` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:41:30.260551 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7eb25499-e46e-4525-935d-d050345bf26d None admin] RBAC: Authorizing `identity:get_user(user_id=neutron)` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:41:30.262084 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7eb25499-e46e-4525-935d-d050345bf26d None admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:41:30.262245 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e09226c2-50d8-4f4e-a8a9-05ef46c22668 None admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:41:30.276783 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application [None req-7eb25499-e46e-4525-935d-d050345bf26d None admin] Could not find user: neutron.: keystone.exception.UserNotFound: Could not find user: neutron. May 23 12:41:30.276783 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application Traceback (most recent call last): May 23 12:41:30.276783 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request May 23 12:41:30.276783 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application rv = self.dispatch_request() May 23 12:41:30.276783 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request May 23 12:41:30.276783 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) May 23 12:41:30.276783 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper May 23 12:41:30.276783 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) May 23 12:41:30.276783 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view May 23 12:41:30.276783 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) May 23 12:41:30.276783 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request May 23 12:41:30.276783 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) May 23 12:41:30.276783 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get May 23 12:41:30.276783 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self._get_user(user_id) May 23 12:41:30.276783 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user May 23 12:41:30.276783 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) May 23 12:41:30.278083 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped May 23 12:41:30.278083 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) May 23 12:41:30.278083 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper May 23 12:41:30.278083 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) May 23 12:41:30.278083 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper May 23 12:41:30.278083 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) May 23 12:41:30.278083 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun May 23 12:41:30.278083 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) May 23 12:41:30.278083 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func May 23 12:41:30.278083 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self.get_or_create( May 23 12:41:30.278083 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create May 23 12:41:30.278083 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application with Lock( May 23 12:41:30.278083 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ May 23 12:41:30.278083 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self._enter() May 23 12:41:30.278083 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter May 23 12:41:30.278083 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) May 23 12:41:30.278083 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create May 23 12:41:30.278083 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self.creator() May 23 12:41:30.279293 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value May 23 12:41:30.279293 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application created_value = creator( May 23 12:41:30.279293 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user May 23 12:41:30.279293 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) May 23 12:41:30.279293 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user May 23 12:41:30.279293 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) May 23 12:41:30.279293 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user May 23 12:41:30.279293 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) May 23 12:41:30.279293 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: neutron. May 23 12:41:30.279293 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application May 23 12:41:30.279293 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 293/595] 10.222.0.30 () {56 vars in 1102 bytes} [Mon May 23 12:41:30 2022] GET /identity/v3/users/neutron => generated 85 bytes in 185 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) May 23 12:41:30.279923 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7eb25499-e46e-4525-935d-d050345bf26d None admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:41:30.297946 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c8425a8-dd15-4280-884a-c2b616aee55b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:41:30.299002 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c8425a8-dd15-4280-884a-c2b616aee55b None admin] REQUEST_METHOD: `PUT` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:41:30.299344 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c8425a8-dd15-4280-884a-c2b616aee55b None admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:41:30.299679 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c8425a8-dd15-4280-884a-c2b616aee55b None admin] PATH_INFO: `/v3/projects/9c389ae65aaa4542bb5a5c4fd0bacf5b/users/5cea0feed4254ec085854986f99acf30/roles/41ab28c1ae3d4941bd09fb5d35f1dc54` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:41:30.316482 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c8425a8-dd15-4280-884a-c2b616aee55b None admin] RBAC: Authorizing `identity:create_grant(project_id=9c389ae65aaa4542bb5a5c4fd0bacf5b, user_id=5cea0feed4254ec085854986f99acf30, role_id=41ab28c1ae3d4941bd09fb5d35f1dc54)` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:41:30.317958 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c8425a8-dd15-4280-884a-c2b616aee55b None admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:41:30.405990 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 303/596] 10.222.0.30 () {58 vars in 1439 bytes} [Mon May 23 12:41:30 2022] PUT /identity/v3/projects/9c389ae65aaa4542bb5a5c4fd0bacf5b/users/5cea0feed4254ec085854986f99acf30/roles/41ab28c1ae3d4941bd09fb5d35f1dc54 => generated 0 bytes in 238 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) May 23 12:41:30.405990 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c8425a8-dd15-4280-884a-c2b616aee55b None admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:41:30.409808 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7eb25499-e46e-4525-935d-d050345bf26d None admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:41:30.483426 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=100 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:41:30.500006 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d7b013c-d1b3-4502-b23a-a3554f61e0e1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:41:30.501273 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d7b013c-d1b3-4502-b23a-a3554f61e0e1 None admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:41:30.501781 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d7b013c-d1b3-4502-b23a-a3554f61e0e1 None admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:41:30.502282 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d7b013c-d1b3-4502-b23a-a3554f61e0e1 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:41:30.528792 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c8425a8-dd15-4280-884a-c2b616aee55b None admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:41:30.535035 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d7b013c-d1b3-4502-b23a-a3554f61e0e1 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:41:30.536799 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d7b013c-d1b3-4502-b23a-a3554f61e0e1 None admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:41:30.540966 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:41:30.567934 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-647a6c56-a782-465d-838f-766f6a1aa878 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:41:30.568653 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-647a6c56-a782-465d-838f-766f6a1aa878 None admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:41:30.568936 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-647a6c56-a782-465d-838f-766f6a1aa878 None admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:41:30.581998 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-647a6c56-a782-465d-838f-766f6a1aa878 None admin] PATH_INFO: `/v3/users` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:41:30.581998 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-647a6c56-a782-465d-838f-766f6a1aa878 None admin] RBAC: Adding query filter params (domain_id=default, name=neutron) {{(pid=92056) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} May 23 12:41:30.581998 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-647a6c56-a782-465d-838f-766f6a1aa878 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:41:30.581998 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-647a6c56-a782-465d-838f-766f6a1aa878 None admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:41:30.584199 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application [None req-1d7b013c-d1b3-4502-b23a-a3554f61e0e1 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. May 23 12:41:30.584199 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application Traceback (most recent call last): May 23 12:41:30.584199 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request May 23 12:41:30.584199 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application rv = self.dispatch_request() May 23 12:41:30.584199 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request May 23 12:41:30.584199 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) May 23 12:41:30.584199 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper May 23 12:41:30.584199 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) May 23 12:41:30.584199 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view May 23 12:41:30.584199 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) May 23 12:41:30.584199 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request May 23 12:41:30.584199 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) May 23 12:41:30.584199 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get May 23 12:41:30.584199 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self._get_domain(domain_id) May 23 12:41:30.584199 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain May 23 12:41:30.586417 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) May 23 12:41:30.586417 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped May 23 12:41:30.586417 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) May 23 12:41:30.586417 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun May 23 12:41:30.586417 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) May 23 12:41:30.586417 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func May 23 12:41:30.586417 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self.get_or_create( May 23 12:41:30.586417 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create May 23 12:41:30.586417 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application with Lock( May 23 12:41:30.586417 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ May 23 12:41:30.586417 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self._enter() May 23 12:41:30.586417 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter May 23 12:41:30.586417 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) May 23 12:41:30.586417 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create May 23 12:41:30.586417 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self.creator() May 23 12:41:30.586417 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value May 23 12:41:30.586417 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application created_value = creator( May 23 12:41:30.587354 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain May 23 12:41:30.587354 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) May 23 12:41:30.587354 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. May 23 12:41:30.587354 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application May 23 12:41:30.589306 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 294/597] 10.222.0.30 () {56 vars in 1108 bytes} [Mon May 23 12:41:30 2022] GET /identity/v3/domains/Default => generated 87 bytes in 310 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) May 23 12:41:30.591645 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 304/598] 10.222.0.30 () {56 vars in 1139 bytes} [Mon May 23 12:41:30 2022] GET /identity/v3/users?domain_id=default&name=neutron => generated 374 bytes in 193 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) May 23 12:41:30.595913 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d7b013c-d1b3-4502-b23a-a3554f61e0e1 None admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:41:30.601945 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-647a6c56-a782-465d-838f-766f6a1aa878 None admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:41:30.746041 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-647a6c56-a782-465d-838f-766f6a1aa878 None admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:41:30.764631 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d7b013c-d1b3-4502-b23a-a3554f61e0e1 None admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:41:30.797103 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16874ccd-182f-4c34-b9a5-4b700de50c9f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:41:30.798306 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16874ccd-182f-4c34-b9a5-4b700de50c9f None admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:41:30.798689 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16874ccd-182f-4c34-b9a5-4b700de50c9f None admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:41:30.799472 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16874ccd-182f-4c34-b9a5-4b700de50c9f None admin] PATH_INFO: `/v3/domains/Default` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:41:30.812818 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16874ccd-182f-4c34-b9a5-4b700de50c9f None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:41:30.814737 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16874ccd-182f-4c34-b9a5-4b700de50c9f None admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:41:30.828100 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2bcb114-f1fb-44b2-80f9-d5347138f043 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:41:30.829035 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2bcb114-f1fb-44b2-80f9-d5347138f043 None admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:41:30.829465 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2bcb114-f1fb-44b2-80f9-d5347138f043 None admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:41:30.829896 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2bcb114-f1fb-44b2-80f9-d5347138f043 None admin] PATH_INFO: `/v3/domains` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:41:30.830766 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2bcb114-f1fb-44b2-80f9-d5347138f043 None admin] RBAC: Adding query filter params (name=Default) {{(pid=92057) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} May 23 12:41:30.831236 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2bcb114-f1fb-44b2-80f9-d5347138f043 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:41:30.832776 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2bcb114-f1fb-44b2-80f9-d5347138f043 None admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:41:30.841900 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application [None req-16874ccd-182f-4c34-b9a5-4b700de50c9f None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. May 23 12:41:30.841900 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application Traceback (most recent call last): May 23 12:41:30.841900 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request May 23 12:41:30.841900 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application rv = self.dispatch_request() May 23 12:41:30.841900 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request May 23 12:41:30.841900 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) May 23 12:41:30.841900 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper May 23 12:41:30.841900 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) May 23 12:41:30.841900 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view May 23 12:41:30.841900 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) May 23 12:41:30.841900 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request May 23 12:41:30.841900 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) May 23 12:41:30.841900 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get May 23 12:41:30.841900 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self._get_domain(domain_id) May 23 12:41:30.841900 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain May 23 12:41:30.842849 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) May 23 12:41:30.842849 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped May 23 12:41:30.842849 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) May 23 12:41:30.842849 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun May 23 12:41:30.842849 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) May 23 12:41:30.842849 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func May 23 12:41:30.842849 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self.get_or_create( May 23 12:41:30.842849 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create May 23 12:41:30.842849 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application with Lock( May 23 12:41:30.842849 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ May 23 12:41:30.842849 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self._enter() May 23 12:41:30.842849 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter May 23 12:41:30.842849 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) May 23 12:41:30.842849 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create May 23 12:41:30.842849 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self.creator() May 23 12:41:30.842849 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value May 23 12:41:30.842849 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application created_value = creator( May 23 12:41:30.843770 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain May 23 12:41:30.843770 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) May 23 12:41:30.843770 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. May 23 12:41:30.843770 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application May 23 12:41:30.843770 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 305/599] 10.222.0.30 () {56 vars in 1108 bytes} [Mon May 23 12:41:30 2022] GET /identity/v3/domains/Default => generated 87 bytes in 238 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) May 23 12:41:30.843770 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16874ccd-182f-4c34-b9a5-4b700de50c9f None admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:41:30.863810 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 295/600] 10.222.0.30 () {56 vars in 1109 bytes} [Mon May 23 12:41:30 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 269 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) May 23 12:41:30.868786 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=100 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:41:30.870073 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2bcb114-f1fb-44b2-80f9-d5347138f043 None admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:41:30.962436 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16874ccd-182f-4c34-b9a5-4b700de50c9f None admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:41:30.983717 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=100 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:41:31.004172 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5efa9d7a-fe8e-47e7-bfe5-824d50d2de97 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:41:31.005261 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5efa9d7a-fe8e-47e7-bfe5-824d50d2de97 None admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:41:31.005640 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5efa9d7a-fe8e-47e7-bfe5-824d50d2de97 None admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:41:31.006045 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5efa9d7a-fe8e-47e7-bfe5-824d50d2de97 None admin] PATH_INFO: `/v3/domains` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:41:31.007304 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5efa9d7a-fe8e-47e7-bfe5-824d50d2de97 None admin] RBAC: Adding query filter params (name=Default) {{(pid=92056) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} May 23 12:41:31.007304 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5efa9d7a-fe8e-47e7-bfe5-824d50d2de97 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:41:31.009565 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5efa9d7a-fe8e-47e7-bfe5-824d50d2de97 None admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:41:31.026788 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 306/601] 10.222.0.30 () {56 vars in 1109 bytes} [Mon May 23 12:41:30 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 184 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) May 23 12:41:31.032357 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5efa9d7a-fe8e-47e7-bfe5-824d50d2de97 None admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:41:31.059034 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2bcb114-f1fb-44b2-80f9-d5347138f043 None admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:41:31.135142 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=100 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:41:31.150788 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0f96dc4-70bf-48e8-9d57-a6f71284dca0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:41:31.152073 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0f96dc4-70bf-48e8-9d57-a6f71284dca0 None admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:41:31.152835 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0f96dc4-70bf-48e8-9d57-a6f71284dca0 None admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:41:31.153331 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0f96dc4-70bf-48e8-9d57-a6f71284dca0 None admin] PATH_INFO: `/v3/projects/service` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:41:31.158335 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5efa9d7a-fe8e-47e7-bfe5-824d50d2de97 None admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:41:31.179335 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0f96dc4-70bf-48e8-9d57-a6f71284dca0 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:41:31.181213 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0f96dc4-70bf-48e8-9d57-a6f71284dca0 None admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:41:31.202133 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application [None req-c0f96dc4-70bf-48e8-9d57-a6f71284dca0 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. May 23 12:41:31.202133 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application Traceback (most recent call last): May 23 12:41:31.202133 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request May 23 12:41:31.202133 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application rv = self.dispatch_request() May 23 12:41:31.202133 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request May 23 12:41:31.202133 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) May 23 12:41:31.202133 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper May 23 12:41:31.202133 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) May 23 12:41:31.202133 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view May 23 12:41:31.202133 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) May 23 12:41:31.202133 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request May 23 12:41:31.202133 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) May 23 12:41:31.202133 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get May 23 12:41:31.202133 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self._get_project(project_id) May 23 12:41:31.202133 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project May 23 12:41:31.203127 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) May 23 12:41:31.203127 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped May 23 12:41:31.203127 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) May 23 12:41:31.203127 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun May 23 12:41:31.203127 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) May 23 12:41:31.203127 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func May 23 12:41:31.203127 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self.get_or_create( May 23 12:41:31.203127 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create May 23 12:41:31.203127 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application with Lock( May 23 12:41:31.203127 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ May 23 12:41:31.203127 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self._enter() May 23 12:41:31.203127 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter May 23 12:41:31.203127 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) May 23 12:41:31.203127 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create May 23 12:41:31.203127 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self.creator() May 23 12:41:31.203127 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value May 23 12:41:31.203127 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application created_value = creator( May 23 12:41:31.204036 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project May 23 12:41:31.204036 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) May 23 12:41:31.204036 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project May 23 12:41:31.204036 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() May 23 12:41:31.204036 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project May 23 12:41:31.204036 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) May 23 12:41:31.204036 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. May 23 12:41:31.204036 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application May 23 12:41:31.205699 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4e015d6-f0bd-45d3-874c-9ebee72eb4f3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:41:31.206529 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4e015d6-f0bd-45d3-874c-9ebee72eb4f3 None admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:41:31.206817 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4e015d6-f0bd-45d3-874c-9ebee72eb4f3 None admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:41:31.207196 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4e015d6-f0bd-45d3-874c-9ebee72eb4f3 None admin] PATH_INFO: `/v3/projects/service` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:41:31.207682 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 296/602] 10.222.0.30 () {56 vars in 1111 bytes} [Mon May 23 12:41:30 2022] GET /identity/v3/projects/service => generated 88 bytes in 338 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) May 23 12:41:31.213374 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0f96dc4-70bf-48e8-9d57-a6f71284dca0 None admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:41:31.218004 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c4e015d6-f0bd-45d3-874c-9ebee72eb4f3 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:41:31.220280 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c4e015d6-f0bd-45d3-874c-9ebee72eb4f3 None admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:41:31.255273 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application [None req-c4e015d6-f0bd-45d3-874c-9ebee72eb4f3 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. May 23 12:41:31.255273 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application Traceback (most recent call last): May 23 12:41:31.255273 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request May 23 12:41:31.255273 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application rv = self.dispatch_request() May 23 12:41:31.255273 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request May 23 12:41:31.255273 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) May 23 12:41:31.255273 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper May 23 12:41:31.255273 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) May 23 12:41:31.255273 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view May 23 12:41:31.255273 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) May 23 12:41:31.255273 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request May 23 12:41:31.255273 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) May 23 12:41:31.255273 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get May 23 12:41:31.255273 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self._get_project(project_id) May 23 12:41:31.255273 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project May 23 12:41:31.256260 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) May 23 12:41:31.256260 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped May 23 12:41:31.256260 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) May 23 12:41:31.256260 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun May 23 12:41:31.256260 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) May 23 12:41:31.256260 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func May 23 12:41:31.256260 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self.get_or_create( May 23 12:41:31.256260 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create May 23 12:41:31.256260 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application with Lock( May 23 12:41:31.256260 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ May 23 12:41:31.256260 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self._enter() May 23 12:41:31.256260 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter May 23 12:41:31.256260 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) May 23 12:41:31.256260 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create May 23 12:41:31.256260 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self.creator() May 23 12:41:31.256260 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value May 23 12:41:31.256260 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application created_value = creator( May 23 12:41:31.257819 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project May 23 12:41:31.257819 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) May 23 12:41:31.257819 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project May 23 12:41:31.257819 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() May 23 12:41:31.257819 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project May 23 12:41:31.257819 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) May 23 12:41:31.257819 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. May 23 12:41:31.257819 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application May 23 12:41:31.263758 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 307/603] 10.222.0.30 () {56 vars in 1111 bytes} [Mon May 23 12:41:31 2022] GET /identity/v3/projects/service => generated 88 bytes in 230 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) May 23 12:41:31.268156 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7502d4b-8e7c-4370-96ca-243becd264e0 None None] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:41:31.268873 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7502d4b-8e7c-4370-96ca-243becd264e0 None None] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:41:31.270024 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7502d4b-8e7c-4370-96ca-243becd264e0 None None] PATH_INFO: `/` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:41:31.271708 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 308/604] 10.222.0.30 () {54 vars in 926 bytes} [Mon May 23 12:41:31 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) May 23 12:41:31.274642 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7502d4b-8e7c-4370-96ca-243becd264e0 None None] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:41:31.348826 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0f96dc4-70bf-48e8-9d57-a6f71284dca0 None admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:41:31.353915 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=100 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:41:31.401664 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-768ac543-dcaa-4288-ad46-a1977b6aa87f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:41:31.403004 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7502d4b-8e7c-4370-96ca-243becd264e0 None None] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:41:31.403366 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-768ac543-dcaa-4288-ad46-a1977b6aa87f None admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:41:31.403847 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-768ac543-dcaa-4288-ad46-a1977b6aa87f None admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:41:31.404623 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-768ac543-dcaa-4288-ad46-a1977b6aa87f None admin] PATH_INFO: `/v3/projects` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:41:31.406474 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-768ac543-dcaa-4288-ad46-a1977b6aa87f None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=92057) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} May 23 12:41:31.406799 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-768ac543-dcaa-4288-ad46-a1977b6aa87f None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:41:31.408270 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-768ac543-dcaa-4288-ad46-a1977b6aa87f None admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:41:31.422327 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 297/605] 10.222.0.30 () {56 vars in 1148 bytes} [Mon May 23 12:41:31 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 209 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) May 23 12:41:31.423665 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ca14a1d-a82c-487f-bfb3-4786f2129611 None None] REQUEST_METHOD: `POST` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:41:31.423955 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ca14a1d-a82c-487f-bfb3-4786f2129611 None None] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:41:31.424234 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ca14a1d-a82c-487f-bfb3-4786f2129611 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:41:31.442136 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.auth.core [None req-6ca14a1d-a82c-487f-bfb3-4786f2129611 None None] MFA Rules not processed for user `0f7119f8fa5c4445977671140ded91a4`. Rule list: `[]` (Enabled: `True`). {{(pid=92057) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 12:41:31.443490 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.fernet_utils [None req-6ca14a1d-a82c-487f-bfb3-4786f2129611 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92057) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 12:41:31.449957 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ac194d0-239b-4979-b2b6-00ba9224590d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:41:31.450853 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ac194d0-239b-4979-b2b6-00ba9224590d None admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:41:31.451349 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ac194d0-239b-4979-b2b6-00ba9224590d None admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:41:31.452331 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ac194d0-239b-4979-b2b6-00ba9224590d None admin] PATH_INFO: `/v3/projects` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:41:31.453314 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ac194d0-239b-4979-b2b6-00ba9224590d None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=92056) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} May 23 12:41:31.453860 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ac194d0-239b-4979-b2b6-00ba9224590d None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:41:31.456364 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ac194d0-239b-4979-b2b6-00ba9224590d None admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:41:31.466302 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=100 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:41:31.473832 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 309/606] 10.222.0.30 () {56 vars in 1148 bytes} [Mon May 23 12:41:31 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 199 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) May 23 12:41:31.474991 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ac194d0-239b-4979-b2b6-00ba9224590d None admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:41:31.525981 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 298/607] 10.222.0.30 () {58 vars in 1023 bytes} [Mon May 23 12:41:31 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 98 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) May 23 12:41:31.527266 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ca14a1d-a82c-487f-bfb3-4786f2129611 None None] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:41:31.597996 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ac194d0-239b-4979-b2b6-00ba9224590d None admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:41:31.643835 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0eb53202-172b-44f8-a88c-bda679ad5050 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:41:31.644759 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0eb53202-172b-44f8-a88c-bda679ad5050 None admin] REQUEST_METHOD: `PUT` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:41:31.645037 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0eb53202-172b-44f8-a88c-bda679ad5050 None admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:41:31.645315 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0eb53202-172b-44f8-a88c-bda679ad5050 None admin] PATH_INFO: `/v3/projects/9c389ae65aaa4542bb5a5c4fd0bacf5b/users/1666097f8ed545c6b0e4809c93e4775a/roles/41ab28c1ae3d4941bd09fb5d35f1dc54` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:41:31.648821 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=100 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:41:31.656759 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ca14a1d-a82c-487f-bfb3-4786f2129611 None None] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:41:31.666310 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0eb53202-172b-44f8-a88c-bda679ad5050 None admin] RBAC: Authorizing `identity:create_grant(project_id=9c389ae65aaa4542bb5a5c4fd0bacf5b, user_id=1666097f8ed545c6b0e4809c93e4775a, role_id=41ab28c1ae3d4941bd09fb5d35f1dc54)` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:41:31.668197 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0eb53202-172b-44f8-a88c-bda679ad5050 None admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:41:31.699307 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-079965e7-199c-4602-8f7f-d06c8df4e42a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:41:31.700300 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-079965e7-199c-4602-8f7f-d06c8df4e42a None admin] REQUEST_METHOD: `PUT` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:41:31.700653 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-079965e7-199c-4602-8f7f-d06c8df4e42a None admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:41:31.701001 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-079965e7-199c-4602-8f7f-d06c8df4e42a None admin] PATH_INFO: `/v3/projects/9c389ae65aaa4542bb5a5c4fd0bacf5b/users/2eb83ca38e79449095f3a2f947bbb06a/roles/41ab28c1ae3d4941bd09fb5d35f1dc54` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:41:31.715792 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-079965e7-199c-4602-8f7f-d06c8df4e42a None admin] RBAC: Authorizing `identity:create_grant(project_id=9c389ae65aaa4542bb5a5c4fd0bacf5b, user_id=2eb83ca38e79449095f3a2f947bbb06a, role_id=41ab28c1ae3d4941bd09fb5d35f1dc54)` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:41:31.730053 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-079965e7-199c-4602-8f7f-d06c8df4e42a None admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:41:31.789920 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 310/608] 10.222.0.30 () {58 vars in 1439 bytes} [Mon May 23 12:41:31 2022] PUT /identity/v3/projects/9c389ae65aaa4542bb5a5c4fd0bacf5b/users/1666097f8ed545c6b0e4809c93e4775a/roles/41ab28c1ae3d4941bd09fb5d35f1dc54 => generated 0 bytes in 309 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) May 23 12:41:31.789920 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0eb53202-172b-44f8-a88c-bda679ad5050 None admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:41:31.824570 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 299/609] 10.222.0.30 () {58 vars in 1439 bytes} [Mon May 23 12:41:31 2022] PUT /identity/v3/projects/9c389ae65aaa4542bb5a5c4fd0bacf5b/users/2eb83ca38e79449095f3a2f947bbb06a/roles/41ab28c1ae3d4941bd09fb5d35f1dc54 => generated 0 bytes in 298 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) May 23 12:41:31.866530 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:41:31.948578 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0eb53202-172b-44f8-a88c-bda679ad5050 None admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:41:32.001557 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92daa78c-7cea-483f-aace-f3336690ec1c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:41:32.002357 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92daa78c-7cea-483f-aace-f3336690ec1c None admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:41:32.002643 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92daa78c-7cea-483f-aace-f3336690ec1c None admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:41:32.002919 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92daa78c-7cea-483f-aace-f3336690ec1c None admin] PATH_INFO: `/v3/roles/service` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:41:32.017394 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92daa78c-7cea-483f-aace-f3336690ec1c None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} May 23 12:41:32.026454 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application [None req-92daa78c-7cea-483f-aace-f3336690ec1c None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. May 23 12:41:32.026454 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application Traceback (most recent call last): May 23 12:41:32.026454 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request May 23 12:41:32.026454 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application rv = self.dispatch_request() May 23 12:41:32.026454 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request May 23 12:41:32.026454 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) May 23 12:41:32.026454 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper May 23 12:41:32.026454 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) May 23 12:41:32.026454 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view May 23 12:41:32.026454 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) May 23 12:41:32.026454 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request May 23 12:41:32.026454 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) May 23 12:41:32.026454 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get May 23 12:41:32.026454 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self._get_role(role_id) May 23 12:41:32.026454 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role May 23 12:41:32.026454 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') May 23 12:41:32.027849 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call May 23 12:41:32.027849 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( May 23 12:41:32.027849 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data May 23 12:41:32.027849 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) May 23 12:41:32.027849 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped May 23 12:41:32.027849 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) May 23 12:41:32.027849 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun May 23 12:41:32.027849 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) May 23 12:41:32.027849 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func May 23 12:41:32.027849 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self.get_or_create( May 23 12:41:32.027849 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create May 23 12:41:32.027849 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application with Lock( May 23 12:41:32.027849 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ May 23 12:41:32.027849 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self._enter() May 23 12:41:32.027849 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter May 23 12:41:32.027849 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) May 23 12:41:32.028775 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create May 23 12:41:32.028775 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self.creator() May 23 12:41:32.028775 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value May 23 12:41:32.028775 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application created_value = creator( May 23 12:41:32.028775 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role May 23 12:41:32.028775 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) May 23 12:41:32.028775 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role May 23 12:41:32.028775 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() May 23 12:41:32.028775 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role May 23 12:41:32.028775 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) May 23 12:41:32.028775 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. May 23 12:41:32.028775 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application May 23 12:41:32.028775 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 311/610] 10.222.0.30 () {56 vars in 1102 bytes} [Mon May 23 12:41:31 2022] GET /identity/v3/roles/service => generated 85 bytes in 244 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) May 23 12:41:32.034385 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-079965e7-199c-4602-8f7f-d06c8df4e42a None admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:41:32.160253 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-079965e7-199c-4602-8f7f-d06c8df4e42a None admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:41:32.204981 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4095f4e-8372-49ae-b96f-0d3314581a0e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:41:32.205649 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4095f4e-8372-49ae-b96f-0d3314581a0e None admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:41:32.205970 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4095f4e-8372-49ae-b96f-0d3314581a0e None admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:41:32.206250 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4095f4e-8372-49ae-b96f-0d3314581a0e None admin] PATH_INFO: `/v3/roles` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:41:32.206992 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f4095f4e-8372-49ae-b96f-0d3314581a0e None admin] RBAC: Adding query filter params (name=service) {{(pid=92057) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} May 23 12:41:32.207300 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f4095f4e-8372-49ae-b96f-0d3314581a0e None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:41:32.208698 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f4095f4e-8372-49ae-b96f-0d3314581a0e None admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:41:32.218120 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 300/611] 10.222.0.30 () {56 vars in 1103 bytes} [Mon May 23 12:41:32 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 184 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) May 23 12:41:32.223592 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92daa78c-7cea-483f-aace-f3336690ec1c None admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:41:32.381277 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92daa78c-7cea-483f-aace-f3336690ec1c None admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:41:32.391002 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=100 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:41:32.418098 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aabc4214-be07-4463-87f2-e6122f4b00de None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:41:32.418898 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aabc4214-be07-4463-87f2-e6122f4b00de None admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:41:32.419183 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aabc4214-be07-4463-87f2-e6122f4b00de None admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:41:32.419467 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aabc4214-be07-4463-87f2-e6122f4b00de None admin] PATH_INFO: `/v3/domains/Default` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:41:32.431470 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aabc4214-be07-4463-87f2-e6122f4b00de None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:41:32.432942 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aabc4214-be07-4463-87f2-e6122f4b00de None admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:41:32.450864 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application [None req-aabc4214-be07-4463-87f2-e6122f4b00de None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. May 23 12:41:32.450864 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application Traceback (most recent call last): May 23 12:41:32.450864 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request May 23 12:41:32.450864 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application rv = self.dispatch_request() May 23 12:41:32.450864 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request May 23 12:41:32.450864 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) May 23 12:41:32.450864 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper May 23 12:41:32.450864 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) May 23 12:41:32.450864 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view May 23 12:41:32.450864 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) May 23 12:41:32.450864 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request May 23 12:41:32.450864 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) May 23 12:41:32.450864 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get May 23 12:41:32.450864 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self._get_domain(domain_id) May 23 12:41:32.450864 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain May 23 12:41:32.451815 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) May 23 12:41:32.451815 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped May 23 12:41:32.451815 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) May 23 12:41:32.451815 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun May 23 12:41:32.451815 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) May 23 12:41:32.451815 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func May 23 12:41:32.451815 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self.get_or_create( May 23 12:41:32.451815 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create May 23 12:41:32.451815 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application with Lock( May 23 12:41:32.451815 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ May 23 12:41:32.451815 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self._enter() May 23 12:41:32.451815 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter May 23 12:41:32.451815 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) May 23 12:41:32.451815 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create May 23 12:41:32.451815 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self.creator() May 23 12:41:32.451815 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value May 23 12:41:32.451815 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application created_value = creator( May 23 12:41:32.452747 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain May 23 12:41:32.452747 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) May 23 12:41:32.452747 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. May 23 12:41:32.452747 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application May 23 12:41:32.452747 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 312/612] 10.222.0.30 () {56 vars in 1108 bytes} [Mon May 23 12:41:32 2022] GET /identity/v3/domains/Default => generated 87 bytes in 229 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) May 23 12:41:32.457636 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4095f4e-8372-49ae-b96f-0d3314581a0e None admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:41:32.472122 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:41:32.537806 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67036b86-3dcd-48a4-866d-a6eeec41c768 None None] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:41:32.539255 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67036b86-3dcd-48a4-866d-a6eeec41c768 None None] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:41:32.539612 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67036b86-3dcd-48a4-866d-a6eeec41c768 None None] PATH_INFO: `/` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:41:32.541990 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 313/613] 10.222.0.30 () {54 vars in 926 bytes} [Mon May 23 12:41:32 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) May 23 12:41:32.547985 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9b3b659-4333-4a91-b165-3135c00595fb None None] REQUEST_METHOD: `POST` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:41:32.548349 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9b3b659-4333-4a91-b165-3135c00595fb None None] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:41:32.548696 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9b3b659-4333-4a91-b165-3135c00595fb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:41:32.575747 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.auth.core [None req-b9b3b659-4333-4a91-b165-3135c00595fb None None] MFA Rules not processed for user `0f7119f8fa5c4445977671140ded91a4`. Rule list: `[]` (Enabled: `True`). {{(pid=92056) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 12:41:32.577325 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.fernet_utils [None req-b9b3b659-4333-4a91-b165-3135c00595fb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92056) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 12:41:32.610894 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4095f4e-8372-49ae-b96f-0d3314581a0e None admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:41:32.683640 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac270756-4b19-40d3-928f-03080a7f26c0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:41:32.685095 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac270756-4b19-40d3-928f-03080a7f26c0 None admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:41:32.686575 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 314/614] 10.222.0.30 () {58 vars in 1023 bytes} [Mon May 23 12:41:32 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 139 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) May 23 12:41:32.687045 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac270756-4b19-40d3-928f-03080a7f26c0 None admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:41:32.687516 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac270756-4b19-40d3-928f-03080a7f26c0 None admin] PATH_INFO: `/v3/domains` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:41:32.688498 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac270756-4b19-40d3-928f-03080a7f26c0 None admin] RBAC: Adding query filter params (name=Default) {{(pid=92057) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} May 23 12:41:32.688994 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac270756-4b19-40d3-928f-03080a7f26c0 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:41:32.690616 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac270756-4b19-40d3-928f-03080a7f26c0 None admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:41:32.713998 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9b3b659-4333-4a91-b165-3135c00595fb None None] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:41:32.716516 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 301/615] 10.222.0.30 () {56 vars in 1109 bytes} [Mon May 23 12:41:32 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 259 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) May 23 12:41:32.726759 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac270756-4b19-40d3-928f-03080a7f26c0 None admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:41:32.850111 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=100 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:41:32.876538 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=100 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:41:32.892162 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9b3b659-4333-4a91-b165-3135c00595fb None None] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:41:32.917547 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac270756-4b19-40d3-928f-03080a7f26c0 None admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:41:32.961430 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6103d6ae-17c4-4801-871f-5e943b2e7447 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:41:32.962758 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6103d6ae-17c4-4801-871f-5e943b2e7447 None admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:41:32.963039 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6103d6ae-17c4-4801-871f-5e943b2e7447 None admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:41:32.963310 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6103d6ae-17c4-4801-871f-5e943b2e7447 None admin] PATH_INFO: `/v3/roles/service` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:41:32.971291 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0084094f-4090-4f30-90c1-ad7a520c0c53 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:41:32.972305 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0084094f-4090-4f30-90c1-ad7a520c0c53 None admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:41:32.972762 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0084094f-4090-4f30-90c1-ad7a520c0c53 None admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:41:32.973209 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0084094f-4090-4f30-90c1-ad7a520c0c53 None admin] PATH_INFO: `/v3/users/swift` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:41:32.981825 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0084094f-4090-4f30-90c1-ad7a520c0c53 None admin] RBAC: Authorizing `identity:get_user(user_id=swift)` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:41:32.983888 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0084094f-4090-4f30-90c1-ad7a520c0c53 None admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:41:32.986599 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6103d6ae-17c4-4801-871f-5e943b2e7447 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} May 23 12:41:32.995347 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application [None req-6103d6ae-17c4-4801-871f-5e943b2e7447 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. May 23 12:41:32.995347 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application Traceback (most recent call last): May 23 12:41:32.995347 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request May 23 12:41:32.995347 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application rv = self.dispatch_request() May 23 12:41:32.995347 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request May 23 12:41:32.995347 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) May 23 12:41:32.995347 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper May 23 12:41:32.995347 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) May 23 12:41:32.995347 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view May 23 12:41:32.995347 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) May 23 12:41:32.995347 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request May 23 12:41:32.995347 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) May 23 12:41:32.995347 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get May 23 12:41:32.995347 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self._get_role(role_id) May 23 12:41:32.995347 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role May 23 12:41:32.995347 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') May 23 12:41:32.996296 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call May 23 12:41:32.996296 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( May 23 12:41:32.996296 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data May 23 12:41:32.996296 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) May 23 12:41:32.996296 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped May 23 12:41:32.996296 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) May 23 12:41:32.996296 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun May 23 12:41:32.996296 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) May 23 12:41:32.996296 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func May 23 12:41:32.996296 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self.get_or_create( May 23 12:41:32.996296 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create May 23 12:41:32.996296 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application with Lock( May 23 12:41:32.996296 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ May 23 12:41:32.996296 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self._enter() May 23 12:41:32.996296 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter May 23 12:41:32.996296 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) May 23 12:41:32.997167 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create May 23 12:41:32.997167 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self.creator() May 23 12:41:32.997167 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value May 23 12:41:32.997167 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application created_value = creator( May 23 12:41:32.997167 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role May 23 12:41:32.997167 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) May 23 12:41:32.997167 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role May 23 12:41:32.997167 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() May 23 12:41:32.997167 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role May 23 12:41:32.997167 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) May 23 12:41:32.997167 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. May 23 12:41:32.997167 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application May 23 12:41:33.003604 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application [None req-0084094f-4090-4f30-90c1-ad7a520c0c53 None admin] Could not find user: swift.: keystone.exception.UserNotFound: Could not find user: swift. May 23 12:41:33.003604 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application Traceback (most recent call last): May 23 12:41:33.003604 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request May 23 12:41:33.003604 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application rv = self.dispatch_request() May 23 12:41:33.003604 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request May 23 12:41:33.003604 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) May 23 12:41:33.003604 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper May 23 12:41:33.003604 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) May 23 12:41:33.003604 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view May 23 12:41:33.003604 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) May 23 12:41:33.003604 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request May 23 12:41:33.003604 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) May 23 12:41:33.003604 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get May 23 12:41:33.003604 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self._get_user(user_id) May 23 12:41:33.003604 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user May 23 12:41:33.003604 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) May 23 12:41:33.004548 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped May 23 12:41:33.004548 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) May 23 12:41:33.004548 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper May 23 12:41:33.004548 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) May 23 12:41:33.004548 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper May 23 12:41:33.004548 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) May 23 12:41:33.004548 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun May 23 12:41:33.004548 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) May 23 12:41:33.004548 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func May 23 12:41:33.004548 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self.get_or_create( May 23 12:41:33.004548 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create May 23 12:41:33.004548 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application with Lock( May 23 12:41:33.004548 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ May 23 12:41:33.004548 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self._enter() May 23 12:41:33.004548 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter May 23 12:41:33.004548 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) May 23 12:41:33.004548 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create May 23 12:41:33.004548 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self.creator() May 23 12:41:33.005709 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value May 23 12:41:33.005709 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application created_value = creator( May 23 12:41:33.005709 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user May 23 12:41:33.005709 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) May 23 12:41:33.005709 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user May 23 12:41:33.005709 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) May 23 12:41:33.005709 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user May 23 12:41:33.005709 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) May 23 12:41:33.005709 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: swift. May 23 12:41:33.005709 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application May 23 12:41:33.007371 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 315/616] 10.222.0.30 () {56 vars in 1102 bytes} [Mon May 23 12:41:32 2022] GET /identity/v3/roles/service => generated 85 bytes in 293 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) May 23 12:41:33.009063 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 302/617] 10.222.0.30 () {56 vars in 1096 bytes} [Mon May 23 12:41:32 2022] GET /identity/v3/users/swift => generated 83 bytes in 283 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) May 23 12:41:33.013560 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6103d6ae-17c4-4801-871f-5e943b2e7447 None admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:41:33.014714 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0084094f-4090-4f30-90c1-ad7a520c0c53 None admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:41:33.158803 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0084094f-4090-4f30-90c1-ad7a520c0c53 None admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:41:33.219540 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=100 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:41:33.231972 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6103d6ae-17c4-4801-871f-5e943b2e7447 None admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:41:33.239995 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d6ef0c3-c93e-4fc9-a926-0005e26537f9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:41:33.240674 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d6ef0c3-c93e-4fc9-a926-0005e26537f9 None admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:41:33.241498 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d6ef0c3-c93e-4fc9-a926-0005e26537f9 None admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:41:33.241771 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d6ef0c3-c93e-4fc9-a926-0005e26537f9 None admin] PATH_INFO: `/v3/users` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:41:33.242917 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d6ef0c3-c93e-4fc9-a926-0005e26537f9 None admin] RBAC: Adding query filter params (domain_id=default, name=swift) {{(pid=92057) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} May 23 12:41:33.243231 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d6ef0c3-c93e-4fc9-a926-0005e26537f9 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:41:33.250948 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d6ef0c3-c93e-4fc9-a926-0005e26537f9 None admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:41:33.281143 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 303/618] 10.222.0.30 () {56 vars in 1135 bytes} [Mon May 23 12:41:33 2022] GET /identity/v3/users?domain_id=default&name=swift => generated 370 bytes in 267 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) May 23 12:41:33.283717 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be058484-56b9-4bdd-be03-baa0968629a0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:41:33.284627 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be058484-56b9-4bdd-be03-baa0968629a0 None admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:41:33.285106 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be058484-56b9-4bdd-be03-baa0968629a0 None admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:41:33.285672 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be058484-56b9-4bdd-be03-baa0968629a0 None admin] PATH_INFO: `/v3/roles` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:41:33.286648 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be058484-56b9-4bdd-be03-baa0968629a0 None admin] RBAC: Adding query filter params (name=service) {{(pid=92056) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} May 23 12:41:33.288347 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d6ef0c3-c93e-4fc9-a926-0005e26537f9 None admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:41:33.289437 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be058484-56b9-4bdd-be03-baa0968629a0 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:41:33.291055 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be058484-56b9-4bdd-be03-baa0968629a0 None admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:41:33.309711 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 316/619] 10.222.0.30 () {56 vars in 1103 bytes} [Mon May 23 12:41:33 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 298 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) May 23 12:41:33.312710 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be058484-56b9-4bdd-be03-baa0968629a0 None admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:41:33.340166 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=100 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:41:33.449676 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d6ef0c3-c93e-4fc9-a926-0005e26537f9 None admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:41:33.458944 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be058484-56b9-4bdd-be03-baa0968629a0 None admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:41:33.499312 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08573d0a-94e2-4159-a634-7991a5341014 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:41:33.501104 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08573d0a-94e2-4159-a634-7991a5341014 None admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:41:33.501609 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08573d0a-94e2-4159-a634-7991a5341014 None admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:41:33.502105 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08573d0a-94e2-4159-a634-7991a5341014 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:41:33.503918 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5cd336a1-6863-4b87-9de6-30b8eaa7a788 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:41:33.505031 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cd336a1-6863-4b87-9de6-30b8eaa7a788 None admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:41:33.505324 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cd336a1-6863-4b87-9de6-30b8eaa7a788 None admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:41:33.505593 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cd336a1-6863-4b87-9de6-30b8eaa7a788 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:41:33.516947 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08573d0a-94e2-4159-a634-7991a5341014 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:41:33.519396 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5cd336a1-6863-4b87-9de6-30b8eaa7a788 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:41:33.519781 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08573d0a-94e2-4159-a634-7991a5341014 None admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:41:33.520841 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5cd336a1-6863-4b87-9de6-30b8eaa7a788 None admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:41:33.541805 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application [None req-5cd336a1-6863-4b87-9de6-30b8eaa7a788 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. May 23 12:41:33.541805 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application Traceback (most recent call last): May 23 12:41:33.541805 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request May 23 12:41:33.541805 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application rv = self.dispatch_request() May 23 12:41:33.541805 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request May 23 12:41:33.541805 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) May 23 12:41:33.541805 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper May 23 12:41:33.541805 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) May 23 12:41:33.541805 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view May 23 12:41:33.541805 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) May 23 12:41:33.541805 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request May 23 12:41:33.541805 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) May 23 12:41:33.541805 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get May 23 12:41:33.541805 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self._get_domain(domain_id) May 23 12:41:33.541805 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain May 23 12:41:33.542818 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) May 23 12:41:33.542818 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped May 23 12:41:33.542818 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) May 23 12:41:33.542818 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun May 23 12:41:33.542818 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) May 23 12:41:33.542818 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func May 23 12:41:33.542818 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self.get_or_create( May 23 12:41:33.542818 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create May 23 12:41:33.542818 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application with Lock( May 23 12:41:33.542818 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ May 23 12:41:33.542818 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self._enter() May 23 12:41:33.542818 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter May 23 12:41:33.542818 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) May 23 12:41:33.542818 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create May 23 12:41:33.542818 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self.creator() May 23 12:41:33.542818 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value May 23 12:41:33.542818 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application created_value = creator( May 23 12:41:33.544352 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain May 23 12:41:33.544352 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) May 23 12:41:33.544352 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. May 23 12:41:33.544352 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application May 23 12:41:33.544352 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 317/620] 10.222.0.30 () {56 vars in 1108 bytes} [Mon May 23 12:41:33 2022] GET /identity/v3/domains/Default => generated 87 bytes in 231 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) May 23 12:41:33.544784 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de4bd5f8-f519-4e43-92df-a869e9be88b3 None None] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:41:33.545072 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de4bd5f8-f519-4e43-92df-a869e9be88b3 None None] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:41:33.545349 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de4bd5f8-f519-4e43-92df-a869e9be88b3 None None] PATH_INFO: `/` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:41:33.546978 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application [None req-08573d0a-94e2-4159-a634-7991a5341014 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. May 23 12:41:33.546978 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application Traceback (most recent call last): May 23 12:41:33.546978 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request May 23 12:41:33.546978 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application rv = self.dispatch_request() May 23 12:41:33.546978 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request May 23 12:41:33.546978 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) May 23 12:41:33.546978 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper May 23 12:41:33.546978 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) May 23 12:41:33.546978 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view May 23 12:41:33.546978 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) May 23 12:41:33.546978 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request May 23 12:41:33.546978 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) May 23 12:41:33.546978 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get May 23 12:41:33.546978 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self._get_domain(domain_id) May 23 12:41:33.546978 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain May 23 12:41:33.548269 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) May 23 12:41:33.548269 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped May 23 12:41:33.548269 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) May 23 12:41:33.548269 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun May 23 12:41:33.548269 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) May 23 12:41:33.548269 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func May 23 12:41:33.548269 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self.get_or_create( May 23 12:41:33.548269 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create May 23 12:41:33.548269 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application with Lock( May 23 12:41:33.548269 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ May 23 12:41:33.548269 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self._enter() May 23 12:41:33.548269 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter May 23 12:41:33.548269 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) May 23 12:41:33.548269 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create May 23 12:41:33.548269 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self.creator() May 23 12:41:33.548269 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value May 23 12:41:33.548269 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application created_value = creator( May 23 12:41:33.549172 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain May 23 12:41:33.549172 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) May 23 12:41:33.549172 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. May 23 12:41:33.549172 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application May 23 12:41:33.550663 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 318/621] 10.222.0.30 () {54 vars in 926 bytes} [Mon May 23 12:41:33 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) May 23 12:41:33.551459 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de4bd5f8-f519-4e43-92df-a869e9be88b3 None None] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:41:33.551692 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 304/622] 10.222.0.30 () {56 vars in 1108 bytes} [Mon May 23 12:41:33 2022] GET /identity/v3/domains/Default => generated 87 bytes in 264 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) May 23 12:41:33.553296 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-288bec3f-35e9-4a57-9f62-0f45722ad88c None None] REQUEST_METHOD: `POST` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:41:33.554680 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-288bec3f-35e9-4a57-9f62-0f45722ad88c None None] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:41:33.556254 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-288bec3f-35e9-4a57-9f62-0f45722ad88c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:41:33.577828 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.auth.core [None req-288bec3f-35e9-4a57-9f62-0f45722ad88c None None] MFA Rules not processed for user `0f7119f8fa5c4445977671140ded91a4`. Rule list: `[]` (Enabled: `True`). {{(pid=92057) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 12:41:33.579914 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.fernet_utils [None req-288bec3f-35e9-4a57-9f62-0f45722ad88c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92057) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 12:41:33.621833 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=100 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:41:33.628191 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=100 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:41:33.674250 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 305/623] 10.222.0.30 () {58 vars in 1023 bytes} [Mon May 23 12:41:33 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 121 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) May 23 12:41:33.675568 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-288bec3f-35e9-4a57-9f62-0f45722ad88c None None] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:41:33.686404 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de4bd5f8-f519-4e43-92df-a869e9be88b3 None None] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:41:33.739365 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93f46c7e-396c-416b-9e4b-b22680dc78ef None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:41:33.740033 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93f46c7e-396c-416b-9e4b-b22680dc78ef None admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:41:33.740318 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93f46c7e-396c-416b-9e4b-b22680dc78ef None admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:41:33.740597 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93f46c7e-396c-416b-9e4b-b22680dc78ef None admin] PATH_INFO: `/v3/domains` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:41:33.741353 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93f46c7e-396c-416b-9e4b-b22680dc78ef None admin] RBAC: Adding query filter params (name=Default) {{(pid=92056) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} May 23 12:41:33.741661 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93f46c7e-396c-416b-9e4b-b22680dc78ef None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:41:33.743112 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93f46c7e-396c-416b-9e4b-b22680dc78ef None admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:41:33.755772 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 319/624] 10.222.0.30 () {56 vars in 1109 bytes} [Mon May 23 12:41:33 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 204 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) May 23 12:41:33.757219 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93f46c7e-396c-416b-9e4b-b22680dc78ef None admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:41:33.794857 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-288bec3f-35e9-4a57-9f62-0f45722ad88c None None] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:41:33.831302 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b760f898-8dff-4634-b839-2fc7e880b274 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:41:33.831969 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b760f898-8dff-4634-b839-2fc7e880b274 None admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:41:33.832250 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b760f898-8dff-4634-b839-2fc7e880b274 None admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:41:33.832527 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b760f898-8dff-4634-b839-2fc7e880b274 None admin] PATH_INFO: `/v3/domains` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:41:33.833240 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b760f898-8dff-4634-b839-2fc7e880b274 None admin] RBAC: Adding query filter params (name=Default) {{(pid=92057) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} May 23 12:41:33.833543 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b760f898-8dff-4634-b839-2fc7e880b274 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:41:33.835774 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b760f898-8dff-4634-b839-2fc7e880b274 None admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:41:33.848227 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 306/625] 10.222.0.30 () {56 vars in 1109 bytes} [Mon May 23 12:41:33 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 173 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) May 23 12:41:33.849138 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b760f898-8dff-4634-b839-2fc7e880b274 None admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:41:33.864782 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93f46c7e-396c-416b-9e4b-b22680dc78ef None admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:41:33.878546 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=100 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:41:33.890438 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=100 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:41:33.903920 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-498bc130-5d39-48f2-b4c0-bce14dc4406c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:41:33.904976 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-498bc130-5d39-48f2-b4c0-bce14dc4406c None admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:41:33.905408 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-498bc130-5d39-48f2-b4c0-bce14dc4406c None admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:41:33.905827 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-498bc130-5d39-48f2-b4c0-bce14dc4406c None admin] PATH_INFO: `/v3/roles/service` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:41:33.919910 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-498bc130-5d39-48f2-b4c0-bce14dc4406c None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} May 23 12:41:33.931314 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application [None req-498bc130-5d39-48f2-b4c0-bce14dc4406c None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. May 23 12:41:33.931314 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application Traceback (most recent call last): May 23 12:41:33.931314 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request May 23 12:41:33.931314 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application rv = self.dispatch_request() May 23 12:41:33.931314 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request May 23 12:41:33.931314 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) May 23 12:41:33.931314 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper May 23 12:41:33.931314 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) May 23 12:41:33.931314 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view May 23 12:41:33.931314 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) May 23 12:41:33.931314 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request May 23 12:41:33.931314 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) May 23 12:41:33.931314 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get May 23 12:41:33.931314 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self._get_role(role_id) May 23 12:41:33.931314 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role May 23 12:41:33.931314 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') May 23 12:41:33.932401 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call May 23 12:41:33.932401 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( May 23 12:41:33.932401 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data May 23 12:41:33.932401 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) May 23 12:41:33.932401 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped May 23 12:41:33.932401 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) May 23 12:41:33.932401 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun May 23 12:41:33.932401 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) May 23 12:41:33.932401 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func May 23 12:41:33.932401 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self.get_or_create( May 23 12:41:33.932401 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create May 23 12:41:33.932401 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application with Lock( May 23 12:41:33.932401 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ May 23 12:41:33.932401 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self._enter() May 23 12:41:33.932401 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter May 23 12:41:33.932401 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) May 23 12:41:33.933290 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create May 23 12:41:33.933290 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self.creator() May 23 12:41:33.933290 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value May 23 12:41:33.933290 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application created_value = creator( May 23 12:41:33.933290 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role May 23 12:41:33.933290 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) May 23 12:41:33.933290 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role May 23 12:41:33.933290 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() May 23 12:41:33.933290 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role May 23 12:41:33.933290 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) May 23 12:41:33.933290 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. May 23 12:41:33.933290 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application May 23 12:41:33.941080 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 320/626] 10.222.0.30 () {56 vars in 1102 bytes} [Mon May 23 12:41:33 2022] GET /identity/v3/roles/service => generated 85 bytes in 180 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) May 23 12:41:33.942715 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-498bc130-5d39-48f2-b4c0-bce14dc4406c None admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:41:33.967613 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b760f898-8dff-4634-b839-2fc7e880b274 None admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:41:34.004944 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5167cd4-6aff-42a1-bf67-f6c04d252f16 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:41:34.006050 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5167cd4-6aff-42a1-bf67-f6c04d252f16 None admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:41:34.006442 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5167cd4-6aff-42a1-bf67-f6c04d252f16 None admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:41:34.007337 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5167cd4-6aff-42a1-bf67-f6c04d252f16 None admin] PATH_INFO: `/v3/users/nova` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:41:34.015890 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a5167cd4-6aff-42a1-bf67-f6c04d252f16 None admin] RBAC: Authorizing `identity:get_user(user_id=nova)` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:41:34.017598 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a5167cd4-6aff-42a1-bf67-f6c04d252f16 None admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:41:34.034023 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application [None req-a5167cd4-6aff-42a1-bf67-f6c04d252f16 None admin] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova. May 23 12:41:34.034023 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application Traceback (most recent call last): May 23 12:41:34.034023 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request May 23 12:41:34.034023 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application rv = self.dispatch_request() May 23 12:41:34.034023 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request May 23 12:41:34.034023 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) May 23 12:41:34.034023 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper May 23 12:41:34.034023 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) May 23 12:41:34.034023 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view May 23 12:41:34.034023 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) May 23 12:41:34.034023 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request May 23 12:41:34.034023 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) May 23 12:41:34.034023 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get May 23 12:41:34.034023 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self._get_user(user_id) May 23 12:41:34.034023 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user May 23 12:41:34.034023 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) May 23 12:41:34.035209 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped May 23 12:41:34.035209 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) May 23 12:41:34.035209 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper May 23 12:41:34.035209 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) May 23 12:41:34.035209 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper May 23 12:41:34.035209 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) May 23 12:41:34.035209 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun May 23 12:41:34.035209 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) May 23 12:41:34.035209 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func May 23 12:41:34.035209 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self.get_or_create( May 23 12:41:34.035209 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create May 23 12:41:34.035209 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application with Lock( May 23 12:41:34.035209 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ May 23 12:41:34.035209 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self._enter() May 23 12:41:34.035209 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter May 23 12:41:34.035209 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) May 23 12:41:34.035209 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create May 23 12:41:34.035209 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self.creator() May 23 12:41:34.036416 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value May 23 12:41:34.036416 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application created_value = creator( May 23 12:41:34.036416 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user May 23 12:41:34.036416 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) May 23 12:41:34.036416 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user May 23 12:41:34.036416 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) May 23 12:41:34.036416 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user May 23 12:41:34.036416 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) May 23 12:41:34.036416 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: nova. May 23 12:41:34.036416 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application May 23 12:41:34.038869 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 307/627] 10.222.0.30 () {56 vars in 1093 bytes} [Mon May 23 12:41:33 2022] GET /identity/v3/users/nova => generated 82 bytes in 190 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) May 23 12:41:34.039841 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5167cd4-6aff-42a1-bf67-f6c04d252f16 None admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:41:34.078070 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-498bc130-5d39-48f2-b4c0-bce14dc4406c None admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:41:34.130316 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef754865-8c40-4602-81be-544e9e26f854 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:41:34.132109 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef754865-8c40-4602-81be-544e9e26f854 None admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:41:34.132546 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef754865-8c40-4602-81be-544e9e26f854 None admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:41:34.132975 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef754865-8c40-4602-81be-544e9e26f854 None admin] PATH_INFO: `/v3/projects/service` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:41:34.138612 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=100 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:41:34.143365 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef754865-8c40-4602-81be-544e9e26f854 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:41:34.145549 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef754865-8c40-4602-81be-544e9e26f854 None admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:41:34.156945 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5167cd4-6aff-42a1-bf67-f6c04d252f16 None admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:41:34.167659 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application [None req-ef754865-8c40-4602-81be-544e9e26f854 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. May 23 12:41:34.167659 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application Traceback (most recent call last): May 23 12:41:34.167659 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request May 23 12:41:34.167659 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application rv = self.dispatch_request() May 23 12:41:34.167659 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request May 23 12:41:34.167659 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) May 23 12:41:34.167659 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper May 23 12:41:34.167659 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) May 23 12:41:34.167659 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view May 23 12:41:34.167659 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) May 23 12:41:34.167659 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request May 23 12:41:34.167659 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) May 23 12:41:34.167659 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get May 23 12:41:34.167659 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self._get_project(project_id) May 23 12:41:34.167659 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project May 23 12:41:34.168859 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) May 23 12:41:34.168859 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped May 23 12:41:34.168859 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) May 23 12:41:34.168859 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun May 23 12:41:34.168859 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) May 23 12:41:34.168859 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func May 23 12:41:34.168859 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self.get_or_create( May 23 12:41:34.168859 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create May 23 12:41:34.168859 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application with Lock( May 23 12:41:34.168859 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ May 23 12:41:34.168859 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self._enter() May 23 12:41:34.168859 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter May 23 12:41:34.168859 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) May 23 12:41:34.168859 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create May 23 12:41:34.168859 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self.creator() May 23 12:41:34.168859 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value May 23 12:41:34.168859 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application created_value = creator( May 23 12:41:34.170030 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project May 23 12:41:34.170030 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) May 23 12:41:34.170030 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project May 23 12:41:34.170030 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() May 23 12:41:34.170030 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project May 23 12:41:34.170030 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) May 23 12:41:34.170030 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. May 23 12:41:34.170030 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application May 23 12:41:34.173497 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 321/628] 10.222.0.30 () {56 vars in 1111 bytes} [Mon May 23 12:41:33 2022] GET /identity/v3/projects/service => generated 88 bytes in 231 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) May 23 12:41:34.175014 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef754865-8c40-4602-81be-544e9e26f854 None admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:41:34.201699 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93429457-a193-4bab-b6ee-b94a83e0046a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:41:34.202575 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93429457-a193-4bab-b6ee-b94a83e0046a None admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:41:34.202953 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93429457-a193-4bab-b6ee-b94a83e0046a None admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:41:34.203328 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93429457-a193-4bab-b6ee-b94a83e0046a None admin] PATH_INFO: `/v3/roles` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:41:34.204381 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93429457-a193-4bab-b6ee-b94a83e0046a None admin] RBAC: Adding query filter params (name=service) {{(pid=92057) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} May 23 12:41:34.204608 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93429457-a193-4bab-b6ee-b94a83e0046a None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:41:34.206135 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93429457-a193-4bab-b6ee-b94a83e0046a None admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:41:34.220116 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 308/629] 10.222.0.30 () {56 vars in 1103 bytes} [Mon May 23 12:41:34 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 176 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) May 23 12:41:34.221413 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93429457-a193-4bab-b6ee-b94a83e0046a None admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:41:34.235301 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=100 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:41:34.295356 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef754865-8c40-4602-81be-544e9e26f854 None admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:41:34.333378 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b106fae-aeb1-4cdd-8ad7-470baf332736 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:41:34.334294 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b106fae-aeb1-4cdd-8ad7-470baf332736 None admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:41:34.335536 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b106fae-aeb1-4cdd-8ad7-470baf332736 None admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:41:34.335839 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b106fae-aeb1-4cdd-8ad7-470baf332736 None admin] PATH_INFO: `/v3/users` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:41:34.336613 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b106fae-aeb1-4cdd-8ad7-470baf332736 None admin] RBAC: Adding query filter params (domain_id=default, name=nova) {{(pid=92056) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} May 23 12:41:34.337015 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b106fae-aeb1-4cdd-8ad7-470baf332736 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:41:34.338684 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b106fae-aeb1-4cdd-8ad7-470baf332736 None admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:41:34.356609 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93429457-a193-4bab-b6ee-b94a83e0046a None admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:41:34.360455 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 322/630] 10.222.0.30 () {56 vars in 1133 bytes} [Mon May 23 12:41:34 2022] GET /identity/v3/users?domain_id=default&name=nova => generated 368 bytes in 186 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) May 23 12:41:34.361322 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b106fae-aeb1-4cdd-8ad7-470baf332736 None admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:41:34.407934 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-703098d2-0ab4-4dbd-8377-3ffb5fe6cbbe None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:41:34.409229 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-703098d2-0ab4-4dbd-8377-3ffb5fe6cbbe None admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:41:34.409514 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-703098d2-0ab4-4dbd-8377-3ffb5fe6cbbe None admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:41:34.409860 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-703098d2-0ab4-4dbd-8377-3ffb5fe6cbbe None admin] PATH_INFO: `/v3/projects` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:41:34.411039 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-703098d2-0ab4-4dbd-8377-3ffb5fe6cbbe None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=92057) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} May 23 12:41:34.411355 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-703098d2-0ab4-4dbd-8377-3ffb5fe6cbbe None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:41:34.413159 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-703098d2-0ab4-4dbd-8377-3ffb5fe6cbbe None admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:41:34.430235 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 309/631] 10.222.0.30 () {56 vars in 1148 bytes} [Mon May 23 12:41:34 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 205 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) May 23 12:41:34.430235 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9a6ff78-8e1f-490b-b92c-699225325824 None None] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:41:34.430235 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9a6ff78-8e1f-490b-b92c-699225325824 None None] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:41:34.430235 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9a6ff78-8e1f-490b-b92c-699225325824 None None] PATH_INFO: `/` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:41:34.430235 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 310/632] 10.222.0.30 () {54 vars in 926 bytes} [Mon May 23 12:41:34 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) May 23 12:41:34.430235 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9a6ff78-8e1f-490b-b92c-699225325824 None None] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:41:34.441113 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=100 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:41:34.479786 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b106fae-aeb1-4cdd-8ad7-470baf332736 None admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:41:34.481826 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=100 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:41:34.514854 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2bda77c4-6cdb-466a-ba0c-c5b125819588 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:41:34.515647 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bda77c4-6cdb-466a-ba0c-c5b125819588 None admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:41:34.515933 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bda77c4-6cdb-466a-ba0c-c5b125819588 None admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:41:34.516212 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bda77c4-6cdb-466a-ba0c-c5b125819588 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:41:34.526954 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2bda77c4-6cdb-466a-ba0c-c5b125819588 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:41:34.528718 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2bda77c4-6cdb-466a-ba0c-c5b125819588 None admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:41:34.539542 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9a6ff78-8e1f-490b-b92c-699225325824 None None] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:41:34.547262 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application [None req-2bda77c4-6cdb-466a-ba0c-c5b125819588 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. May 23 12:41:34.547262 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application Traceback (most recent call last): May 23 12:41:34.547262 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request May 23 12:41:34.547262 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application rv = self.dispatch_request() May 23 12:41:34.547262 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request May 23 12:41:34.547262 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) May 23 12:41:34.547262 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper May 23 12:41:34.547262 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) May 23 12:41:34.547262 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view May 23 12:41:34.547262 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) May 23 12:41:34.547262 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request May 23 12:41:34.547262 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) May 23 12:41:34.547262 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get May 23 12:41:34.547262 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self._get_domain(domain_id) May 23 12:41:34.547262 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain May 23 12:41:34.548180 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) May 23 12:41:34.548180 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped May 23 12:41:34.548180 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) May 23 12:41:34.548180 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun May 23 12:41:34.548180 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) May 23 12:41:34.548180 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func May 23 12:41:34.548180 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self.get_or_create( May 23 12:41:34.548180 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create May 23 12:41:34.548180 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application with Lock( May 23 12:41:34.548180 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ May 23 12:41:34.548180 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self._enter() May 23 12:41:34.548180 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter May 23 12:41:34.548180 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) May 23 12:41:34.548180 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create May 23 12:41:34.548180 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self.creator() May 23 12:41:34.548180 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value May 23 12:41:34.548180 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application created_value = creator( May 23 12:41:34.549699 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain May 23 12:41:34.549699 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) May 23 12:41:34.549699 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. May 23 12:41:34.549699 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application May 23 12:41:34.549699 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 323/633] 10.222.0.30 () {56 vars in 1108 bytes} [Mon May 23 12:41:34 2022] GET /identity/v3/domains/Default => generated 87 bytes in 188 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) May 23 12:41:34.549699 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2bda77c4-6cdb-466a-ba0c-c5b125819588 None admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:41:34.580717 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b9f81fd-5085-4780-a7ea-4e2ca6bb4516 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:41:34.581368 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b9f81fd-5085-4780-a7ea-4e2ca6bb4516 None admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:41:34.581658 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b9f81fd-5085-4780-a7ea-4e2ca6bb4516 None admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:41:34.581977 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b9f81fd-5085-4780-a7ea-4e2ca6bb4516 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:41:34.650215 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b9f81fd-5085-4780-a7ea-4e2ca6bb4516 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:41:34.651678 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b9f81fd-5085-4780-a7ea-4e2ca6bb4516 None admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:41:34.652909 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2bda77c4-6cdb-466a-ba0c-c5b125819588 None admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:41:34.661285 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=100 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:41:34.688776 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae5d595c-d718-4cec-b1d5-7a80cb6cf2c1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:41:34.689675 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae5d595c-d718-4cec-b1d5-7a80cb6cf2c1 None admin] REQUEST_METHOD: `PUT` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:41:34.689994 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae5d595c-d718-4cec-b1d5-7a80cb6cf2c1 None admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:41:34.690273 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae5d595c-d718-4cec-b1d5-7a80cb6cf2c1 None admin] PATH_INFO: `/v3/projects/9c389ae65aaa4542bb5a5c4fd0bacf5b/users/138d91edb99f450e846765f9eef00d62/roles/41ab28c1ae3d4941bd09fb5d35f1dc54` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:41:34.705338 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae5d595c-d718-4cec-b1d5-7a80cb6cf2c1 None admin] RBAC: Authorizing `identity:create_grant(project_id=9c389ae65aaa4542bb5a5c4fd0bacf5b, user_id=138d91edb99f450e846765f9eef00d62, role_id=41ab28c1ae3d4941bd09fb5d35f1dc54)` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:41:34.706831 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae5d595c-d718-4cec-b1d5-7a80cb6cf2c1 None admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:41:34.749998 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 311/634] 10.222.0.30 () {58 vars in 1282 bytes} [Mon May 23 12:41:34 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 320 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) May 23 12:41:34.751290 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc7500dd-2bf6-4eb5-a316-e5a32bfdb7e5 None None] REQUEST_METHOD: `POST` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:41:34.751582 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc7500dd-2bf6-4eb5-a316-e5a32bfdb7e5 None None] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:41:34.751863 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc7500dd-2bf6-4eb5-a316-e5a32bfdb7e5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:41:34.768706 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.auth.core [None req-dc7500dd-2bf6-4eb5-a316-e5a32bfdb7e5 None None] MFA Rules not processed for user `0f7119f8fa5c4445977671140ded91a4`. Rule list: `[]` (Enabled: `True`). {{(pid=92057) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 12:41:34.769900 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.fernet_utils [None req-dc7500dd-2bf6-4eb5-a316-e5a32bfdb7e5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92057) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 12:41:34.787790 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 324/635] 10.222.0.30 () {58 vars in 1439 bytes} [Mon May 23 12:41:34 2022] PUT /identity/v3/projects/9c389ae65aaa4542bb5a5c4fd0bacf5b/users/138d91edb99f450e846765f9eef00d62/roles/41ab28c1ae3d4941bd09fb5d35f1dc54 => generated 0 bytes in 238 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) May 23 12:41:34.788612 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae5d595c-d718-4cec-b1d5-7a80cb6cf2c1 None admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:41:34.792160 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:41:34.852019 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 312/636] 10.222.0.30 () {58 vars in 1023 bytes} [Mon May 23 12:41:34 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 101 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) May 23 12:41:34.853460 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9c8190d-a875-44f8-9cde-6f65af744797 None None] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:41:34.853751 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9c8190d-a875-44f8-9cde-6f65af744797 None None] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:41:34.854070 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9c8190d-a875-44f8-9cde-6f65af744797 None None] PATH_INFO: `/` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:41:34.857951 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 313/637] 10.222.0.30 () {54 vars in 926 bytes} [Mon May 23 12:41:34 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) May 23 12:41:34.857951 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9c8190d-a875-44f8-9cde-6f65af744797 None None] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:41:34.914742 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=100 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:41:34.922546 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae5d595c-d718-4cec-b1d5-7a80cb6cf2c1 None admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:41:34.960682 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d97faa59-8212-459b-a594-1f6b3d97967b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:41:34.961363 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d97faa59-8212-459b-a594-1f6b3d97967b None admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:41:34.961644 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d97faa59-8212-459b-a594-1f6b3d97967b None admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:41:34.961941 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d97faa59-8212-459b-a594-1f6b3d97967b None admin] PATH_INFO: `/v3/domains` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:41:34.962674 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d97faa59-8212-459b-a594-1f6b3d97967b None admin] RBAC: Adding query filter params (name=Default) {{(pid=92056) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} May 23 12:41:34.963263 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d97faa59-8212-459b-a594-1f6b3d97967b None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:41:34.964361 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d97faa59-8212-459b-a594-1f6b3d97967b None admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:41:34.980116 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 325/638] 10.222.0.30 () {56 vars in 1109 bytes} [Mon May 23 12:41:34 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 192 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) May 23 12:41:34.981467 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e70150d9-dc38-4a9c-9aec-b131ced62612 None None] REQUEST_METHOD: `POST` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:41:34.981755 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e70150d9-dc38-4a9c-9aec-b131ced62612 None None] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:41:34.982070 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e70150d9-dc38-4a9c-9aec-b131ced62612 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:41:34.992409 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9c8190d-a875-44f8-9cde-6f65af744797 None None] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:41:35.001129 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.auth.core [None req-e70150d9-dc38-4a9c-9aec-b131ced62612 None None] MFA Rules not processed for user `0f7119f8fa5c4445977671140ded91a4`. Rule list: `[]` (Enabled: `True`). {{(pid=92056) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 12:41:35.002755 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.fernet_utils [None req-e70150d9-dc38-4a9c-9aec-b131ced62612 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92056) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 12:41:35.031362 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5642bd54-c2ca-4da3-b527-c5025c938265 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:41:35.032147 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5642bd54-c2ca-4da3-b527-c5025c938265 None admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:41:35.032452 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5642bd54-c2ca-4da3-b527-c5025c938265 None admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:41:35.032726 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5642bd54-c2ca-4da3-b527-c5025c938265 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:41:35.044819 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5642bd54-c2ca-4da3-b527-c5025c938265 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:41:35.046472 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5642bd54-c2ca-4da3-b527-c5025c938265 None admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:41:35.063723 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=100 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:41:35.065661 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application [None req-5642bd54-c2ca-4da3-b527-c5025c938265 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. May 23 12:41:35.065661 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application Traceback (most recent call last): May 23 12:41:35.065661 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request May 23 12:41:35.065661 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application rv = self.dispatch_request() May 23 12:41:35.065661 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request May 23 12:41:35.065661 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) May 23 12:41:35.065661 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper May 23 12:41:35.065661 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) May 23 12:41:35.065661 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view May 23 12:41:35.065661 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) May 23 12:41:35.065661 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request May 23 12:41:35.065661 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) May 23 12:41:35.065661 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get May 23 12:41:35.065661 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self._get_domain(domain_id) May 23 12:41:35.065661 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain May 23 12:41:35.066765 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) May 23 12:41:35.066765 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped May 23 12:41:35.066765 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) May 23 12:41:35.066765 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun May 23 12:41:35.066765 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) May 23 12:41:35.066765 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func May 23 12:41:35.066765 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self.get_or_create( May 23 12:41:35.066765 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create May 23 12:41:35.066765 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application with Lock( May 23 12:41:35.066765 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ May 23 12:41:35.066765 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self._enter() May 23 12:41:35.066765 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter May 23 12:41:35.066765 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) May 23 12:41:35.066765 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create May 23 12:41:35.066765 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self.creator() May 23 12:41:35.066765 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value May 23 12:41:35.066765 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application created_value = creator( May 23 12:41:35.067746 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain May 23 12:41:35.067746 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) May 23 12:41:35.067746 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. May 23 12:41:35.067746 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application May 23 12:41:35.067746 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 314/639] 10.222.0.30 () {56 vars in 1108 bytes} [Mon May 23 12:41:34 2022] GET /identity/v3/domains/Default => generated 87 bytes in 211 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) May 23 12:41:35.067746 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5642bd54-c2ca-4da3-b527-c5025c938265 None admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:41:35.076868 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 326/640] 10.222.0.30 () {58 vars in 1023 bytes} [Mon May 23 12:41:34 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 96 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) May 23 12:41:35.078015 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e70150d9-dc38-4a9c-9aec-b131ced62612 None None] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:41:35.170270 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=100 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:41:35.180753 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5642bd54-c2ca-4da3-b527-c5025c938265 None admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:41:35.191967 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e70150d9-dc38-4a9c-9aec-b131ced62612 None None] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:41:35.222546 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80b851bf-e809-4bb5-a5a5-5079b860ca7b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:41:35.223369 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80b851bf-e809-4bb5-a5a5-5079b860ca7b None admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:41:35.223650 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80b851bf-e809-4bb5-a5a5-5079b860ca7b None admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:41:35.223924 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80b851bf-e809-4bb5-a5a5-5079b860ca7b None admin] PATH_INFO: `/v3/roles/service` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:41:35.230030 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-441ecc1d-74ad-4a14-8c51-8495c1b4e160 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:41:35.230833 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-441ecc1d-74ad-4a14-8c51-8495c1b4e160 None admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:41:35.231117 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-441ecc1d-74ad-4a14-8c51-8495c1b4e160 None admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:41:35.231395 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-441ecc1d-74ad-4a14-8c51-8495c1b4e160 None admin] PATH_INFO: `/v3/users/glance` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:41:35.234843 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80b851bf-e809-4bb5-a5a5-5079b860ca7b None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} May 23 12:41:35.238972 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-441ecc1d-74ad-4a14-8c51-8495c1b4e160 None admin] RBAC: Authorizing `identity:get_user(user_id=glance)` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:41:35.240408 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-441ecc1d-74ad-4a14-8c51-8495c1b4e160 None admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:41:35.243285 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application [None req-80b851bf-e809-4bb5-a5a5-5079b860ca7b None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. May 23 12:41:35.243285 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application Traceback (most recent call last): May 23 12:41:35.243285 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request May 23 12:41:35.243285 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application rv = self.dispatch_request() May 23 12:41:35.243285 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request May 23 12:41:35.243285 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) May 23 12:41:35.243285 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper May 23 12:41:35.243285 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) May 23 12:41:35.243285 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view May 23 12:41:35.243285 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) May 23 12:41:35.243285 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request May 23 12:41:35.243285 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) May 23 12:41:35.243285 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get May 23 12:41:35.243285 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self._get_role(role_id) May 23 12:41:35.243285 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role May 23 12:41:35.243285 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') May 23 12:41:35.244367 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call May 23 12:41:35.244367 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( May 23 12:41:35.244367 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data May 23 12:41:35.244367 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) May 23 12:41:35.244367 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped May 23 12:41:35.244367 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) May 23 12:41:35.244367 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun May 23 12:41:35.244367 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) May 23 12:41:35.244367 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func May 23 12:41:35.244367 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self.get_or_create( May 23 12:41:35.244367 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create May 23 12:41:35.244367 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application with Lock( May 23 12:41:35.244367 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ May 23 12:41:35.244367 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self._enter() May 23 12:41:35.244367 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter May 23 12:41:35.244367 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) May 23 12:41:35.245287 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create May 23 12:41:35.245287 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self.creator() May 23 12:41:35.245287 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value May 23 12:41:35.245287 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application created_value = creator( May 23 12:41:35.245287 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role May 23 12:41:35.245287 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) May 23 12:41:35.245287 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role May 23 12:41:35.245287 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() May 23 12:41:35.245287 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role May 23 12:41:35.245287 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) May 23 12:41:35.245287 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. May 23 12:41:35.245287 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application May 23 12:41:35.245287 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 315/641] 10.222.0.30 () {56 vars in 1102 bytes} [Mon May 23 12:41:35 2022] GET /identity/v3/roles/service => generated 85 bytes in 177 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) May 23 12:41:35.245287 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80b851bf-e809-4bb5-a5a5-5079b860ca7b None admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:41:35.261236 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application [None req-441ecc1d-74ad-4a14-8c51-8495c1b4e160 None admin] Could not find user: glance.: keystone.exception.UserNotFound: Could not find user: glance. May 23 12:41:35.261236 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application Traceback (most recent call last): May 23 12:41:35.261236 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request May 23 12:41:35.261236 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application rv = self.dispatch_request() May 23 12:41:35.261236 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request May 23 12:41:35.261236 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) May 23 12:41:35.261236 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper May 23 12:41:35.261236 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) May 23 12:41:35.261236 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view May 23 12:41:35.261236 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) May 23 12:41:35.261236 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request May 23 12:41:35.261236 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) May 23 12:41:35.261236 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get May 23 12:41:35.261236 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self._get_user(user_id) May 23 12:41:35.261236 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user May 23 12:41:35.261236 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) May 23 12:41:35.262344 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped May 23 12:41:35.262344 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) May 23 12:41:35.262344 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper May 23 12:41:35.262344 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) May 23 12:41:35.262344 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper May 23 12:41:35.262344 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) May 23 12:41:35.262344 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun May 23 12:41:35.262344 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) May 23 12:41:35.262344 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func May 23 12:41:35.262344 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self.get_or_create( May 23 12:41:35.262344 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create May 23 12:41:35.262344 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application with Lock( May 23 12:41:35.262344 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ May 23 12:41:35.262344 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self._enter() May 23 12:41:35.262344 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter May 23 12:41:35.262344 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) May 23 12:41:35.262344 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create May 23 12:41:35.262344 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self.creator() May 23 12:41:35.263336 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value May 23 12:41:35.263336 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application created_value = creator( May 23 12:41:35.263336 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user May 23 12:41:35.263336 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) May 23 12:41:35.263336 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user May 23 12:41:35.263336 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) May 23 12:41:35.263336 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user May 23 12:41:35.263336 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) May 23 12:41:35.263336 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: glance. May 23 12:41:35.263336 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application May 23 12:41:35.263336 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 327/642] 10.222.0.30 () {56 vars in 1099 bytes} [Mon May 23 12:41:35 2022] GET /identity/v3/users/glance => generated 84 bytes in 185 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) May 23 12:41:35.263336 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-441ecc1d-74ad-4a14-8c51-8495c1b4e160 None admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:41:35.323511 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=100 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:41:35.360935 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80b851bf-e809-4bb5-a5a5-5079b860ca7b None admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:41:35.374257 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-441ecc1d-74ad-4a14-8c51-8495c1b4e160 None admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:41:35.396765 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-adc69141-e00e-41d6-8ce9-f066ef5de594 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:41:35.397781 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-adc69141-e00e-41d6-8ce9-f066ef5de594 None admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:41:35.398107 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-adc69141-e00e-41d6-8ce9-f066ef5de594 None admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:41:35.398385 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-adc69141-e00e-41d6-8ce9-f066ef5de594 None admin] PATH_INFO: `/v3/domains` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:41:35.399095 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-adc69141-e00e-41d6-8ce9-f066ef5de594 None admin] RBAC: Adding query filter params (name=Default) {{(pid=92057) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} May 23 12:41:35.399401 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-adc69141-e00e-41d6-8ce9-f066ef5de594 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:41:35.400806 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-adc69141-e00e-41d6-8ce9-f066ef5de594 None admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:41:35.413026 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dafddd49-96e7-4d12-b52d-194c8debc754 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:41:35.414157 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dafddd49-96e7-4d12-b52d-194c8debc754 None admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:41:35.414270 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 316/643] 10.222.0.30 () {56 vars in 1109 bytes} [Mon May 23 12:41:35 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 169 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) May 23 12:41:35.414350 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dafddd49-96e7-4d12-b52d-194c8debc754 None admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:41:35.414424 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dafddd49-96e7-4d12-b52d-194c8debc754 None admin] PATH_INFO: `/v3/roles/service` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:41:35.418986 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-adc69141-e00e-41d6-8ce9-f066ef5de594 None admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:41:35.425808 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dafddd49-96e7-4d12-b52d-194c8debc754 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} May 23 12:41:35.431919 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=100 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:41:35.434232 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application [None req-dafddd49-96e7-4d12-b52d-194c8debc754 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. May 23 12:41:35.434232 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application Traceback (most recent call last): May 23 12:41:35.434232 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request May 23 12:41:35.434232 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application rv = self.dispatch_request() May 23 12:41:35.434232 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request May 23 12:41:35.434232 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) May 23 12:41:35.434232 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper May 23 12:41:35.434232 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) May 23 12:41:35.434232 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view May 23 12:41:35.434232 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) May 23 12:41:35.434232 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request May 23 12:41:35.434232 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) May 23 12:41:35.434232 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get May 23 12:41:35.434232 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self._get_role(role_id) May 23 12:41:35.434232 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role May 23 12:41:35.434232 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') May 23 12:41:35.435177 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call May 23 12:41:35.435177 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( May 23 12:41:35.435177 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data May 23 12:41:35.435177 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) May 23 12:41:35.435177 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped May 23 12:41:35.435177 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) May 23 12:41:35.435177 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun May 23 12:41:35.435177 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) May 23 12:41:35.435177 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func May 23 12:41:35.435177 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self.get_or_create( May 23 12:41:35.435177 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create May 23 12:41:35.435177 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application with Lock( May 23 12:41:35.435177 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ May 23 12:41:35.435177 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self._enter() May 23 12:41:35.435177 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter May 23 12:41:35.435177 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) May 23 12:41:35.436115 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create May 23 12:41:35.436115 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self.creator() May 23 12:41:35.436115 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value May 23 12:41:35.436115 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application created_value = creator( May 23 12:41:35.436115 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role May 23 12:41:35.436115 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) May 23 12:41:35.436115 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role May 23 12:41:35.436115 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() May 23 12:41:35.436115 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role May 23 12:41:35.436115 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) May 23 12:41:35.436115 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. May 23 12:41:35.436115 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application May 23 12:41:35.436115 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 328/644] 10.222.0.30 () {56 vars in 1102 bytes} [Mon May 23 12:41:35 2022] GET /identity/v3/roles/service => generated 85 bytes in 172 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) May 23 12:41:35.437916 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dafddd49-96e7-4d12-b52d-194c8debc754 None admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:41:35.533241 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-adc69141-e00e-41d6-8ce9-f066ef5de594 None admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:41:35.548387 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dafddd49-96e7-4d12-b52d-194c8debc754 None admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:41:35.562249 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=100 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:41:35.571557 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-544586bc-2c1b-4776-b123-5ef8f15259df None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:41:35.572280 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-544586bc-2c1b-4776-b123-5ef8f15259df None admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:41:35.572565 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-544586bc-2c1b-4776-b123-5ef8f15259df None admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:41:35.572839 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-544586bc-2c1b-4776-b123-5ef8f15259df None admin] PATH_INFO: `/v3/roles` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:41:35.573713 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-544586bc-2c1b-4776-b123-5ef8f15259df None admin] RBAC: Adding query filter params (name=service) {{(pid=92057) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} May 23 12:41:35.574055 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-544586bc-2c1b-4776-b123-5ef8f15259df None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:41:35.575446 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-544586bc-2c1b-4776-b123-5ef8f15259df None admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:41:35.583535 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-438be8c6-e7be-44a1-8186-ef104648d6b5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:41:35.584199 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-438be8c6-e7be-44a1-8186-ef104648d6b5 None admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:41:35.589720 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-438be8c6-e7be-44a1-8186-ef104648d6b5 None admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:41:35.589862 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 317/645] 10.222.0.30 () {56 vars in 1103 bytes} [Mon May 23 12:41:35 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 166 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) May 23 12:41:35.589976 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-438be8c6-e7be-44a1-8186-ef104648d6b5 None admin] PATH_INFO: `/v3/users` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:41:35.590053 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-544586bc-2c1b-4776-b123-5ef8f15259df None admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:41:35.590126 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-438be8c6-e7be-44a1-8186-ef104648d6b5 None admin] RBAC: Adding query filter params (domain_id=default, name=glance) {{(pid=92056) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} May 23 12:41:35.590126 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-438be8c6-e7be-44a1-8186-ef104648d6b5 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:41:35.590126 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-438be8c6-e7be-44a1-8186-ef104648d6b5 None admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:41:35.608555 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 329/646] 10.222.0.30 () {56 vars in 1137 bytes} [Mon May 23 12:41:35 2022] GET /identity/v3/users?domain_id=default&name=glance => generated 372 bytes in 173 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) May 23 12:41:35.609815 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-438be8c6-e7be-44a1-8186-ef104648d6b5 None admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:41:35.669854 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=100 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:41:35.706858 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-544586bc-2c1b-4776-b123-5ef8f15259df None admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:41:35.716018 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-438be8c6-e7be-44a1-8186-ef104648d6b5 None admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:41:35.743185 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c9e1ddb-a8d5-4cf4-a2af-5e85cb8eaa96 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:41:35.744184 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c9e1ddb-a8d5-4cf4-a2af-5e85cb8eaa96 None admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:41:35.744468 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c9e1ddb-a8d5-4cf4-a2af-5e85cb8eaa96 None admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:41:35.744746 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c9e1ddb-a8d5-4cf4-a2af-5e85cb8eaa96 None admin] PATH_INFO: `/v3/projects/service` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:41:35.751632 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c9e1ddb-a8d5-4cf4-a2af-5e85cb8eaa96 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:41:35.752530 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7617a1f7-2d9a-4c46-b795-054b0db0f2f9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:41:35.753066 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c9e1ddb-a8d5-4cf4-a2af-5e85cb8eaa96 None admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:41:35.753585 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7617a1f7-2d9a-4c46-b795-054b0db0f2f9 None admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:41:35.753971 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7617a1f7-2d9a-4c46-b795-054b0db0f2f9 None admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:41:35.754248 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7617a1f7-2d9a-4c46-b795-054b0db0f2f9 None admin] PATH_INFO: `/v3/roles` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:41:35.754985 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7617a1f7-2d9a-4c46-b795-054b0db0f2f9 None admin] RBAC: Adding query filter params (name=service) {{(pid=92056) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} May 23 12:41:35.755288 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7617a1f7-2d9a-4c46-b795-054b0db0f2f9 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:41:35.757236 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7617a1f7-2d9a-4c46-b795-054b0db0f2f9 None admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:41:35.769841 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 330/647] 10.222.0.30 () {56 vars in 1103 bytes} [Mon May 23 12:41:35 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 156 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) May 23 12:41:35.769841 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7617a1f7-2d9a-4c46-b795-054b0db0f2f9 None admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:41:35.770328 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application [None req-8c9e1ddb-a8d5-4cf4-a2af-5e85cb8eaa96 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. May 23 12:41:35.770328 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application Traceback (most recent call last): May 23 12:41:35.770328 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request May 23 12:41:35.770328 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application rv = self.dispatch_request() May 23 12:41:35.770328 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request May 23 12:41:35.770328 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) May 23 12:41:35.770328 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper May 23 12:41:35.770328 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) May 23 12:41:35.770328 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view May 23 12:41:35.770328 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) May 23 12:41:35.770328 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request May 23 12:41:35.770328 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) May 23 12:41:35.770328 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get May 23 12:41:35.770328 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self._get_project(project_id) May 23 12:41:35.770328 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project May 23 12:41:35.771191 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) May 23 12:41:35.771191 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped May 23 12:41:35.771191 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) May 23 12:41:35.771191 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun May 23 12:41:35.771191 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) May 23 12:41:35.771191 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func May 23 12:41:35.771191 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self.get_or_create( May 23 12:41:35.771191 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create May 23 12:41:35.771191 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application with Lock( May 23 12:41:35.771191 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ May 23 12:41:35.771191 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self._enter() May 23 12:41:35.771191 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter May 23 12:41:35.771191 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) May 23 12:41:35.771191 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create May 23 12:41:35.771191 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self.creator() May 23 12:41:35.771191 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value May 23 12:41:35.771191 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application created_value = creator( May 23 12:41:35.772181 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project May 23 12:41:35.772181 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) May 23 12:41:35.772181 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project May 23 12:41:35.772181 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() May 23 12:41:35.772181 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project May 23 12:41:35.772181 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) May 23 12:41:35.772181 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. May 23 12:41:35.772181 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application May 23 12:41:35.772181 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 318/648] 10.222.0.30 () {56 vars in 1111 bytes} [Mon May 23 12:41:35 2022] GET /identity/v3/projects/service => generated 88 bytes in 184 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) May 23 12:41:35.772181 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c9e1ddb-a8d5-4cf4-a2af-5e85cb8eaa96 None admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:41:35.808685 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=100 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:41:35.875322 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7617a1f7-2d9a-4c46-b795-054b0db0f2f9 None admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:41:35.893284 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c9e1ddb-a8d5-4cf4-a2af-5e85cb8eaa96 None admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:41:35.913813 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27bddb01-a38a-4eb7-9c25-17ac2ee047df None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:41:35.914629 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27bddb01-a38a-4eb7-9c25-17ac2ee047df None admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:41:35.914909 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27bddb01-a38a-4eb7-9c25-17ac2ee047df None admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:41:35.915240 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27bddb01-a38a-4eb7-9c25-17ac2ee047df None admin] PATH_INFO: `/v3/domains/Default` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:41:35.926356 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=100 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:41:35.926483 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-27bddb01-a38a-4eb7-9c25-17ac2ee047df None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:41:35.928550 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-27bddb01-a38a-4eb7-9c25-17ac2ee047df None admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:41:35.931409 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a033ea3c-492a-460d-b530-33a713649a2a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:41:35.932061 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a033ea3c-492a-460d-b530-33a713649a2a None admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:41:35.932349 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a033ea3c-492a-460d-b530-33a713649a2a None admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:41:35.932629 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a033ea3c-492a-460d-b530-33a713649a2a None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:41:35.947551 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application [None req-27bddb01-a38a-4eb7-9c25-17ac2ee047df None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. May 23 12:41:35.947551 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application Traceback (most recent call last): May 23 12:41:35.947551 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request May 23 12:41:35.947551 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application rv = self.dispatch_request() May 23 12:41:35.947551 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request May 23 12:41:35.947551 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) May 23 12:41:35.947551 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper May 23 12:41:35.947551 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) May 23 12:41:35.947551 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view May 23 12:41:35.947551 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) May 23 12:41:35.947551 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request May 23 12:41:35.947551 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) May 23 12:41:35.947551 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get May 23 12:41:35.947551 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self._get_domain(domain_id) May 23 12:41:35.947551 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain May 23 12:41:35.948478 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) May 23 12:41:35.948478 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped May 23 12:41:35.948478 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) May 23 12:41:35.948478 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun May 23 12:41:35.948478 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) May 23 12:41:35.948478 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func May 23 12:41:35.948478 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self.get_or_create( May 23 12:41:35.948478 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create May 23 12:41:35.948478 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application with Lock( May 23 12:41:35.948478 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ May 23 12:41:35.948478 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self._enter() May 23 12:41:35.948478 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter May 23 12:41:35.948478 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) May 23 12:41:35.948478 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create May 23 12:41:35.948478 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self.creator() May 23 12:41:35.948478 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value May 23 12:41:35.948478 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application created_value = creator( May 23 12:41:35.949390 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain May 23 12:41:35.949390 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) May 23 12:41:35.949390 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. May 23 12:41:35.949390 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application May 23 12:41:35.949390 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 331/649] 10.222.0.30 () {56 vars in 1108 bytes} [Mon May 23 12:41:35 2022] GET /identity/v3/domains/Default => generated 87 bytes in 182 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) May 23 12:41:35.949683 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27bddb01-a38a-4eb7-9c25-17ac2ee047df None admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:41:36.008851 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a033ea3c-492a-460d-b530-33a713649a2a None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:41:36.010358 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a033ea3c-492a-460d-b530-33a713649a2a None admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:41:36.038839 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=100 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:41:36.054712 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27bddb01-a38a-4eb7-9c25-17ac2ee047df None admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:41:36.090650 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f8aae8f-a580-4571-8c00-52d1ffc8fdde None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:41:36.091719 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f8aae8f-a580-4571-8c00-52d1ffc8fdde None admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:41:36.091999 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f8aae8f-a580-4571-8c00-52d1ffc8fdde None admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:41:36.092266 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f8aae8f-a580-4571-8c00-52d1ffc8fdde None admin] PATH_INFO: `/v3/domains/Default` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:41:36.104050 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f8aae8f-a580-4571-8c00-52d1ffc8fdde None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:41:36.106058 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f8aae8f-a580-4571-8c00-52d1ffc8fdde None admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:41:36.118409 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 319/650] 10.222.0.30 () {58 vars in 1282 bytes} [Mon May 23 12:41:35 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 349 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) May 23 12:41:36.122790 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a033ea3c-492a-460d-b530-33a713649a2a None admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:41:36.123701 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application [None req-2f8aae8f-a580-4571-8c00-52d1ffc8fdde None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. May 23 12:41:36.123701 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application Traceback (most recent call last): May 23 12:41:36.123701 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request May 23 12:41:36.123701 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application rv = self.dispatch_request() May 23 12:41:36.123701 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request May 23 12:41:36.123701 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) May 23 12:41:36.123701 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper May 23 12:41:36.123701 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) May 23 12:41:36.123701 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view May 23 12:41:36.123701 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) May 23 12:41:36.123701 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request May 23 12:41:36.123701 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) May 23 12:41:36.123701 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get May 23 12:41:36.123701 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self._get_domain(domain_id) May 23 12:41:36.123701 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain May 23 12:41:36.124571 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) May 23 12:41:36.124571 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped May 23 12:41:36.124571 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) May 23 12:41:36.124571 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun May 23 12:41:36.124571 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) May 23 12:41:36.124571 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func May 23 12:41:36.124571 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self.get_or_create( May 23 12:41:36.124571 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create May 23 12:41:36.124571 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application with Lock( May 23 12:41:36.124571 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ May 23 12:41:36.124571 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self._enter() May 23 12:41:36.124571 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter May 23 12:41:36.124571 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) May 23 12:41:36.124571 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create May 23 12:41:36.124571 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self.creator() May 23 12:41:36.124571 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value May 23 12:41:36.124571 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application created_value = creator( May 23 12:41:36.125480 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain May 23 12:41:36.125480 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) May 23 12:41:36.125480 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. May 23 12:41:36.125480 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application May 23 12:41:36.129579 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 332/651] 10.222.0.30 () {56 vars in 1108 bytes} [Mon May 23 12:41:35 2022] GET /identity/v3/domains/Default => generated 87 bytes in 180 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) May 23 12:41:36.130516 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f8aae8f-a580-4571-8c00-52d1ffc8fdde None admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:41:36.156385 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=100 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:41:36.225693 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a033ea3c-492a-460d-b530-33a713649a2a None admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:41:36.238176 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f8aae8f-a580-4571-8c00-52d1ffc8fdde None admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:41:36.266432 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d073e8a6-9d07-4d6a-8865-596779d765f2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:41:36.267395 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d073e8a6-9d07-4d6a-8865-596779d765f2 None admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:41:36.267677 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d073e8a6-9d07-4d6a-8865-596779d765f2 None admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:41:36.267949 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d073e8a6-9d07-4d6a-8865-596779d765f2 None admin] PATH_INFO: `/v3/projects` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:41:36.268721 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d073e8a6-9d07-4d6a-8865-596779d765f2 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=92057) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} May 23 12:41:36.269130 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d073e8a6-9d07-4d6a-8865-596779d765f2 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:41:36.270488 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d073e8a6-9d07-4d6a-8865-596779d765f2 None admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:41:36.275427 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c047ce0-1111-4630-9ec7-992dcbf232fe None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:41:36.276175 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c047ce0-1111-4630-9ec7-992dcbf232fe None admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:41:36.276529 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c047ce0-1111-4630-9ec7-992dcbf232fe None admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:41:36.276874 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c047ce0-1111-4630-9ec7-992dcbf232fe None admin] PATH_INFO: `/v3/domains` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:41:36.277998 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c047ce0-1111-4630-9ec7-992dcbf232fe None admin] RBAC: Adding query filter params (name=Default) {{(pid=92056) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} May 23 12:41:36.278377 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c047ce0-1111-4630-9ec7-992dcbf232fe None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:41:36.279872 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c047ce0-1111-4630-9ec7-992dcbf232fe None admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:41:36.282886 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 320/652] 10.222.0.30 () {56 vars in 1148 bytes} [Mon May 23 12:41:36 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 164 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) May 23 12:41:36.283851 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d073e8a6-9d07-4d6a-8865-596779d765f2 None admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:41:36.287340 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=100 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:41:36.293445 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 333/653] 10.222.0.30 () {56 vars in 1109 bytes} [Mon May 23 12:41:36 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 163 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) May 23 12:41:36.294494 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c047ce0-1111-4630-9ec7-992dcbf232fe None admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:41:36.396467 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=100 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:41:36.404710 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c047ce0-1111-4630-9ec7-992dcbf232fe None admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:41:36.406415 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d073e8a6-9d07-4d6a-8865-596779d765f2 None admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:41:36.443353 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0e65649-4060-48bf-aeee-effb724fdb74 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:41:36.444654 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0e65649-4060-48bf-aeee-effb724fdb74 None admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:41:36.444654 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0e65649-4060-48bf-aeee-effb724fdb74 None admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:41:36.444654 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0e65649-4060-48bf-aeee-effb724fdb74 None admin] PATH_INFO: `/v3/domains` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:41:36.445280 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0e65649-4060-48bf-aeee-effb724fdb74 None admin] RBAC: Adding query filter params (name=Default) {{(pid=92056) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} May 23 12:41:36.445582 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0e65649-4060-48bf-aeee-effb724fdb74 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:41:36.446997 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0e65649-4060-48bf-aeee-effb724fdb74 None admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:41:36.449589 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6063a93-1ad9-41b5-8d03-102a4d46499d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:41:36.450405 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6063a93-1ad9-41b5-8d03-102a4d46499d None admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:41:36.450684 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6063a93-1ad9-41b5-8d03-102a4d46499d None admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:41:36.450953 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6063a93-1ad9-41b5-8d03-102a4d46499d None admin] PATH_INFO: `/v3/domains/Default` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:41:36.459202 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 334/654] 10.222.0.30 () {56 vars in 1109 bytes} [Mon May 23 12:41:36 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 165 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) May 23 12:41:36.460848 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0e65649-4060-48bf-aeee-effb724fdb74 None admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:41:36.467267 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6063a93-1ad9-41b5-8d03-102a4d46499d None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:41:36.468824 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6063a93-1ad9-41b5-8d03-102a4d46499d None admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:41:36.487474 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application [None req-f6063a93-1ad9-41b5-8d03-102a4d46499d None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. May 23 12:41:36.487474 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application Traceback (most recent call last): May 23 12:41:36.487474 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request May 23 12:41:36.487474 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application rv = self.dispatch_request() May 23 12:41:36.487474 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request May 23 12:41:36.487474 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) May 23 12:41:36.487474 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper May 23 12:41:36.487474 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) May 23 12:41:36.487474 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view May 23 12:41:36.487474 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) May 23 12:41:36.487474 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request May 23 12:41:36.487474 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) May 23 12:41:36.487474 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get May 23 12:41:36.487474 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self._get_domain(domain_id) May 23 12:41:36.487474 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain May 23 12:41:36.488410 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) May 23 12:41:36.488410 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped May 23 12:41:36.488410 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) May 23 12:41:36.488410 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun May 23 12:41:36.488410 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) May 23 12:41:36.488410 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func May 23 12:41:36.488410 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self.get_or_create( May 23 12:41:36.488410 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create May 23 12:41:36.488410 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application with Lock( May 23 12:41:36.488410 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ May 23 12:41:36.488410 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self._enter() May 23 12:41:36.488410 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter May 23 12:41:36.488410 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) May 23 12:41:36.488410 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create May 23 12:41:36.488410 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self.creator() May 23 12:41:36.488410 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value May 23 12:41:36.488410 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application created_value = creator( May 23 12:41:36.489365 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain May 23 12:41:36.489365 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) May 23 12:41:36.489365 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. May 23 12:41:36.489365 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application May 23 12:41:36.489365 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 321/655] 10.222.0.30 () {56 vars in 1108 bytes} [Mon May 23 12:41:36 2022] GET /identity/v3/domains/Default => generated 87 bytes in 205 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) May 23 12:41:36.489365 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6063a93-1ad9-41b5-8d03-102a4d46499d None admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:41:36.533774 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=100 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:41:36.577352 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0e65649-4060-48bf-aeee-effb724fdb74 None admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:41:36.593488 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6063a93-1ad9-41b5-8d03-102a4d46499d None admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:41:36.611213 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4117bf6b-db91-4417-a494-60d2c9280f31 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:41:36.611869 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4117bf6b-db91-4417-a494-60d2c9280f31 None admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:41:36.612150 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4117bf6b-db91-4417-a494-60d2c9280f31 None admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:41:36.612424 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4117bf6b-db91-4417-a494-60d2c9280f31 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:41:36.613242 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4117bf6b-db91-4417-a494-60d2c9280f31 None admin] RBAC: Adding query filter params (role.id=41ab28c1ae3d4941bd09fb5d35f1dc54, scope.project.id=9c389ae65aaa4542bb5a5c4fd0bacf5b, user.id=cbe9931d94d44cc7ace0e83a7e5015fc) {{(pid=92056) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} May 23 12:41:36.613550 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4117bf6b-db91-4417-a494-60d2c9280f31 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:41:36.615024 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4117bf6b-db91-4417-a494-60d2c9280f31 None admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:41:36.621141 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 335/656] 10.222.0.30 () {56 vars in 1374 bytes} [Mon May 23 12:41:36 2022] GET /identity/v3/role_assignments?user.id=cbe9931d94d44cc7ace0e83a7e5015fc&scope.project.id=9c389ae65aaa4542bb5a5c4fd0bacf5b&role.id=41ab28c1ae3d4941bd09fb5d35f1dc54 => generated 608 bytes in 161 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) May 23 12:41:36.621948 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4117bf6b-db91-4417-a494-60d2c9280f31 None admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:41:36.642973 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8007254-4363-416e-8623-399a28934361 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:41:36.644210 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8007254-4363-416e-8623-399a28934361 None admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:41:36.644676 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8007254-4363-416e-8623-399a28934361 None admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:41:36.645134 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8007254-4363-416e-8623-399a28934361 None admin] PATH_INFO: `/v3/users/neutron` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:41:36.657530 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8007254-4363-416e-8623-399a28934361 None admin] RBAC: Authorizing `identity:get_user(user_id=neutron)` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:41:36.659191 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8007254-4363-416e-8623-399a28934361 None admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:41:36.675568 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application [None req-f8007254-4363-416e-8623-399a28934361 None admin] Could not find user: neutron.: keystone.exception.UserNotFound: Could not find user: neutron. May 23 12:41:36.675568 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application Traceback (most recent call last): May 23 12:41:36.675568 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request May 23 12:41:36.675568 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application rv = self.dispatch_request() May 23 12:41:36.675568 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request May 23 12:41:36.675568 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) May 23 12:41:36.675568 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper May 23 12:41:36.675568 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) May 23 12:41:36.675568 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view May 23 12:41:36.675568 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) May 23 12:41:36.675568 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request May 23 12:41:36.675568 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) May 23 12:41:36.675568 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get May 23 12:41:36.675568 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self._get_user(user_id) May 23 12:41:36.675568 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user May 23 12:41:36.675568 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) May 23 12:41:36.676595 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped May 23 12:41:36.676595 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) May 23 12:41:36.676595 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper May 23 12:41:36.676595 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) May 23 12:41:36.676595 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper May 23 12:41:36.676595 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) May 23 12:41:36.676595 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun May 23 12:41:36.676595 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) May 23 12:41:36.676595 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func May 23 12:41:36.676595 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self.get_or_create( May 23 12:41:36.676595 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create May 23 12:41:36.676595 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application with Lock( May 23 12:41:36.676595 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ May 23 12:41:36.676595 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self._enter() May 23 12:41:36.676595 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter May 23 12:41:36.676595 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) May 23 12:41:36.676595 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create May 23 12:41:36.676595 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self.creator() May 23 12:41:36.677557 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value May 23 12:41:36.677557 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application created_value = creator( May 23 12:41:36.677557 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user May 23 12:41:36.677557 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) May 23 12:41:36.677557 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user May 23 12:41:36.677557 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) May 23 12:41:36.677557 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user May 23 12:41:36.677557 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) May 23 12:41:36.677557 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: neutron. May 23 12:41:36.677557 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application May 23 12:41:36.681836 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 322/657] 10.222.0.30 () {56 vars in 1102 bytes} [Mon May 23 12:41:36 2022] GET /identity/v3/users/neutron => generated 85 bytes in 193 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) May 23 12:41:36.682813 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8007254-4363-416e-8623-399a28934361 None admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:41:36.703779 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=100 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:41:36.751710 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4117bf6b-db91-4417-a494-60d2c9280f31 None admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:41:36.778464 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=100 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:41:36.791237 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12ee6f94-12de-409c-a428-bfbe349ed361 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:41:36.792116 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12ee6f94-12de-409c-a428-bfbe349ed361 None admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:41:36.792321 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12ee6f94-12de-409c-a428-bfbe349ed361 None admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:41:36.792651 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12ee6f94-12de-409c-a428-bfbe349ed361 None admin] PATH_INFO: `/v3/users/cinder` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:41:36.801913 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12ee6f94-12de-409c-a428-bfbe349ed361 None admin] RBAC: Authorizing `identity:get_user(user_id=cinder)` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:41:36.803689 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12ee6f94-12de-409c-a428-bfbe349ed361 None admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:41:36.821681 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application [None req-12ee6f94-12de-409c-a428-bfbe349ed361 None admin] Could not find user: cinder.: keystone.exception.UserNotFound: Could not find user: cinder. May 23 12:41:36.821681 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application Traceback (most recent call last): May 23 12:41:36.821681 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request May 23 12:41:36.821681 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application rv = self.dispatch_request() May 23 12:41:36.821681 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request May 23 12:41:36.821681 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) May 23 12:41:36.821681 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper May 23 12:41:36.821681 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) May 23 12:41:36.821681 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view May 23 12:41:36.821681 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) May 23 12:41:36.821681 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request May 23 12:41:36.821681 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) May 23 12:41:36.821681 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get May 23 12:41:36.821681 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self._get_user(user_id) May 23 12:41:36.821681 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user May 23 12:41:36.821681 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) May 23 12:41:36.822788 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped May 23 12:41:36.822788 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) May 23 12:41:36.822788 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper May 23 12:41:36.822788 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) May 23 12:41:36.822788 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper May 23 12:41:36.822788 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) May 23 12:41:36.822788 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun May 23 12:41:36.822788 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) May 23 12:41:36.822788 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func May 23 12:41:36.822788 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self.get_or_create( May 23 12:41:36.822788 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create May 23 12:41:36.822788 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application with Lock( May 23 12:41:36.822788 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ May 23 12:41:36.822788 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self._enter() May 23 12:41:36.822788 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter May 23 12:41:36.822788 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) May 23 12:41:36.822788 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create May 23 12:41:36.822788 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self.creator() May 23 12:41:36.823772 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value May 23 12:41:36.823772 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application created_value = creator( May 23 12:41:36.823772 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user May 23 12:41:36.823772 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) May 23 12:41:36.823772 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user May 23 12:41:36.823772 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) May 23 12:41:36.823772 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user May 23 12:41:36.823772 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) May 23 12:41:36.823772 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: cinder. May 23 12:41:36.823772 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application May 23 12:41:36.823772 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 336/658] 10.222.0.30 () {56 vars in 1099 bytes} [Mon May 23 12:41:36 2022] GET /identity/v3/users/cinder => generated 84 bytes in 201 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) May 23 12:41:36.824417 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12ee6f94-12de-409c-a428-bfbe349ed361 None admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:41:36.848918 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8007254-4363-416e-8623-399a28934361 None admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:41:36.894257 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd8a7aee-2d32-463e-ba08-3e5796cecd66 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:41:36.895175 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd8a7aee-2d32-463e-ba08-3e5796cecd66 None admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:41:36.895459 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd8a7aee-2d32-463e-ba08-3e5796cecd66 None admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:41:36.895740 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd8a7aee-2d32-463e-ba08-3e5796cecd66 None admin] PATH_INFO: `/v3/domains` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:41:36.896455 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd8a7aee-2d32-463e-ba08-3e5796cecd66 None admin] RBAC: Adding query filter params (name=Default) {{(pid=92057) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} May 23 12:41:36.896877 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd8a7aee-2d32-463e-ba08-3e5796cecd66 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:41:36.898296 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd8a7aee-2d32-463e-ba08-3e5796cecd66 None admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:41:36.915541 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 323/659] 10.222.0.30 () {56 vars in 1109 bytes} [Mon May 23 12:41:36 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 233 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) May 23 12:41:36.916297 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd8a7aee-2d32-463e-ba08-3e5796cecd66 None admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:41:36.951966 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12ee6f94-12de-409c-a428-bfbe349ed361 None admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:41:36.993428 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=100 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:41:36.997082 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14f2971d-1ebf-4840-9628-5b08a0e2af35 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:41:36.997949 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14f2971d-1ebf-4840-9628-5b08a0e2af35 None admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:41:36.998238 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14f2971d-1ebf-4840-9628-5b08a0e2af35 None admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:41:36.999021 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14f2971d-1ebf-4840-9628-5b08a0e2af35 None admin] PATH_INFO: `/v3/users` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:41:37.000199 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14f2971d-1ebf-4840-9628-5b08a0e2af35 None admin] RBAC: Adding query filter params (domain_id=default, name=neutron) {{(pid=92056) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} May 23 12:41:37.000529 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14f2971d-1ebf-4840-9628-5b08a0e2af35 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:41:37.005156 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14f2971d-1ebf-4840-9628-5b08a0e2af35 None admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:41:37.025119 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 337/660] 10.222.0.30 () {56 vars in 1139 bytes} [Mon May 23 12:41:36 2022] GET /identity/v3/users?domain_id=default&name=neutron => generated 374 bytes in 201 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) May 23 12:41:37.026004 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14f2971d-1ebf-4840-9628-5b08a0e2af35 None admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:41:37.038915 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd8a7aee-2d32-463e-ba08-3e5796cecd66 None admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:41:37.073283 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=100 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:41:37.073966 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0eee0fe2-1364-44d7-ad17-af0675166a9b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:41:37.074649 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0eee0fe2-1364-44d7-ad17-af0675166a9b None admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:41:37.074935 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0eee0fe2-1364-44d7-ad17-af0675166a9b None admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:41:37.075213 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0eee0fe2-1364-44d7-ad17-af0675166a9b None admin] PATH_INFO: `/v3/users` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:41:37.076045 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0eee0fe2-1364-44d7-ad17-af0675166a9b None admin] RBAC: Adding query filter params (domain_id=default, name=cinder) {{(pid=92057) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} May 23 12:41:37.076360 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0eee0fe2-1364-44d7-ad17-af0675166a9b None admin] RBAC: Authorizing `identity:list_users()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:41:37.077829 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0eee0fe2-1364-44d7-ad17-af0675166a9b None admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:41:37.095299 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 324/661] 10.222.0.30 () {56 vars in 1137 bytes} [Mon May 23 12:41:36 2022] GET /identity/v3/users?domain_id=default&name=cinder => generated 372 bytes in 179 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) May 23 12:41:37.096600 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9594f813-aaf7-4ed5-919c-9766892bcde8 None None] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:41:37.096888 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9594f813-aaf7-4ed5-919c-9766892bcde8 None None] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:41:37.097166 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9594f813-aaf7-4ed5-919c-9766892bcde8 None None] PATH_INFO: `/` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:41:37.102639 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 325/662] 10.222.0.30 () {54 vars in 926 bytes} [Mon May 23 12:41:37 2022] GET /identity => generated 270 bytes in 7 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) May 23 12:41:37.103443 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9594f813-aaf7-4ed5-919c-9766892bcde8 None None] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:41:37.144132 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14f2971d-1ebf-4840-9628-5b08a0e2af35 None admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:41:37.183584 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eec14b5b-ed1d-437a-a6bc-0bc9b9e91141 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:41:37.184763 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eec14b5b-ed1d-437a-a6bc-0bc9b9e91141 None admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:41:37.185191 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eec14b5b-ed1d-437a-a6bc-0bc9b9e91141 None admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:41:37.185621 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eec14b5b-ed1d-437a-a6bc-0bc9b9e91141 None admin] PATH_INFO: `/v3/projects/service` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:41:37.193124 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eec14b5b-ed1d-437a-a6bc-0bc9b9e91141 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:41:37.196168 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eec14b5b-ed1d-437a-a6bc-0bc9b9e91141 None admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:41:37.210301 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application [None req-eec14b5b-ed1d-437a-a6bc-0bc9b9e91141 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. May 23 12:41:37.210301 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application Traceback (most recent call last): May 23 12:41:37.210301 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request May 23 12:41:37.210301 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application rv = self.dispatch_request() May 23 12:41:37.210301 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request May 23 12:41:37.210301 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) May 23 12:41:37.210301 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper May 23 12:41:37.210301 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) May 23 12:41:37.210301 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view May 23 12:41:37.210301 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) May 23 12:41:37.210301 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request May 23 12:41:37.210301 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) May 23 12:41:37.210301 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get May 23 12:41:37.210301 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self._get_project(project_id) May 23 12:41:37.210301 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project May 23 12:41:37.211301 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) May 23 12:41:37.211301 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped May 23 12:41:37.211301 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) May 23 12:41:37.211301 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun May 23 12:41:37.211301 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) May 23 12:41:37.211301 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func May 23 12:41:37.211301 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self.get_or_create( May 23 12:41:37.211301 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create May 23 12:41:37.211301 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application with Lock( May 23 12:41:37.211301 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ May 23 12:41:37.211301 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self._enter() May 23 12:41:37.211301 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter May 23 12:41:37.211301 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) May 23 12:41:37.211301 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create May 23 12:41:37.211301 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self.creator() May 23 12:41:37.211301 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value May 23 12:41:37.211301 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application created_value = creator( May 23 12:41:37.212239 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project May 23 12:41:37.212239 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) May 23 12:41:37.212239 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project May 23 12:41:37.212239 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() May 23 12:41:37.212239 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project May 23 12:41:37.212239 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) May 23 12:41:37.212239 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. May 23 12:41:37.212239 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application May 23 12:41:37.212239 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 338/663] 10.222.0.30 () {56 vars in 1111 bytes} [Mon May 23 12:41:37 2022] GET /identity/v3/projects/service => generated 88 bytes in 186 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) May 23 12:41:37.212239 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eec14b5b-ed1d-437a-a6bc-0bc9b9e91141 None admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:41:37.216486 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9594f813-aaf7-4ed5-919c-9766892bcde8 None None] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:41:37.241070 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=100 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:41:37.252562 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-701d11e8-c654-4536-bc91-c9e15366fcc5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:41:37.253196 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-701d11e8-c654-4536-bc91-c9e15366fcc5 None admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:41:37.253484 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-701d11e8-c654-4536-bc91-c9e15366fcc5 None admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:41:37.254424 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-701d11e8-c654-4536-bc91-c9e15366fcc5 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:41:37.330324 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=100 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:41:37.330782 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-701d11e8-c654-4536-bc91-c9e15366fcc5 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:41:37.332658 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-701d11e8-c654-4536-bc91-c9e15366fcc5 None admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:41:37.333393 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eec14b5b-ed1d-437a-a6bc-0bc9b9e91141 None admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:41:37.369925 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57242600-02fe-4c35-affe-3b60b1a3dada None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:41:37.371235 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57242600-02fe-4c35-affe-3b60b1a3dada None admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:41:37.371875 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57242600-02fe-4c35-affe-3b60b1a3dada None admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:41:37.372325 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57242600-02fe-4c35-affe-3b60b1a3dada None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:41:37.436531 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 326/664] 10.222.0.30 () {58 vars in 1282 bytes} [Mon May 23 12:41:37 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 333 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) May 23 12:41:37.437923 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-584ca959-7e73-447d-b7ac-52cb3cdd6355 None None] REQUEST_METHOD: `POST` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:41:37.438217 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-584ca959-7e73-447d-b7ac-52cb3cdd6355 None None] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:41:37.438496 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-584ca959-7e73-447d-b7ac-52cb3cdd6355 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:41:37.447629 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57242600-02fe-4c35-affe-3b60b1a3dada None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:41:37.449346 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57242600-02fe-4c35-affe-3b60b1a3dada None admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:41:37.455777 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.auth.core [None req-584ca959-7e73-447d-b7ac-52cb3cdd6355 None None] MFA Rules not processed for user `0f7119f8fa5c4445977671140ded91a4`. Rule list: `[]` (Enabled: `True`). {{(pid=92057) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 12:41:37.456942 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.fernet_utils [None req-584ca959-7e73-447d-b7ac-52cb3cdd6355 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92057) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 12:41:37.474766 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=100 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:41:37.537920 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 327/665] 10.222.0.30 () {58 vars in 1023 bytes} [Mon May 23 12:41:37 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 90 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) May 23 12:41:37.537920 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-584ca959-7e73-447d-b7ac-52cb3cdd6355 None None] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:41:37.551761 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=100 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:41:37.557073 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 339/666] 10.222.0.30 () {58 vars in 1282 bytes} [Mon May 23 12:41:37 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 345 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) May 23 12:41:37.558061 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57242600-02fe-4c35-affe-3b60b1a3dada None admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:41:37.656565 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-584ca959-7e73-447d-b7ac-52cb3cdd6355 None None] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:41:37.669070 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57242600-02fe-4c35-affe-3b60b1a3dada None admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:41:37.693267 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df925ccc-d58c-4e23-a9a0-df84d97645e5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:41:37.694429 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df925ccc-d58c-4e23-a9a0-df84d97645e5 None admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:41:37.694925 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df925ccc-d58c-4e23-a9a0-df84d97645e5 None admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:41:37.695386 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df925ccc-d58c-4e23-a9a0-df84d97645e5 None admin] PATH_INFO: `/v3/projects` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:41:37.696180 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df925ccc-d58c-4e23-a9a0-df84d97645e5 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=92057) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} May 23 12:41:37.696787 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df925ccc-d58c-4e23-a9a0-df84d97645e5 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:41:37.697944 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df925ccc-d58c-4e23-a9a0-df84d97645e5 None admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:41:37.710032 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 328/667] 10.222.0.30 () {56 vars in 1148 bytes} [Mon May 23 12:41:37 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 182 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) May 23 12:41:37.710838 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df925ccc-d58c-4e23-a9a0-df84d97645e5 None admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:41:37.717325 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b44d448f-eb18-4461-8554-fcabd1ef9b9d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:41:37.718399 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b44d448f-eb18-4461-8554-fcabd1ef9b9d None admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:41:37.718908 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b44d448f-eb18-4461-8554-fcabd1ef9b9d None admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:41:37.719812 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b44d448f-eb18-4461-8554-fcabd1ef9b9d None admin] PATH_INFO: `/v3/domains/Default` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:41:37.726040 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=100 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:41:37.734099 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b44d448f-eb18-4461-8554-fcabd1ef9b9d None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:41:37.736195 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b44d448f-eb18-4461-8554-fcabd1ef9b9d None admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:41:37.760006 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application [None req-b44d448f-eb18-4461-8554-fcabd1ef9b9d None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. May 23 12:41:37.760006 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application Traceback (most recent call last): May 23 12:41:37.760006 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request May 23 12:41:37.760006 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application rv = self.dispatch_request() May 23 12:41:37.760006 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request May 23 12:41:37.760006 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) May 23 12:41:37.760006 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper May 23 12:41:37.760006 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) May 23 12:41:37.760006 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view May 23 12:41:37.760006 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) May 23 12:41:37.760006 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request May 23 12:41:37.760006 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) May 23 12:41:37.760006 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get May 23 12:41:37.760006 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self._get_domain(domain_id) May 23 12:41:37.760006 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain May 23 12:41:37.761034 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) May 23 12:41:37.761034 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped May 23 12:41:37.761034 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) May 23 12:41:37.761034 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun May 23 12:41:37.761034 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) May 23 12:41:37.761034 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func May 23 12:41:37.761034 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self.get_or_create( May 23 12:41:37.761034 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create May 23 12:41:37.761034 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application with Lock( May 23 12:41:37.761034 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ May 23 12:41:37.761034 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self._enter() May 23 12:41:37.761034 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter May 23 12:41:37.761034 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) May 23 12:41:37.761034 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create May 23 12:41:37.761034 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self.creator() May 23 12:41:37.761034 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value May 23 12:41:37.761034 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application created_value = creator( May 23 12:41:37.762011 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain May 23 12:41:37.762011 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) May 23 12:41:37.762011 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. May 23 12:41:37.762011 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application May 23 12:41:37.767023 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 340/668] 10.222.0.30 () {56 vars in 1108 bytes} [Mon May 23 12:41:37 2022] GET /identity/v3/domains/Default => generated 87 bytes in 205 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) May 23 12:41:37.767023 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b44d448f-eb18-4461-8554-fcabd1ef9b9d None admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:41:37.814237 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df925ccc-d58c-4e23-a9a0-df84d97645e5 None admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:41:37.824388 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=100 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:41:37.849251 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6738efe-146f-44eb-b5d3-38d02931b086 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:41:37.850671 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6738efe-146f-44eb-b5d3-38d02931b086 None admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:41:37.850963 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6738efe-146f-44eb-b5d3-38d02931b086 None admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:41:37.851243 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6738efe-146f-44eb-b5d3-38d02931b086 None admin] PATH_INFO: `/v3/roles/service` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:41:37.862055 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6738efe-146f-44eb-b5d3-38d02931b086 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} May 23 12:41:37.870756 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application [None req-d6738efe-146f-44eb-b5d3-38d02931b086 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. May 23 12:41:37.870756 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application Traceback (most recent call last): May 23 12:41:37.870756 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request May 23 12:41:37.870756 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application rv = self.dispatch_request() May 23 12:41:37.870756 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request May 23 12:41:37.870756 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) May 23 12:41:37.870756 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper May 23 12:41:37.870756 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) May 23 12:41:37.870756 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view May 23 12:41:37.870756 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) May 23 12:41:37.870756 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request May 23 12:41:37.870756 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) May 23 12:41:37.870756 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get May 23 12:41:37.870756 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self._get_role(role_id) May 23 12:41:37.870756 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role May 23 12:41:37.870756 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') May 23 12:41:37.871727 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call May 23 12:41:37.871727 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( May 23 12:41:37.871727 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data May 23 12:41:37.871727 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) May 23 12:41:37.871727 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped May 23 12:41:37.871727 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) May 23 12:41:37.871727 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun May 23 12:41:37.871727 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) May 23 12:41:37.871727 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func May 23 12:41:37.871727 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self.get_or_create( May 23 12:41:37.871727 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create May 23 12:41:37.871727 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application with Lock( May 23 12:41:37.871727 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ May 23 12:41:37.871727 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self._enter() May 23 12:41:37.871727 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter May 23 12:41:37.871727 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) May 23 12:41:37.872620 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create May 23 12:41:37.872620 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self.creator() May 23 12:41:37.872620 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value May 23 12:41:37.872620 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application created_value = creator( May 23 12:41:37.872620 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role May 23 12:41:37.872620 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) May 23 12:41:37.872620 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role May 23 12:41:37.872620 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() May 23 12:41:37.872620 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role May 23 12:41:37.872620 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) May 23 12:41:37.872620 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. May 23 12:41:37.872620 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application May 23 12:41:37.872620 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 329/669] 10.222.0.30 () {56 vars in 1102 bytes} [Mon May 23 12:41:37 2022] GET /identity/v3/roles/service => generated 85 bytes in 162 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) May 23 12:41:37.873345 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6738efe-146f-44eb-b5d3-38d02931b086 None admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:41:37.879999 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b44d448f-eb18-4461-8554-fcabd1ef9b9d None admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:41:37.919410 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1826d8b-257c-4642-a9bb-e07978367a0e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:41:37.920124 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1826d8b-257c-4642-a9bb-e07978367a0e None admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:41:37.920402 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1826d8b-257c-4642-a9bb-e07978367a0e None admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:41:37.920683 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1826d8b-257c-4642-a9bb-e07978367a0e None admin] PATH_INFO: `/v3/domains/Default` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:41:37.932405 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1826d8b-257c-4642-a9bb-e07978367a0e None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:41:37.933848 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1826d8b-257c-4642-a9bb-e07978367a0e None admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:41:37.952156 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application [None req-c1826d8b-257c-4642-a9bb-e07978367a0e None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. May 23 12:41:37.952156 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application Traceback (most recent call last): May 23 12:41:37.952156 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request May 23 12:41:37.952156 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application rv = self.dispatch_request() May 23 12:41:37.952156 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request May 23 12:41:37.952156 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) May 23 12:41:37.952156 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper May 23 12:41:37.952156 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) May 23 12:41:37.952156 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view May 23 12:41:37.952156 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) May 23 12:41:37.952156 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request May 23 12:41:37.952156 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) May 23 12:41:37.952156 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get May 23 12:41:37.952156 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self._get_domain(domain_id) May 23 12:41:37.952156 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain May 23 12:41:37.958200 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) May 23 12:41:37.958200 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped May 23 12:41:37.958200 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) May 23 12:41:37.958200 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun May 23 12:41:37.958200 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) May 23 12:41:37.958200 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func May 23 12:41:37.958200 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self.get_or_create( May 23 12:41:37.958200 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create May 23 12:41:37.958200 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application with Lock( May 23 12:41:37.958200 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ May 23 12:41:37.958200 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self._enter() May 23 12:41:37.958200 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter May 23 12:41:37.958200 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) May 23 12:41:37.958200 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create May 23 12:41:37.958200 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self.creator() May 23 12:41:37.958200 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value May 23 12:41:37.958200 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application created_value = creator( May 23 12:41:37.959876 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain May 23 12:41:37.959876 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) May 23 12:41:37.959876 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. May 23 12:41:37.959876 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application May 23 12:41:37.959876 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 341/670] 10.222.0.30 () {56 vars in 1108 bytes} [Mon May 23 12:41:37 2022] GET /identity/v3/domains/Default => generated 87 bytes in 190 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) May 23 12:41:37.959876 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1826d8b-257c-4642-a9bb-e07978367a0e None admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:41:37.961060 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=100 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:41:37.995457 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6738efe-146f-44eb-b5d3-38d02931b086 None admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:41:38.033584 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6404d05-8b8d-4fba-aeba-e92093720aad None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:41:38.034281 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6404d05-8b8d-4fba-aeba-e92093720aad None admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:41:38.034569 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6404d05-8b8d-4fba-aeba-e92093720aad None admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:41:38.034846 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6404d05-8b8d-4fba-aeba-e92093720aad None admin] PATH_INFO: `/v3/role_assignments` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:41:38.035673 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6404d05-8b8d-4fba-aeba-e92093720aad None admin] RBAC: Adding query filter params (role.id=41ab28c1ae3d4941bd09fb5d35f1dc54, scope.project.id=9c389ae65aaa4542bb5a5c4fd0bacf5b, user.id=5cea0feed4254ec085854986f99acf30) {{(pid=92057) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} May 23 12:41:38.035988 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6404d05-8b8d-4fba-aeba-e92093720aad None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:41:38.038050 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6404d05-8b8d-4fba-aeba-e92093720aad None admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:41:38.044248 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 330/671] 10.222.0.30 () {56 vars in 1374 bytes} [Mon May 23 12:41:37 2022] GET /identity/v3/role_assignments?user.id=5cea0feed4254ec085854986f99acf30&scope.project.id=9c389ae65aaa4542bb5a5c4fd0bacf5b&role.id=41ab28c1ae3d4941bd09fb5d35f1dc54 => generated 608 bytes in 171 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) May 23 12:41:38.047473 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6404d05-8b8d-4fba-aeba-e92093720aad None admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:41:38.070785 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1826d8b-257c-4642-a9bb-e07978367a0e None admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:41:38.074783 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=100 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:41:38.116856 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7ba6ab7-6dad-4c5a-a831-f634749e2bb0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:41:38.117914 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7ba6ab7-6dad-4c5a-a831-f634749e2bb0 None admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:41:38.118397 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7ba6ab7-6dad-4c5a-a831-f634749e2bb0 None admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:41:38.118852 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7ba6ab7-6dad-4c5a-a831-f634749e2bb0 None admin] PATH_INFO: `/v3/domains` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:41:38.119762 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7ba6ab7-6dad-4c5a-a831-f634749e2bb0 None admin] RBAC: Adding query filter params (name=Default) {{(pid=92056) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} May 23 12:41:38.120278 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7ba6ab7-6dad-4c5a-a831-f634749e2bb0 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:41:38.121940 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7ba6ab7-6dad-4c5a-a831-f634749e2bb0 None admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:41:38.135927 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 342/672] 10.222.0.30 () {56 vars in 1109 bytes} [Mon May 23 12:41:37 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 182 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) May 23 12:41:38.137155 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7ba6ab7-6dad-4c5a-a831-f634749e2bb0 None admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:41:38.191571 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6404d05-8b8d-4fba-aeba-e92093720aad None admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:41:38.231080 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f4e6904-ec08-4b06-9a8f-bcdd842e3c59 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:41:38.231782 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f4e6904-ec08-4b06-9a8f-bcdd842e3c59 None admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:41:38.232068 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f4e6904-ec08-4b06-9a8f-bcdd842e3c59 None admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:41:38.232335 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f4e6904-ec08-4b06-9a8f-bcdd842e3c59 None admin] PATH_INFO: `/v3/roles` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:41:38.233070 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f4e6904-ec08-4b06-9a8f-bcdd842e3c59 None admin] RBAC: Adding query filter params (name=service) {{(pid=92057) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} May 23 12:41:38.233371 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f4e6904-ec08-4b06-9a8f-bcdd842e3c59 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:41:38.235745 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f4e6904-ec08-4b06-9a8f-bcdd842e3c59 None admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:41:38.238204 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=100 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:41:38.246485 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 331/673] 10.222.0.30 () {56 vars in 1103 bytes} [Mon May 23 12:41:38 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 200 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) May 23 12:41:38.247410 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f4e6904-ec08-4b06-9a8f-bcdd842e3c59 None admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:41:38.250968 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7ba6ab7-6dad-4c5a-a831-f634749e2bb0 None admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:41:38.291891 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-136dbd90-666a-4ff9-bda2-9c96461b19c8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:41:38.292990 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-136dbd90-666a-4ff9-bda2-9c96461b19c8 None admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:41:38.293467 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-136dbd90-666a-4ff9-bda2-9c96461b19c8 None admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:41:38.294040 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-136dbd90-666a-4ff9-bda2-9c96461b19c8 None admin] PATH_INFO: `/v3/domains` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:41:38.294961 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-136dbd90-666a-4ff9-bda2-9c96461b19c8 None admin] RBAC: Adding query filter params (name=Default) {{(pid=92056) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} May 23 12:41:38.295482 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-136dbd90-666a-4ff9-bda2-9c96461b19c8 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:41:38.297056 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-136dbd90-666a-4ff9-bda2-9c96461b19c8 None admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:41:38.314586 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 343/674] 10.222.0.30 () {56 vars in 1109 bytes} [Mon May 23 12:41:38 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 178 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) May 23 12:41:38.315834 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-136dbd90-666a-4ff9-bda2-9c96461b19c8 None admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:41:38.345257 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=100 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:41:38.370649 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f4e6904-ec08-4b06-9a8f-bcdd842e3c59 None admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:41:38.416479 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b94f1835-9283-49d3-8792-a9611e5e4595 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:41:38.418095 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b94f1835-9283-49d3-8792-a9611e5e4595 None admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:41:38.418768 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b94f1835-9283-49d3-8792-a9611e5e4595 None admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:41:38.419209 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b94f1835-9283-49d3-8792-a9611e5e4595 None admin] PATH_INFO: `/v3/projects/service` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:41:38.428638 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b94f1835-9283-49d3-8792-a9611e5e4595 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:41:38.430846 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b94f1835-9283-49d3-8792-a9611e5e4595 None admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:41:38.436585 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-136dbd90-666a-4ff9-bda2-9c96461b19c8 None admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:41:38.449653 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application [None req-b94f1835-9283-49d3-8792-a9611e5e4595 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. May 23 12:41:38.449653 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application Traceback (most recent call last): May 23 12:41:38.449653 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request May 23 12:41:38.449653 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application rv = self.dispatch_request() May 23 12:41:38.449653 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request May 23 12:41:38.449653 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) May 23 12:41:38.449653 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper May 23 12:41:38.449653 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) May 23 12:41:38.449653 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view May 23 12:41:38.449653 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) May 23 12:41:38.449653 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request May 23 12:41:38.449653 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) May 23 12:41:38.449653 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get May 23 12:41:38.449653 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self._get_project(project_id) May 23 12:41:38.449653 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project May 23 12:41:38.450782 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) May 23 12:41:38.450782 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped May 23 12:41:38.450782 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) May 23 12:41:38.450782 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun May 23 12:41:38.450782 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) May 23 12:41:38.450782 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func May 23 12:41:38.450782 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self.get_or_create( May 23 12:41:38.450782 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create May 23 12:41:38.450782 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application with Lock( May 23 12:41:38.450782 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ May 23 12:41:38.450782 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self._enter() May 23 12:41:38.450782 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter May 23 12:41:38.450782 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) May 23 12:41:38.450782 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create May 23 12:41:38.450782 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self.creator() May 23 12:41:38.450782 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value May 23 12:41:38.450782 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application created_value = creator( May 23 12:41:38.451894 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project May 23 12:41:38.451894 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) May 23 12:41:38.451894 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project May 23 12:41:38.451894 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() May 23 12:41:38.451894 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project May 23 12:41:38.451894 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) May 23 12:41:38.451894 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. May 23 12:41:38.451894 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application May 23 12:41:38.454277 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 332/675] 10.222.0.30 () {56 vars in 1111 bytes} [Mon May 23 12:41:38 2022] GET /identity/v3/projects/service => generated 88 bytes in 207 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) May 23 12:41:38.455192 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b94f1835-9283-49d3-8792-a9611e5e4595 None admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:41:38.483754 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bedec6f8-234d-4ed3-9fdd-1565b071000c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:41:38.484874 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bedec6f8-234d-4ed3-9fdd-1565b071000c None admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:41:38.485423 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bedec6f8-234d-4ed3-9fdd-1565b071000c None admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:41:38.485844 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bedec6f8-234d-4ed3-9fdd-1565b071000c None admin] PATH_INFO: `/v3/domains/Default` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:41:38.499785 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bedec6f8-234d-4ed3-9fdd-1565b071000c None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:41:38.502059 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bedec6f8-234d-4ed3-9fdd-1565b071000c None admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:41:38.525717 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application [None req-bedec6f8-234d-4ed3-9fdd-1565b071000c None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. May 23 12:41:38.525717 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application Traceback (most recent call last): May 23 12:41:38.525717 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request May 23 12:41:38.525717 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application rv = self.dispatch_request() May 23 12:41:38.525717 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request May 23 12:41:38.525717 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) May 23 12:41:38.525717 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper May 23 12:41:38.525717 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) May 23 12:41:38.525717 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view May 23 12:41:38.525717 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) May 23 12:41:38.525717 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request May 23 12:41:38.525717 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) May 23 12:41:38.525717 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get May 23 12:41:38.525717 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self._get_domain(domain_id) May 23 12:41:38.525717 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain May 23 12:41:38.526882 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) May 23 12:41:38.526882 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped May 23 12:41:38.526882 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) May 23 12:41:38.526882 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun May 23 12:41:38.526882 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) May 23 12:41:38.526882 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func May 23 12:41:38.526882 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self.get_or_create( May 23 12:41:38.526882 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create May 23 12:41:38.526882 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application with Lock( May 23 12:41:38.526882 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ May 23 12:41:38.526882 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self._enter() May 23 12:41:38.526882 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter May 23 12:41:38.526882 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) May 23 12:41:38.526882 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create May 23 12:41:38.526882 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self.creator() May 23 12:41:38.526882 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value May 23 12:41:38.526882 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application created_value = creator( May 23 12:41:38.528264 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain May 23 12:41:38.528264 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) May 23 12:41:38.528264 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. May 23 12:41:38.528264 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application May 23 12:41:38.530542 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 344/676] 10.222.0.30 () {56 vars in 1108 bytes} [Mon May 23 12:41:38 2022] GET /identity/v3/domains/Default => generated 87 bytes in 215 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) May 23 12:41:38.531472 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bedec6f8-234d-4ed3-9fdd-1565b071000c None admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:41:38.538603 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=100 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:41:38.588892 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b94f1835-9283-49d3-8792-a9611e5e4595 None admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:41:38.631201 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d902ee4a-6444-4012-94d6-78be84e7a3c0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:41:38.632797 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d902ee4a-6444-4012-94d6-78be84e7a3c0 None admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:41:38.633079 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d902ee4a-6444-4012-94d6-78be84e7a3c0 None admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:41:38.633518 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d902ee4a-6444-4012-94d6-78be84e7a3c0 None admin] PATH_INFO: `/v3/projects/service` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:41:38.635368 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=100 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:41:38.642689 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d902ee4a-6444-4012-94d6-78be84e7a3c0 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:41:38.649571 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d902ee4a-6444-4012-94d6-78be84e7a3c0 None admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:41:38.658954 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application [None req-d902ee4a-6444-4012-94d6-78be84e7a3c0 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. May 23 12:41:38.658954 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application Traceback (most recent call last): May 23 12:41:38.658954 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request May 23 12:41:38.658954 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application rv = self.dispatch_request() May 23 12:41:38.658954 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request May 23 12:41:38.658954 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) May 23 12:41:38.658954 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper May 23 12:41:38.658954 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) May 23 12:41:38.658954 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view May 23 12:41:38.658954 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) May 23 12:41:38.658954 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request May 23 12:41:38.658954 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) May 23 12:41:38.658954 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get May 23 12:41:38.658954 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self._get_project(project_id) May 23 12:41:38.658954 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project May 23 12:41:38.660300 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) May 23 12:41:38.660300 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped May 23 12:41:38.660300 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) May 23 12:41:38.660300 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun May 23 12:41:38.660300 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) May 23 12:41:38.660300 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func May 23 12:41:38.660300 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self.get_or_create( May 23 12:41:38.660300 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create May 23 12:41:38.660300 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application with Lock( May 23 12:41:38.660300 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ May 23 12:41:38.660300 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self._enter() May 23 12:41:38.660300 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter May 23 12:41:38.660300 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) May 23 12:41:38.660300 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create May 23 12:41:38.660300 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self.creator() May 23 12:41:38.660300 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value May 23 12:41:38.660300 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application created_value = creator( May 23 12:41:38.661259 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project May 23 12:41:38.661259 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) May 23 12:41:38.661259 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project May 23 12:41:38.661259 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() May 23 12:41:38.661259 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project May 23 12:41:38.661259 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) May 23 12:41:38.661259 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. May 23 12:41:38.661259 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application May 23 12:41:38.661723 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bedec6f8-234d-4ed3-9fdd-1565b071000c None admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:41:38.663975 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 333/677] 10.222.0.30 () {56 vars in 1111 bytes} [Mon May 23 12:41:38 2022] GET /identity/v3/projects/service => generated 88 bytes in 209 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) May 23 12:41:38.664951 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d902ee4a-6444-4012-94d6-78be84e7a3c0 None admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:41:38.709225 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e13ff9ee-2d2f-46f9-9467-fbb83cac4840 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:41:38.710299 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e13ff9ee-2d2f-46f9-9467-fbb83cac4840 None admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:41:38.710679 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e13ff9ee-2d2f-46f9-9467-fbb83cac4840 None admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:41:38.711029 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e13ff9ee-2d2f-46f9-9467-fbb83cac4840 None admin] PATH_INFO: `/v3/projects` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:41:38.711893 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e13ff9ee-2d2f-46f9-9467-fbb83cac4840 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=92056) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} May 23 12:41:38.712278 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e13ff9ee-2d2f-46f9-9467-fbb83cac4840 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:41:38.714145 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e13ff9ee-2d2f-46f9-9467-fbb83cac4840 None admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:41:38.731877 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 345/678] 10.222.0.30 () {56 vars in 1148 bytes} [Mon May 23 12:41:38 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 200 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) May 23 12:41:38.732849 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e13ff9ee-2d2f-46f9-9467-fbb83cac4840 None admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:41:38.781258 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d902ee4a-6444-4012-94d6-78be84e7a3c0 None admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:41:38.800236 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=100 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:41:38.817697 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17802d4b-3986-446c-8b17-26d2735155da None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:41:38.819237 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17802d4b-3986-446c-8b17-26d2735155da None admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:41:38.819899 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17802d4b-3986-446c-8b17-26d2735155da None admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:41:38.820351 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17802d4b-3986-446c-8b17-26d2735155da None admin] PATH_INFO: `/v3/domains` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:41:38.821262 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17802d4b-3986-446c-8b17-26d2735155da None admin] RBAC: Adding query filter params (name=Default) {{(pid=92057) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} May 23 12:41:38.821740 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17802d4b-3986-446c-8b17-26d2735155da None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:41:38.823299 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17802d4b-3986-446c-8b17-26d2735155da None admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:41:38.836809 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 334/679] 10.222.0.30 () {56 vars in 1109 bytes} [Mon May 23 12:41:38 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 172 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) May 23 12:41:38.837951 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17802d4b-3986-446c-8b17-26d2735155da None admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:41:38.866996 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e13ff9ee-2d2f-46f9-9467-fbb83cac4840 None admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:41:38.904813 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78ac7aed-43f8-4e25-bc52-70f3123ef12d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:41:38.905774 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78ac7aed-43f8-4e25-bc52-70f3123ef12d None admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:41:38.906287 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78ac7aed-43f8-4e25-bc52-70f3123ef12d None admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:41:38.906793 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78ac7aed-43f8-4e25-bc52-70f3123ef12d None admin] PATH_INFO: `/v3/projects` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:41:38.907928 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-78ac7aed-43f8-4e25-bc52-70f3123ef12d None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=92056) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} May 23 12:41:38.908431 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-78ac7aed-43f8-4e25-bc52-70f3123ef12d None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:41:38.910115 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-78ac7aed-43f8-4e25-bc52-70f3123ef12d None admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:41:38.913602 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=100 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:41:38.929309 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 346/680] 10.222.0.30 () {56 vars in 1148 bytes} [Mon May 23 12:41:38 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 191 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) May 23 12:41:38.930527 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78ac7aed-43f8-4e25-bc52-70f3123ef12d None admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:41:38.962181 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17802d4b-3986-446c-8b17-26d2735155da None admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:41:38.998112 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06049129-23ba-4cac-ac9e-26def3739eb9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:41:38.999086 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06049129-23ba-4cac-ac9e-26def3739eb9 None admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:41:38.999553 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06049129-23ba-4cac-ac9e-26def3739eb9 None admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:41:39.000027 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06049129-23ba-4cac-ac9e-26def3739eb9 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:41:39.001026 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06049129-23ba-4cac-ac9e-26def3739eb9 None admin] RBAC: Adding query filter params (role.id=41ab28c1ae3d4941bd09fb5d35f1dc54, scope.project.id=9c389ae65aaa4542bb5a5c4fd0bacf5b, user.id=2eb83ca38e79449095f3a2f947bbb06a) {{(pid=92057) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} May 23 12:41:39.001609 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06049129-23ba-4cac-ac9e-26def3739eb9 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:41:39.003324 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06049129-23ba-4cac-ac9e-26def3739eb9 None admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:41:39.010864 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 335/681] 10.222.0.30 () {56 vars in 1374 bytes} [Mon May 23 12:41:38 2022] GET /identity/v3/role_assignments?user.id=2eb83ca38e79449095f3a2f947bbb06a&scope.project.id=9c389ae65aaa4542bb5a5c4fd0bacf5b&role.id=41ab28c1ae3d4941bd09fb5d35f1dc54 => generated 608 bytes in 173 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) May 23 12:41:39.011950 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06049129-23ba-4cac-ac9e-26def3739eb9 None admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:41:39.071183 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78ac7aed-43f8-4e25-bc52-70f3123ef12d None admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:41:39.080015 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=100 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:41:39.122942 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2320a345-5fcf-4b23-8602-29306927b170 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:41:39.124431 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2320a345-5fcf-4b23-8602-29306927b170 None admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:41:39.124996 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2320a345-5fcf-4b23-8602-29306927b170 None admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:41:39.126102 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2320a345-5fcf-4b23-8602-29306927b170 None admin] PATH_INFO: `/v3/users/swift` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:41:39.138371 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2320a345-5fcf-4b23-8602-29306927b170 None admin] RBAC: Authorizing `identity:get_user(user_id=swift)` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:41:39.140945 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2320a345-5fcf-4b23-8602-29306927b170 None admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:41:39.149146 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06049129-23ba-4cac-ac9e-26def3739eb9 None admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:41:39.164452 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application [None req-2320a345-5fcf-4b23-8602-29306927b170 None admin] Could not find user: swift.: keystone.exception.UserNotFound: Could not find user: swift. May 23 12:41:39.164452 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application Traceback (most recent call last): May 23 12:41:39.164452 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request May 23 12:41:39.164452 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application rv = self.dispatch_request() May 23 12:41:39.164452 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request May 23 12:41:39.164452 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) May 23 12:41:39.164452 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper May 23 12:41:39.164452 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) May 23 12:41:39.164452 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view May 23 12:41:39.164452 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) May 23 12:41:39.164452 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request May 23 12:41:39.164452 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) May 23 12:41:39.164452 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get May 23 12:41:39.164452 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self._get_user(user_id) May 23 12:41:39.164452 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user May 23 12:41:39.164452 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) May 23 12:41:39.165541 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped May 23 12:41:39.165541 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) May 23 12:41:39.165541 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper May 23 12:41:39.165541 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) May 23 12:41:39.165541 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper May 23 12:41:39.165541 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) May 23 12:41:39.165541 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun May 23 12:41:39.165541 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) May 23 12:41:39.165541 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func May 23 12:41:39.165541 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self.get_or_create( May 23 12:41:39.165541 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create May 23 12:41:39.165541 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application with Lock( May 23 12:41:39.165541 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ May 23 12:41:39.165541 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self._enter() May 23 12:41:39.165541 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter May 23 12:41:39.165541 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) May 23 12:41:39.165541 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create May 23 12:41:39.165541 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self.creator() May 23 12:41:39.166575 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value May 23 12:41:39.166575 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application created_value = creator( May 23 12:41:39.166575 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user May 23 12:41:39.166575 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) May 23 12:41:39.166575 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user May 23 12:41:39.166575 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) May 23 12:41:39.166575 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user May 23 12:41:39.166575 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) May 23 12:41:39.166575 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: swift. May 23 12:41:39.166575 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application May 23 12:41:39.167805 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 347/682] 10.222.0.30 () {56 vars in 1096 bytes} [Mon May 23 12:41:38 2022] GET /identity/v3/users/swift => generated 83 bytes in 237 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) May 23 12:41:39.170010 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42712441-9481-407e-990f-29773e4b8023 None None] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:41:39.170296 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42712441-9481-407e-990f-29773e4b8023 None None] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:41:39.170569 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42712441-9481-407e-990f-29773e4b8023 None None] PATH_INFO: `/` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:41:39.176212 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 348/683] 10.222.0.30 () {54 vars in 926 bytes} [Mon May 23 12:41:39 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) May 23 12:41:39.177120 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42712441-9481-407e-990f-29773e4b8023 None None] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:41:39.193982 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35733de1-5f5e-459f-ad77-0b5be76e7c22 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:41:39.195173 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35733de1-5f5e-459f-ad77-0b5be76e7c22 None admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:41:39.195696 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35733de1-5f5e-459f-ad77-0b5be76e7c22 None admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:41:39.196188 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35733de1-5f5e-459f-ad77-0b5be76e7c22 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:41:39.197281 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35733de1-5f5e-459f-ad77-0b5be76e7c22 None admin] RBAC: Adding query filter params (role.id=41ab28c1ae3d4941bd09fb5d35f1dc54, scope.project.id=9c389ae65aaa4542bb5a5c4fd0bacf5b, user.id=1666097f8ed545c6b0e4809c93e4775a) {{(pid=92057) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} May 23 12:41:39.197799 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35733de1-5f5e-459f-ad77-0b5be76e7c22 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:41:39.199715 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35733de1-5f5e-459f-ad77-0b5be76e7c22 None admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:41:39.211289 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 336/684] 10.222.0.30 () {56 vars in 1374 bytes} [Mon May 23 12:41:39 2022] GET /identity/v3/role_assignments?user.id=1666097f8ed545c6b0e4809c93e4775a&scope.project.id=9c389ae65aaa4542bb5a5c4fd0bacf5b&role.id=41ab28c1ae3d4941bd09fb5d35f1dc54 => generated 608 bytes in 195 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) May 23 12:41:39.214004 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0d209d2-83aa-4700-bdf7-a5947cdff471 None None] REQUEST_METHOD: `POST` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:41:39.214004 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0d209d2-83aa-4700-bdf7-a5947cdff471 None None] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:41:39.214004 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0d209d2-83aa-4700-bdf7-a5947cdff471 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:41:39.242321 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.auth.core [None req-e0d209d2-83aa-4700-bdf7-a5947cdff471 None None] MFA Rules not processed for user `0f7119f8fa5c4445977671140ded91a4`. Rule list: `[]` (Enabled: `True`). {{(pid=92057) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 12:41:39.243563 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.fernet_utils [None req-e0d209d2-83aa-4700-bdf7-a5947cdff471 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92057) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 12:41:39.261924 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=100 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:41:39.323509 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42712441-9481-407e-990f-29773e4b8023 None None] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:41:39.339643 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 337/685] 10.222.0.30 () {58 vars in 1023 bytes} [Mon May 23 12:41:39 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 129 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) May 23 12:41:39.374324 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0d209d2-83aa-4700-bdf7-a5947cdff471 None None] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:41:39.375264 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fbfb36a6-b45d-4362-948a-695f87c9a263 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:41:39.376144 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbfb36a6-b45d-4362-948a-695f87c9a263 None admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:41:39.376577 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbfb36a6-b45d-4362-948a-695f87c9a263 None admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:41:39.377003 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbfb36a6-b45d-4362-948a-695f87c9a263 None admin] PATH_INFO: `/v3/users` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:41:39.378047 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fbfb36a6-b45d-4362-948a-695f87c9a263 None admin] RBAC: Adding query filter params (domain_id=default, name=swift) {{(pid=92056) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} May 23 12:41:39.378506 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fbfb36a6-b45d-4362-948a-695f87c9a263 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:41:39.380160 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fbfb36a6-b45d-4362-948a-695f87c9a263 None admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:41:39.402913 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 349/686] 10.222.0.30 () {56 vars in 1135 bytes} [Mon May 23 12:41:39 2022] GET /identity/v3/users?domain_id=default&name=swift => generated 370 bytes in 226 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) May 23 12:41:39.409040 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fbfb36a6-b45d-4362-948a-695f87c9a263 None admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:41:39.429457 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=100 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:41:39.506717 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0d209d2-83aa-4700-bdf7-a5947cdff471 None None] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:41:39.531674 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fbfb36a6-b45d-4362-948a-695f87c9a263 None admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:41:39.544749 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f5e3c2a-6fc9-4a3c-95fb-04719d8959e3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:41:39.545558 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f5e3c2a-6fc9-4a3c-95fb-04719d8959e3 None admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:41:39.545842 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f5e3c2a-6fc9-4a3c-95fb-04719d8959e3 None admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:41:39.546151 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f5e3c2a-6fc9-4a3c-95fb-04719d8959e3 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:41:39.557267 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f5e3c2a-6fc9-4a3c-95fb-04719d8959e3 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} May 23 12:41:39.557867 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=100 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:41:39.565978 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application [None req-5f5e3c2a-6fc9-4a3c-95fb-04719d8959e3 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. May 23 12:41:39.565978 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application Traceback (most recent call last): May 23 12:41:39.565978 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request May 23 12:41:39.565978 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application rv = self.dispatch_request() May 23 12:41:39.565978 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request May 23 12:41:39.565978 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) May 23 12:41:39.565978 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper May 23 12:41:39.565978 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) May 23 12:41:39.565978 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view May 23 12:41:39.565978 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) May 23 12:41:39.565978 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request May 23 12:41:39.565978 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) May 23 12:41:39.565978 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get May 23 12:41:39.565978 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self._get_role(role_id) May 23 12:41:39.565978 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role May 23 12:41:39.565978 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') May 23 12:41:39.572942 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call May 23 12:41:39.572942 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( May 23 12:41:39.572942 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data May 23 12:41:39.572942 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) May 23 12:41:39.572942 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped May 23 12:41:39.572942 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) May 23 12:41:39.572942 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun May 23 12:41:39.572942 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) May 23 12:41:39.572942 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func May 23 12:41:39.572942 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self.get_or_create( May 23 12:41:39.572942 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create May 23 12:41:39.572942 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application with Lock( May 23 12:41:39.572942 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ May 23 12:41:39.572942 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self._enter() May 23 12:41:39.572942 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter May 23 12:41:39.572942 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) May 23 12:41:39.573842 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create May 23 12:41:39.573842 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self.creator() May 23 12:41:39.573842 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value May 23 12:41:39.573842 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application created_value = creator( May 23 12:41:39.573842 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role May 23 12:41:39.573842 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) May 23 12:41:39.573842 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role May 23 12:41:39.573842 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() May 23 12:41:39.573842 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role May 23 12:41:39.573842 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) May 23 12:41:39.573842 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. May 23 12:41:39.573842 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application May 23 12:41:39.573842 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 338/687] 10.222.0.30 () {56 vars in 1096 bytes} [Mon May 23 12:41:39 2022] GET /identity/v3/roles/admin => generated 83 bytes in 193 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) May 23 12:41:39.574948 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c95c72a-6e96-4017-bf48-9d7c1fda2a30 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:41:39.574948 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c95c72a-6e96-4017-bf48-9d7c1fda2a30 None admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:41:39.574948 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c95c72a-6e96-4017-bf48-9d7c1fda2a30 None admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:41:39.574948 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c95c72a-6e96-4017-bf48-9d7c1fda2a30 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:41:39.575224 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f5e3c2a-6fc9-4a3c-95fb-04719d8959e3 None admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:41:39.656226 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c95c72a-6e96-4017-bf48-9d7c1fda2a30 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:41:39.657685 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c95c72a-6e96-4017-bf48-9d7c1fda2a30 None admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:41:39.684334 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f5e3c2a-6fc9-4a3c-95fb-04719d8959e3 None admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:41:39.694923 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=100 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:41:39.721901 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e9f4295-0b5d-4879-88e3-ad72b9f707a8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:41:39.722942 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e9f4295-0b5d-4879-88e3-ad72b9f707a8 None admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:41:39.723432 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e9f4295-0b5d-4879-88e3-ad72b9f707a8 None admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:41:39.723910 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e9f4295-0b5d-4879-88e3-ad72b9f707a8 None admin] PATH_INFO: `/v3/roles` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:41:39.725054 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e9f4295-0b5d-4879-88e3-ad72b9f707a8 None admin] RBAC: Adding query filter params (name=admin) {{(pid=92057) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} May 23 12:41:39.725561 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e9f4295-0b5d-4879-88e3-ad72b9f707a8 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:41:39.727283 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e9f4295-0b5d-4879-88e3-ad72b9f707a8 None admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:41:39.737122 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 339/688] 10.222.0.30 () {56 vars in 1099 bytes} [Mon May 23 12:41:39 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 164 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) May 23 12:41:39.742720 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e9f4295-0b5d-4879-88e3-ad72b9f707a8 None admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:41:39.838811 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 350/689] 10.222.0.30 () {58 vars in 1282 bytes} [Mon May 23 12:41:39 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 430 msecs (HTTP/1.1 200) 6 headers in 358 bytes (2 switches on core 0) May 23 12:41:39.847044 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c95c72a-6e96-4017-bf48-9d7c1fda2a30 None admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:41:39.875795 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=100 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:41:39.942202 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e9f4295-0b5d-4879-88e3-ad72b9f707a8 None admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:41:39.982781 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd0b1a9a-c00e-4558-94da-c65527952a57 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:41:39.983506 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd0b1a9a-c00e-4558-94da-c65527952a57 None admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:41:39.983840 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd0b1a9a-c00e-4558-94da-c65527952a57 None admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:41:39.984122 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd0b1a9a-c00e-4558-94da-c65527952a57 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:41:39.995496 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd0b1a9a-c00e-4558-94da-c65527952a57 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:41:39.996979 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd0b1a9a-c00e-4558-94da-c65527952a57 None admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:41:40.011500 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c95c72a-6e96-4017-bf48-9d7c1fda2a30 None admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:41:40.020894 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application [None req-dd0b1a9a-c00e-4558-94da-c65527952a57 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. May 23 12:41:40.020894 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application Traceback (most recent call last): May 23 12:41:40.020894 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request May 23 12:41:40.020894 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application rv = self.dispatch_request() May 23 12:41:40.020894 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request May 23 12:41:40.020894 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) May 23 12:41:40.020894 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper May 23 12:41:40.020894 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) May 23 12:41:40.020894 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view May 23 12:41:40.020894 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) May 23 12:41:40.020894 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request May 23 12:41:40.020894 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) May 23 12:41:40.020894 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get May 23 12:41:40.020894 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self._get_domain(domain_id) May 23 12:41:40.020894 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain May 23 12:41:40.030971 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) May 23 12:41:40.030971 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped May 23 12:41:40.030971 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) May 23 12:41:40.030971 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun May 23 12:41:40.030971 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) May 23 12:41:40.030971 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func May 23 12:41:40.030971 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self.get_or_create( May 23 12:41:40.030971 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create May 23 12:41:40.030971 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application with Lock( May 23 12:41:40.030971 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ May 23 12:41:40.030971 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self._enter() May 23 12:41:40.030971 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter May 23 12:41:40.030971 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) May 23 12:41:40.030971 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create May 23 12:41:40.030971 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self.creator() May 23 12:41:40.030971 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value May 23 12:41:40.030971 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application created_value = creator( May 23 12:41:40.031942 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain May 23 12:41:40.031942 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) May 23 12:41:40.031942 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. May 23 12:41:40.031942 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application May 23 12:41:40.031942 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 340/690] 10.222.0.30 () {56 vars in 1108 bytes} [Mon May 23 12:41:39 2022] GET /identity/v3/domains/Default => generated 87 bytes in 280 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) May 23 12:41:40.031942 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd0b1a9a-c00e-4558-94da-c65527952a57 None admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:41:40.050675 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d502132e-63d2-4ebe-9763-9c76decdaec3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:41:40.051564 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d502132e-63d2-4ebe-9763-9c76decdaec3 None admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:41:40.051920 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d502132e-63d2-4ebe-9763-9c76decdaec3 None admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:41:40.052266 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d502132e-63d2-4ebe-9763-9c76decdaec3 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:41:40.054765 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=100 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:41:40.065085 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d502132e-63d2-4ebe-9763-9c76decdaec3 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:41:40.066691 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d502132e-63d2-4ebe-9763-9c76decdaec3 None admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:41:40.091093 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application [None req-d502132e-63d2-4ebe-9763-9c76decdaec3 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. May 23 12:41:40.091093 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application Traceback (most recent call last): May 23 12:41:40.091093 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request May 23 12:41:40.091093 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application rv = self.dispatch_request() May 23 12:41:40.091093 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request May 23 12:41:40.091093 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) May 23 12:41:40.091093 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper May 23 12:41:40.091093 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) May 23 12:41:40.091093 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view May 23 12:41:40.091093 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) May 23 12:41:40.091093 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request May 23 12:41:40.091093 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) May 23 12:41:40.091093 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get May 23 12:41:40.091093 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self._get_domain(domain_id) May 23 12:41:40.091093 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain May 23 12:41:40.092454 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) May 23 12:41:40.092454 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped May 23 12:41:40.092454 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) May 23 12:41:40.092454 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun May 23 12:41:40.092454 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) May 23 12:41:40.092454 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func May 23 12:41:40.092454 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self.get_or_create( May 23 12:41:40.092454 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create May 23 12:41:40.092454 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application with Lock( May 23 12:41:40.092454 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ May 23 12:41:40.092454 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self._enter() May 23 12:41:40.092454 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter May 23 12:41:40.092454 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) May 23 12:41:40.092454 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create May 23 12:41:40.092454 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self.creator() May 23 12:41:40.092454 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value May 23 12:41:40.092454 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application created_value = creator( May 23 12:41:40.094249 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain May 23 12:41:40.094249 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) May 23 12:41:40.094249 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. May 23 12:41:40.094249 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application May 23 12:41:40.094249 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 351/691] 10.222.0.30 () {56 vars in 1108 bytes} [Mon May 23 12:41:39 2022] GET /identity/v3/domains/Default => generated 87 bytes in 246 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) May 23 12:41:40.099121 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d502132e-63d2-4ebe-9763-9c76decdaec3 None admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:41:40.206241 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=100 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:41:40.215024 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd0b1a9a-c00e-4558-94da-c65527952a57 None admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:41:40.252413 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34c35a36-8bd0-4ff8-8130-91a0e5dc332a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:41:40.253136 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34c35a36-8bd0-4ff8-8130-91a0e5dc332a None admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:41:40.253425 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34c35a36-8bd0-4ff8-8130-91a0e5dc332a None admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:41:40.253752 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34c35a36-8bd0-4ff8-8130-91a0e5dc332a None admin] PATH_INFO: `/v3/domains` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:41:40.254527 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34c35a36-8bd0-4ff8-8130-91a0e5dc332a None admin] RBAC: Adding query filter params (name=Default) {{(pid=92057) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} May 23 12:41:40.254837 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34c35a36-8bd0-4ff8-8130-91a0e5dc332a None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:41:40.256514 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34c35a36-8bd0-4ff8-8130-91a0e5dc332a None admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:41:40.268867 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 341/692] 10.222.0.30 () {56 vars in 1109 bytes} [Mon May 23 12:41:40 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 241 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) May 23 12:41:40.273774 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d502132e-63d2-4ebe-9763-9c76decdaec3 None admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:41:40.275011 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34c35a36-8bd0-4ff8-8130-91a0e5dc332a None admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:41:40.351034 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=100 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:41:40.380131 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34c35a36-8bd0-4ff8-8130-91a0e5dc332a None admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:41:40.381641 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2543559-330b-40c8-92f3-3ba3bed6487f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:41:40.382677 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2543559-330b-40c8-92f3-3ba3bed6487f None admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:41:40.390600 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2543559-330b-40c8-92f3-3ba3bed6487f None admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:41:40.390600 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2543559-330b-40c8-92f3-3ba3bed6487f None admin] PATH_INFO: `/v3/domains` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:41:40.390600 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2543559-330b-40c8-92f3-3ba3bed6487f None admin] RBAC: Adding query filter params (name=Default) {{(pid=92056) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} May 23 12:41:40.390600 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2543559-330b-40c8-92f3-3ba3bed6487f None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:41:40.390600 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2543559-330b-40c8-92f3-3ba3bed6487f None admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:41:40.416342 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 352/693] 10.222.0.30 () {56 vars in 1109 bytes} [Mon May 23 12:41:40 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 317 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) May 23 12:41:40.425718 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1773f6f-23fb-4bfb-aa06-2f9e09365729 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:41:40.426578 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1773f6f-23fb-4bfb-aa06-2f9e09365729 None admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:41:40.426866 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1773f6f-23fb-4bfb-aa06-2f9e09365729 None admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:41:40.427146 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1773f6f-23fb-4bfb-aa06-2f9e09365729 None admin] PATH_INFO: `/v3/users/nova` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:41:40.427807 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2543559-330b-40c8-92f3-3ba3bed6487f None admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:41:40.437859 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1773f6f-23fb-4bfb-aa06-2f9e09365729 None admin] RBAC: Authorizing `identity:get_user(user_id=nova)` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:41:40.439458 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1773f6f-23fb-4bfb-aa06-2f9e09365729 None admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:41:40.455241 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application [None req-b1773f6f-23fb-4bfb-aa06-2f9e09365729 None admin] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova. May 23 12:41:40.455241 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application Traceback (most recent call last): May 23 12:41:40.455241 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request May 23 12:41:40.455241 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application rv = self.dispatch_request() May 23 12:41:40.455241 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request May 23 12:41:40.455241 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) May 23 12:41:40.455241 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper May 23 12:41:40.455241 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) May 23 12:41:40.455241 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view May 23 12:41:40.455241 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) May 23 12:41:40.455241 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request May 23 12:41:40.455241 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) May 23 12:41:40.455241 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get May 23 12:41:40.455241 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self._get_user(user_id) May 23 12:41:40.455241 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user May 23 12:41:40.455241 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) May 23 12:41:40.456286 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped May 23 12:41:40.456286 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) May 23 12:41:40.456286 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper May 23 12:41:40.456286 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) May 23 12:41:40.456286 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper May 23 12:41:40.456286 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) May 23 12:41:40.456286 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun May 23 12:41:40.456286 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) May 23 12:41:40.456286 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func May 23 12:41:40.456286 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self.get_or_create( May 23 12:41:40.456286 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create May 23 12:41:40.456286 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application with Lock( May 23 12:41:40.456286 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ May 23 12:41:40.456286 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self._enter() May 23 12:41:40.456286 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter May 23 12:41:40.456286 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) May 23 12:41:40.456286 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create May 23 12:41:40.456286 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self.creator() May 23 12:41:40.457262 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value May 23 12:41:40.457262 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application created_value = creator( May 23 12:41:40.457262 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user May 23 12:41:40.457262 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) May 23 12:41:40.457262 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user May 23 12:41:40.457262 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) May 23 12:41:40.457262 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user May 23 12:41:40.457262 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) May 23 12:41:40.457262 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: nova. May 23 12:41:40.457262 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application May 23 12:41:40.457262 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 342/694] 10.222.0.30 () {56 vars in 1093 bytes} [Mon May 23 12:41:40 2022] GET /identity/v3/users/nova => generated 82 bytes in 182 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) May 23 12:41:40.462643 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1773f6f-23fb-4bfb-aa06-2f9e09365729 None admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:41:40.598299 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1773f6f-23fb-4bfb-aa06-2f9e09365729 None admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:41:40.630287 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=100 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:41:40.638578 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb38ebbf-aa39-4394-bd7b-9c05bfcb60bb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:41:40.639461 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb38ebbf-aa39-4394-bd7b-9c05bfcb60bb None admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:41:40.639754 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb38ebbf-aa39-4394-bd7b-9c05bfcb60bb None admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:41:40.640033 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb38ebbf-aa39-4394-bd7b-9c05bfcb60bb None admin] PATH_INFO: `/v3/users` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:41:40.640881 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb38ebbf-aa39-4394-bd7b-9c05bfcb60bb None admin] RBAC: Adding query filter params (domain_id=default, name=nova) {{(pid=92057) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} May 23 12:41:40.641703 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb38ebbf-aa39-4394-bd7b-9c05bfcb60bb None admin] RBAC: Authorizing `identity:list_users()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:41:40.643262 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb38ebbf-aa39-4394-bd7b-9c05bfcb60bb None admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:41:40.664398 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 343/695] 10.222.0.30 () {56 vars in 1133 bytes} [Mon May 23 12:41:40 2022] GET /identity/v3/users?domain_id=default&name=nova => generated 368 bytes in 202 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) May 23 12:41:40.670005 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb38ebbf-aa39-4394-bd7b-9c05bfcb60bb None admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:41:40.681665 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2543559-330b-40c8-92f3-3ba3bed6487f None admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:41:40.720998 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98371f66-59b4-4f69-8d22-c9849c28d6b9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:41:40.721856 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98371f66-59b4-4f69-8d22-c9849c28d6b9 None admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:41:40.722175 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98371f66-59b4-4f69-8d22-c9849c28d6b9 None admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:41:40.722490 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98371f66-59b4-4f69-8d22-c9849c28d6b9 None admin] PATH_INFO: `/v3/projects/service` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:41:40.729470 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-98371f66-59b4-4f69-8d22-c9849c28d6b9 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:41:40.731032 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-98371f66-59b4-4f69-8d22-c9849c28d6b9 None admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:41:40.734244 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=100 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:41:40.751181 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application [None req-98371f66-59b4-4f69-8d22-c9849c28d6b9 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. May 23 12:41:40.751181 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application Traceback (most recent call last): May 23 12:41:40.751181 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request May 23 12:41:40.751181 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application rv = self.dispatch_request() May 23 12:41:40.751181 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request May 23 12:41:40.751181 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) May 23 12:41:40.751181 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper May 23 12:41:40.751181 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) May 23 12:41:40.751181 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view May 23 12:41:40.751181 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) May 23 12:41:40.751181 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request May 23 12:41:40.751181 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) May 23 12:41:40.751181 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get May 23 12:41:40.751181 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self._get_project(project_id) May 23 12:41:40.751181 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project May 23 12:41:40.752106 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) May 23 12:41:40.752106 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped May 23 12:41:40.752106 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) May 23 12:41:40.752106 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun May 23 12:41:40.752106 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) May 23 12:41:40.752106 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func May 23 12:41:40.752106 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self.get_or_create( May 23 12:41:40.752106 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create May 23 12:41:40.752106 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application with Lock( May 23 12:41:40.752106 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ May 23 12:41:40.752106 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self._enter() May 23 12:41:40.752106 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter May 23 12:41:40.752106 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) May 23 12:41:40.752106 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create May 23 12:41:40.752106 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self.creator() May 23 12:41:40.752106 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value May 23 12:41:40.752106 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application created_value = creator( May 23 12:41:40.753020 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project May 23 12:41:40.753020 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) May 23 12:41:40.753020 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project May 23 12:41:40.753020 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() May 23 12:41:40.753020 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project May 23 12:41:40.753020 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) May 23 12:41:40.753020 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. May 23 12:41:40.753020 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application May 23 12:41:40.753463 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 353/696] 10.222.0.30 () {56 vars in 1111 bytes} [Mon May 23 12:41:40 2022] GET /identity/v3/projects/service => generated 88 bytes in 327 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) May 23 12:41:40.754874 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71e3681e-4e37-4916-87e0-2edc61cf2ce5 None None] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:41:40.755171 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71e3681e-4e37-4916-87e0-2edc61cf2ce5 None None] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:41:40.755461 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71e3681e-4e37-4916-87e0-2edc61cf2ce5 None None] PATH_INFO: `/` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:41:40.761445 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 354/697] 10.222.0.30 () {54 vars in 926 bytes} [Mon May 23 12:41:40 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) May 23 12:41:40.763352 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71e3681e-4e37-4916-87e0-2edc61cf2ce5 None None] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:41:40.805087 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb38ebbf-aa39-4394-bd7b-9c05bfcb60bb None admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:41:40.843841 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b7bf181-096c-45cf-b72c-43de8a7a3bc0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:41:40.844492 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b7bf181-096c-45cf-b72c-43de8a7a3bc0 None admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:41:40.844777 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b7bf181-096c-45cf-b72c-43de8a7a3bc0 None admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:41:40.845055 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b7bf181-096c-45cf-b72c-43de8a7a3bc0 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:41:40.900112 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71e3681e-4e37-4916-87e0-2edc61cf2ce5 None None] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:41:40.906455 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=100 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:41:40.921007 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b7bf181-096c-45cf-b72c-43de8a7a3bc0 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:41:40.922527 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b7bf181-096c-45cf-b72c-43de8a7a3bc0 None admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:41:40.947862 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5ae0960-7b71-4a2c-b00d-572912552f5a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:41:40.948912 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5ae0960-7b71-4a2c-b00d-572912552f5a None admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:41:40.949660 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5ae0960-7b71-4a2c-b00d-572912552f5a None admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:41:40.950428 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5ae0960-7b71-4a2c-b00d-572912552f5a None admin] PATH_INFO: `/v3/projects` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:41:40.951548 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5ae0960-7b71-4a2c-b00d-572912552f5a None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=92056) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} May 23 12:41:40.952301 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5ae0960-7b71-4a2c-b00d-572912552f5a None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:41:40.954220 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5ae0960-7b71-4a2c-b00d-572912552f5a None admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:41:40.970331 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 355/698] 10.222.0.30 () {56 vars in 1148 bytes} [Mon May 23 12:41:40 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 208 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) May 23 12:41:40.971804 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b783ee22-30e1-4209-a1b6-647b93a9e2c2 None None] REQUEST_METHOD: `POST` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:41:40.972244 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b783ee22-30e1-4209-a1b6-647b93a9e2c2 None None] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:41:40.972701 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b783ee22-30e1-4209-a1b6-647b93a9e2c2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:41:40.994098 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.auth.core [None req-b783ee22-30e1-4209-a1b6-647b93a9e2c2 None None] MFA Rules not processed for user `0f7119f8fa5c4445977671140ded91a4`. Rule list: `[]` (Enabled: `True`). {{(pid=92056) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 12:41:40.996069 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.fernet_utils [None req-b783ee22-30e1-4209-a1b6-647b93a9e2c2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92056) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 12:41:41.041189 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 344/699] 10.222.0.30 () {58 vars in 1282 bytes} [Mon May 23 12:41:40 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 371 msecs (HTTP/1.1 200) 6 headers in 358 bytes (2 switches on core 0) May 23 12:41:41.041858 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b7bf181-096c-45cf-b72c-43de8a7a3bc0 None admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:41:41.054196 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=100 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:41:41.093961 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 356/700] 10.222.0.30 () {58 vars in 1023 bytes} [Mon May 23 12:41:40 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 117 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) May 23 12:41:41.097250 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b783ee22-30e1-4209-a1b6-647b93a9e2c2 None None] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:41:41.174623 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=100 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:41:41.190501 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b7bf181-096c-45cf-b72c-43de8a7a3bc0 None admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:41:41.219649 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b783ee22-30e1-4209-a1b6-647b93a9e2c2 None None] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:41:41.239175 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fee3b58b-aafd-418e-9a4a-859916a9b4e4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:41:41.239847 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fee3b58b-aafd-418e-9a4a-859916a9b4e4 None admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:41:41.240159 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fee3b58b-aafd-418e-9a4a-859916a9b4e4 None admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:41:41.240443 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fee3b58b-aafd-418e-9a4a-859916a9b4e4 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:41:41.241275 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fee3b58b-aafd-418e-9a4a-859916a9b4e4 None admin] RBAC: Adding query filter params (role.id=41ab28c1ae3d4941bd09fb5d35f1dc54, scope.project.id=9c389ae65aaa4542bb5a5c4fd0bacf5b, user.id=138d91edb99f450e846765f9eef00d62) {{(pid=92057) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} May 23 12:41:41.241590 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fee3b58b-aafd-418e-9a4a-859916a9b4e4 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:41:41.243918 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fee3b58b-aafd-418e-9a4a-859916a9b4e4 None admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:41:41.253949 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 345/701] 10.222.0.30 () {56 vars in 1374 bytes} [Mon May 23 12:41:41 2022] GET /identity/v3/role_assignments?user.id=138d91edb99f450e846765f9eef00d62&scope.project.id=9c389ae65aaa4542bb5a5c4fd0bacf5b&role.id=41ab28c1ae3d4941bd09fb5d35f1dc54 => generated 608 bytes in 211 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) May 23 12:41:41.254102 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fee3b58b-aafd-418e-9a4a-859916a9b4e4 None admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:41:41.278702 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b87d209-3b5d-4fd3-bb76-39014d5373c7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:41:41.279519 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b87d209-3b5d-4fd3-bb76-39014d5373c7 None admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:41:41.279801 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b87d209-3b5d-4fd3-bb76-39014d5373c7 None admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:41:41.280076 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b87d209-3b5d-4fd3-bb76-39014d5373c7 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:41:41.304252 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b87d209-3b5d-4fd3-bb76-39014d5373c7 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:41:41.305747 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b87d209-3b5d-4fd3-bb76-39014d5373c7 None admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:41:41.343169 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application [None req-3b87d209-3b5d-4fd3-bb76-39014d5373c7 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. May 23 12:41:41.343169 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application Traceback (most recent call last): May 23 12:41:41.343169 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request May 23 12:41:41.343169 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application rv = self.dispatch_request() May 23 12:41:41.343169 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request May 23 12:41:41.343169 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) May 23 12:41:41.343169 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper May 23 12:41:41.343169 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) May 23 12:41:41.343169 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view May 23 12:41:41.343169 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) May 23 12:41:41.343169 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request May 23 12:41:41.343169 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) May 23 12:41:41.343169 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get May 23 12:41:41.343169 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self._get_domain(domain_id) May 23 12:41:41.343169 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain May 23 12:41:41.344159 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) May 23 12:41:41.344159 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped May 23 12:41:41.344159 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) May 23 12:41:41.344159 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun May 23 12:41:41.344159 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) May 23 12:41:41.344159 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func May 23 12:41:41.344159 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self.get_or_create( May 23 12:41:41.344159 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create May 23 12:41:41.344159 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application with Lock( May 23 12:41:41.344159 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ May 23 12:41:41.344159 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self._enter() May 23 12:41:41.344159 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter May 23 12:41:41.344159 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) May 23 12:41:41.344159 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create May 23 12:41:41.344159 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self.creator() May 23 12:41:41.344159 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value May 23 12:41:41.344159 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application created_value = creator( May 23 12:41:41.345078 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain May 23 12:41:41.345078 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) May 23 12:41:41.345078 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. May 23 12:41:41.345078 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application May 23 12:41:41.354392 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 357/702] 10.222.0.30 () {56 vars in 1108 bytes} [Mon May 23 12:41:41 2022] GET /identity/v3/domains/Default => generated 87 bytes in 257 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) May 23 12:41:41.359860 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b87d209-3b5d-4fd3-bb76-39014d5373c7 None admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:41:41.379293 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fee3b58b-aafd-418e-9a4a-859916a9b4e4 None admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:41:41.428336 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bdefeef1-08b0-4f7b-a1c8-c13f19e562a8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:41:41.429194 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdefeef1-08b0-4f7b-a1c8-c13f19e562a8 None admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:41:41.429397 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdefeef1-08b0-4f7b-a1c8-c13f19e562a8 None admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:41:41.433909 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdefeef1-08b0-4f7b-a1c8-c13f19e562a8 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:41:41.434304 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=100 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:41:41.462074 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bdefeef1-08b0-4f7b-a1c8-c13f19e562a8 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:41:41.463590 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bdefeef1-08b0-4f7b-a1c8-c13f19e562a8 None admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:41:41.509590 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application [None req-bdefeef1-08b0-4f7b-a1c8-c13f19e562a8 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. May 23 12:41:41.509590 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application Traceback (most recent call last): May 23 12:41:41.509590 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request May 23 12:41:41.509590 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application rv = self.dispatch_request() May 23 12:41:41.509590 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request May 23 12:41:41.509590 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) May 23 12:41:41.509590 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper May 23 12:41:41.509590 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) May 23 12:41:41.509590 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view May 23 12:41:41.509590 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) May 23 12:41:41.509590 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request May 23 12:41:41.509590 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) May 23 12:41:41.509590 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get May 23 12:41:41.509590 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self._get_domain(domain_id) May 23 12:41:41.509590 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain May 23 12:41:41.510682 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) May 23 12:41:41.510682 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped May 23 12:41:41.510682 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) May 23 12:41:41.510682 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun May 23 12:41:41.510682 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) May 23 12:41:41.510682 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func May 23 12:41:41.510682 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self.get_or_create( May 23 12:41:41.510682 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create May 23 12:41:41.510682 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application with Lock( May 23 12:41:41.510682 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ May 23 12:41:41.510682 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self._enter() May 23 12:41:41.510682 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter May 23 12:41:41.510682 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) May 23 12:41:41.510682 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create May 23 12:41:41.510682 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self.creator() May 23 12:41:41.510682 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value May 23 12:41:41.510682 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application created_value = creator( May 23 12:41:41.511669 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain May 23 12:41:41.511669 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) May 23 12:41:41.511669 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. May 23 12:41:41.511669 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application May 23 12:41:41.516542 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 346/703] 10.222.0.30 () {56 vars in 1108 bytes} [Mon May 23 12:41:41 2022] GET /identity/v3/domains/Default => generated 87 bytes in 257 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) May 23 12:41:41.517802 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bdefeef1-08b0-4f7b-a1c8-c13f19e562a8 None admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:41:41.523575 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=100 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:41:41.582723 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b87d209-3b5d-4fd3-bb76-39014d5373c7 None admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:41:41.624320 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ba5cc0f-1eb7-4ba2-a571-19310e0715bc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:41:41.624993 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ba5cc0f-1eb7-4ba2-a571-19310e0715bc None admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:41:41.625280 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ba5cc0f-1eb7-4ba2-a571-19310e0715bc None admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:41:41.625560 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ba5cc0f-1eb7-4ba2-a571-19310e0715bc None admin] PATH_INFO: `/v3/domains` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:41:41.626532 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ba5cc0f-1eb7-4ba2-a571-19310e0715bc None admin] RBAC: Adding query filter params (name=Default) {{(pid=92056) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} May 23 12:41:41.626697 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ba5cc0f-1eb7-4ba2-a571-19310e0715bc None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:41:41.628090 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ba5cc0f-1eb7-4ba2-a571-19310e0715bc None admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:41:41.648233 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 358/704] 10.222.0.30 () {56 vars in 1109 bytes} [Mon May 23 12:41:41 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 289 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) May 23 12:41:41.649175 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ba5cc0f-1eb7-4ba2-a571-19310e0715bc None admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:41:41.656510 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bdefeef1-08b0-4f7b-a1c8-c13f19e562a8 None admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:41:41.705871 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31f58294-5f54-4727-b09d-8d67829c19b7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:41:41.707348 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31f58294-5f54-4727-b09d-8d67829c19b7 None admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:41:41.707790 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31f58294-5f54-4727-b09d-8d67829c19b7 None admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:41:41.708218 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31f58294-5f54-4727-b09d-8d67829c19b7 None admin] PATH_INFO: `/v3/domains` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:41:41.709142 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31f58294-5f54-4727-b09d-8d67829c19b7 None admin] RBAC: Adding query filter params (name=Default) {{(pid=92057) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} May 23 12:41:41.709659 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31f58294-5f54-4727-b09d-8d67829c19b7 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:41:41.711819 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31f58294-5f54-4727-b09d-8d67829c19b7 None admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:41:41.733443 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 347/705] 10.222.0.30 () {56 vars in 1109 bytes} [Mon May 23 12:41:41 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 212 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) May 23 12:41:41.736884 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31f58294-5f54-4727-b09d-8d67829c19b7 None admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:41:41.761776 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=100 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:41:41.779852 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ba5cc0f-1eb7-4ba2-a571-19310e0715bc None admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:41:41.821153 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad49827f-1fcb-4544-a7d6-9ad53e932ba0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:41:41.822205 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad49827f-1fcb-4544-a7d6-9ad53e932ba0 None admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:41:41.822830 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad49827f-1fcb-4544-a7d6-9ad53e932ba0 None admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:41:41.823108 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad49827f-1fcb-4544-a7d6-9ad53e932ba0 None admin] PATH_INFO: `/v3/projects/service` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:41:41.825457 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=100 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:41:41.832474 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad49827f-1fcb-4544-a7d6-9ad53e932ba0 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:41:41.835225 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad49827f-1fcb-4544-a7d6-9ad53e932ba0 None admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:41:41.850601 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application [None req-ad49827f-1fcb-4544-a7d6-9ad53e932ba0 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. May 23 12:41:41.850601 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application Traceback (most recent call last): May 23 12:41:41.850601 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request May 23 12:41:41.850601 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application rv = self.dispatch_request() May 23 12:41:41.850601 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request May 23 12:41:41.850601 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) May 23 12:41:41.850601 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper May 23 12:41:41.850601 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) May 23 12:41:41.850601 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view May 23 12:41:41.850601 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) May 23 12:41:41.850601 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request May 23 12:41:41.850601 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) May 23 12:41:41.850601 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get May 23 12:41:41.850601 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self._get_project(project_id) May 23 12:41:41.850601 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project May 23 12:41:41.857351 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) May 23 12:41:41.857351 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped May 23 12:41:41.857351 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) May 23 12:41:41.857351 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun May 23 12:41:41.857351 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) May 23 12:41:41.857351 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func May 23 12:41:41.857351 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self.get_or_create( May 23 12:41:41.857351 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create May 23 12:41:41.857351 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application with Lock( May 23 12:41:41.857351 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ May 23 12:41:41.857351 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self._enter() May 23 12:41:41.857351 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter May 23 12:41:41.857351 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) May 23 12:41:41.857351 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create May 23 12:41:41.857351 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self.creator() May 23 12:41:41.857351 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value May 23 12:41:41.857351 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application created_value = creator( May 23 12:41:41.858302 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project May 23 12:41:41.858302 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) May 23 12:41:41.858302 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project May 23 12:41:41.858302 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() May 23 12:41:41.858302 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project May 23 12:41:41.858302 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) May 23 12:41:41.858302 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. May 23 12:41:41.858302 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application May 23 12:41:41.858302 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 359/706] 10.222.0.30 () {56 vars in 1111 bytes} [Mon May 23 12:41:41 2022] GET /identity/v3/projects/service => generated 88 bytes in 203 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) May 23 12:41:41.858302 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1cde1859-e9d9-4f3e-a32e-2a63225d061e None None] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:41:41.858302 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1cde1859-e9d9-4f3e-a32e-2a63225d061e None None] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:41:41.858932 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1cde1859-e9d9-4f3e-a32e-2a63225d061e None None] PATH_INFO: `/` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:41:41.858932 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 360/707] 10.222.0.30 () {54 vars in 926 bytes} [Mon May 23 12:41:41 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) May 23 12:41:41.860200 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1cde1859-e9d9-4f3e-a32e-2a63225d061e None None] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:41:41.875551 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31f58294-5f54-4727-b09d-8d67829c19b7 None admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:41:41.920075 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4877bdc-5c08-4ec3-aba1-5f40e51c622b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:41:41.921551 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4877bdc-5c08-4ec3-aba1-5f40e51c622b None admin] REQUEST_METHOD: `POST` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:41:41.922325 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4877bdc-5c08-4ec3-aba1-5f40e51c622b None admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:41:41.922801 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4877bdc-5c08-4ec3-aba1-5f40e51c622b None admin] PATH_INFO: `/v3/users` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:41:41.924202 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4877bdc-5c08-4ec3-aba1-5f40e51c622b None admin] RBAC: Authorizing `identity:create_user()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:41:41.926334 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4877bdc-5c08-4ec3-aba1-5f40e51c622b None admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:41:41.971171 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1cde1859-e9d9-4f3e-a32e-2a63225d061e None None] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:41:42.005806 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58badfbf-dd6d-4684-98f9-079d97b18e88 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:41:42.006507 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58badfbf-dd6d-4684-98f9-079d97b18e88 None admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:41:42.006783 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58badfbf-dd6d-4684-98f9-079d97b18e88 None admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:41:42.007056 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58badfbf-dd6d-4684-98f9-079d97b18e88 None admin] PATH_INFO: `/v3/projects` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:41:42.007866 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58badfbf-dd6d-4684-98f9-079d97b18e88 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=92056) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} May 23 12:41:42.008171 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58badfbf-dd6d-4684-98f9-079d97b18e88 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:41:42.009628 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58badfbf-dd6d-4684-98f9-079d97b18e88 None admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:41:42.021135 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 361/708] 10.222.0.30 () {56 vars in 1148 bytes} [Mon May 23 12:41:41 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 161 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) May 23 12:41:42.022474 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82538863-d05d-464d-869b-935c35d5c679 None None] REQUEST_METHOD: `POST` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:41:42.022762 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82538863-d05d-464d-869b-935c35d5c679 None None] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:41:42.023040 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82538863-d05d-464d-869b-935c35d5c679 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:41:42.023682 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 348/709] 10.222.0.30 () {60 vars in 1132 bytes} [Mon May 23 12:41:41 2022] POST /identity/v3/users => generated 254 bytes in 289 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) May 23 12:41:42.025137 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3714217d-b757-4038-a359-b8d825e5f618 None None] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:41:42.025427 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3714217d-b757-4038-a359-b8d825e5f618 None None] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:41:42.025703 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3714217d-b757-4038-a359-b8d825e5f618 None None] PATH_INFO: `/` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:41:42.027389 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 349/710] 10.222.0.30 () {54 vars in 926 bytes} [Mon May 23 12:41:42 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) May 23 12:41:42.034112 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17d6304b-1f96-423e-90aa-6f7b46843ff5 None None] REQUEST_METHOD: `POST` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:41:42.034398 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17d6304b-1f96-423e-90aa-6f7b46843ff5 None None] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:41:42.034676 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17d6304b-1f96-423e-90aa-6f7b46843ff5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:41:42.046445 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.auth.core [None req-82538863-d05d-464d-869b-935c35d5c679 None None] MFA Rules not processed for user `0f7119f8fa5c4445977671140ded91a4`. Rule list: `[]` (Enabled: `True`). {{(pid=92056) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 12:41:42.048030 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.fernet_utils [None req-82538863-d05d-464d-869b-935c35d5c679 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92056) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 12:41:42.050752 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=100 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:41:42.059684 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.auth.core [None req-17d6304b-1f96-423e-90aa-6f7b46843ff5 None None] MFA Rules not processed for user `0f7119f8fa5c4445977671140ded91a4`. Rule list: `[]` (Enabled: `True`). {{(pid=92057) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 12:41:42.061669 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.fernet_utils [None req-17d6304b-1f96-423e-90aa-6f7b46843ff5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92057) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 12:41:42.142230 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 362/711] 10.222.0.30 () {58 vars in 1023 bytes} [Mon May 23 12:41:42 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 116 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) May 23 12:41:42.142230 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82538863-d05d-464d-869b-935c35d5c679 None None] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:41:42.174898 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 350/712] 10.222.0.30 () {58 vars in 1023 bytes} [Mon May 23 12:41:42 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 143 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) May 23 12:41:42.181981 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17d6304b-1f96-423e-90aa-6f7b46843ff5 None None] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:41:42.239328 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:41:42.286280 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82538863-d05d-464d-869b-935c35d5c679 None None] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:41:42.308481 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17d6304b-1f96-423e-90aa-6f7b46843ff5 None None] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:41:42.334064 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56dd4228-e7f5-474c-86ef-f3e105639c08 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:41:42.334967 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56dd4228-e7f5-474c-86ef-f3e105639c08 None admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:41:42.335510 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56dd4228-e7f5-474c-86ef-f3e105639c08 None admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:41:42.335786 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56dd4228-e7f5-474c-86ef-f3e105639c08 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:41:42.336616 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56dd4228-e7f5-474c-86ef-f3e105639c08 None admin] RBAC: Adding query filter params (role.id=bc3c3c9d654940cbaa5177a6be23d1bc, scope.project.id=9c389ae65aaa4542bb5a5c4fd0bacf5b, user.id=cbe9931d94d44cc7ace0e83a7e5015fc) {{(pid=92056) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} May 23 12:41:42.336932 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56dd4228-e7f5-474c-86ef-f3e105639c08 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:41:42.338422 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56dd4228-e7f5-474c-86ef-f3e105639c08 None admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:41:42.344803 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 363/713] 10.222.0.30 () {56 vars in 1374 bytes} [Mon May 23 12:41:42 2022] GET /identity/v3/role_assignments?user.id=cbe9931d94d44cc7ace0e83a7e5015fc&scope.project.id=9c389ae65aaa4542bb5a5c4fd0bacf5b&role.id=bc3c3c9d654940cbaa5177a6be23d1bc => generated 259 bytes in 206 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) May 23 12:41:42.345684 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56dd4228-e7f5-474c-86ef-f3e105639c08 None admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:41:42.348389 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=100 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:41:42.353357 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b23f5b7-e417-4870-ae8a-3fe60c21c00a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:41:42.354450 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b23f5b7-e417-4870-ae8a-3fe60c21c00a None admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:41:42.354877 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b23f5b7-e417-4870-ae8a-3fe60c21c00a None admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:41:42.355324 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b23f5b7-e417-4870-ae8a-3fe60c21c00a None admin] PATH_INFO: `/v3/services/network` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:41:42.356157 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b23f5b7-e417-4870-ae8a-3fe60c21c00a None admin] RBAC: Authorizing `identity:get_service(service_id=network)` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:41:42.357758 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b23f5b7-e417-4870-ae8a-3fe60c21c00a None admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:41:42.374239 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application [None req-6b23f5b7-e417-4870-ae8a-3fe60c21c00a None admin] Could not find service: network.: keystone.exception.ServiceNotFound: Could not find service: network. May 23 12:41:42.374239 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application Traceback (most recent call last): May 23 12:41:42.374239 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service May 23 12:41:42.374239 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) May 23 12:41:42.374239 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service May 23 12:41:42.374239 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() May 23 12:41:42.374239 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service May 23 12:41:42.374239 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) May 23 12:41:42.374239 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: network. May 23 12:41:42.374239 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application May 23 12:41:42.374239 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: May 23 12:41:42.374239 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application May 23 12:41:42.374239 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application Traceback (most recent call last): May 23 12:41:42.374239 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request May 23 12:41:42.374239 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application rv = self.dispatch_request() May 23 12:41:42.374239 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request May 23 12:41:42.374239 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) May 23 12:41:42.381684 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper May 23 12:41:42.381684 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) May 23 12:41:42.381684 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view May 23 12:41:42.381684 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) May 23 12:41:42.381684 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request May 23 12:41:42.381684 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) May 23 12:41:42.381684 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get May 23 12:41:42.381684 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self._get_service(service_id) May 23 12:41:42.381684 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service May 23 12:41:42.381684 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) May 23 12:41:42.381684 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped May 23 12:41:42.381684 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) May 23 12:41:42.381684 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun May 23 12:41:42.381684 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) May 23 12:41:42.381684 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func May 23 12:41:42.381684 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self.get_or_create( May 23 12:41:42.382956 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create May 23 12:41:42.382956 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application with Lock( May 23 12:41:42.382956 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ May 23 12:41:42.382956 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self._enter() May 23 12:41:42.382956 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter May 23 12:41:42.382956 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) May 23 12:41:42.382956 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create May 23 12:41:42.382956 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self.creator() May 23 12:41:42.382956 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value May 23 12:41:42.382956 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application created_value = creator( May 23 12:41:42.382956 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service May 23 12:41:42.382956 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) May 23 12:41:42.382956 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: network. May 23 12:41:42.382956 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application May 23 12:41:42.382956 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 351/714] 10.222.0.30 () {56 vars in 1111 bytes} [Mon May 23 12:41:42 2022] GET /identity/v3/services/network => generated 88 bytes in 201 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) May 23 12:41:42.386430 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b23f5b7-e417-4870-ae8a-3fe60c21c00a None admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:41:42.471646 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56dd4228-e7f5-474c-86ef-f3e105639c08 None admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:41:42.510288 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e7b0a2c-ee91-4acd-a2eb-bfaaf85fc35a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:41:42.511018 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e7b0a2c-ee91-4acd-a2eb-bfaaf85fc35a None admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:41:42.511362 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e7b0a2c-ee91-4acd-a2eb-bfaaf85fc35a None admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:41:42.511633 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e7b0a2c-ee91-4acd-a2eb-bfaaf85fc35a None admin] PATH_INFO: `/v3/services/block-storage` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:41:42.512338 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e7b0a2c-ee91-4acd-a2eb-bfaaf85fc35a None admin] RBAC: Authorizing `identity:get_service(service_id=block-storage)` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:41:42.513716 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e7b0a2c-ee91-4acd-a2eb-bfaaf85fc35a None admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:41:42.547351 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application [None req-0e7b0a2c-ee91-4acd-a2eb-bfaaf85fc35a None admin] Could not find service: block-storage.: keystone.exception.ServiceNotFound: Could not find service: block-storage. May 23 12:41:42.547351 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application Traceback (most recent call last): May 23 12:41:42.547351 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service May 23 12:41:42.547351 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) May 23 12:41:42.547351 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service May 23 12:41:42.547351 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() May 23 12:41:42.547351 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service May 23 12:41:42.547351 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) May 23 12:41:42.547351 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: block-storage. May 23 12:41:42.547351 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application May 23 12:41:42.547351 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: May 23 12:41:42.547351 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application May 23 12:41:42.547351 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application Traceback (most recent call last): May 23 12:41:42.547351 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request May 23 12:41:42.547351 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application rv = self.dispatch_request() May 23 12:41:42.547351 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request May 23 12:41:42.548389 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) May 23 12:41:42.548389 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper May 23 12:41:42.548389 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) May 23 12:41:42.548389 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view May 23 12:41:42.548389 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) May 23 12:41:42.548389 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request May 23 12:41:42.548389 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) May 23 12:41:42.548389 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get May 23 12:41:42.548389 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self._get_service(service_id) May 23 12:41:42.548389 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service May 23 12:41:42.548389 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) May 23 12:41:42.548389 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped May 23 12:41:42.548389 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) May 23 12:41:42.548389 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun May 23 12:41:42.548389 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) May 23 12:41:42.548389 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func May 23 12:41:42.549254 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self.get_or_create( May 23 12:41:42.549254 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create May 23 12:41:42.549254 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application with Lock( May 23 12:41:42.549254 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ May 23 12:41:42.549254 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self._enter() May 23 12:41:42.549254 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter May 23 12:41:42.549254 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) May 23 12:41:42.549254 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create May 23 12:41:42.549254 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self.creator() May 23 12:41:42.549254 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value May 23 12:41:42.549254 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application created_value = creator( May 23 12:41:42.549254 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service May 23 12:41:42.549254 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) May 23 12:41:42.549254 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: block-storage. May 23 12:41:42.549254 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application May 23 12:41:42.549254 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 364/715] 10.222.0.30 () {56 vars in 1129 bytes} [Mon May 23 12:41:42 2022] GET /identity/v3/services/block-storage => generated 94 bytes in 203 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) May 23 12:41:42.556714 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e7b0a2c-ee91-4acd-a2eb-bfaaf85fc35a None admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:41:42.592136 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b23f5b7-e417-4870-ae8a-3fe60c21c00a None admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:41:42.614538 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=100 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:41:42.638298 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d94e7ed4-b4e0-480f-b48d-f1efcd5c6695 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:41:42.639354 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d94e7ed4-b4e0-480f-b48d-f1efcd5c6695 None admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:41:42.639836 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d94e7ed4-b4e0-480f-b48d-f1efcd5c6695 None admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:41:42.640307 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d94e7ed4-b4e0-480f-b48d-f1efcd5c6695 None admin] PATH_INFO: `/v3/services` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:41:42.641224 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d94e7ed4-b4e0-480f-b48d-f1efcd5c6695 None admin] RBAC: Adding query filter params (name=network) {{(pid=92057) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} May 23 12:41:42.641734 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d94e7ed4-b4e0-480f-b48d-f1efcd5c6695 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:41:42.643777 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d94e7ed4-b4e0-480f-b48d-f1efcd5c6695 None admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:41:42.646238 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=100 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:41:42.651740 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 352/716] 10.222.0.30 () {56 vars in 1112 bytes} [Mon May 23 12:41:42 2022] GET /identity/v3/services?name=network => generated 124 bytes in 266 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) May 23 12:41:42.657436 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d94e7ed4-b4e0-480f-b48d-f1efcd5c6695 None admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:41:42.774459 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e7b0a2c-ee91-4acd-a2eb-bfaaf85fc35a None admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:41:42.778282 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d94e7ed4-b4e0-480f-b48d-f1efcd5c6695 None admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:41:42.820664 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c90fa6e7-1d95-41f2-8016-e8d30289f7d9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:41:42.821715 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c90fa6e7-1d95-41f2-8016-e8d30289f7d9 None admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:41:42.822232 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c90fa6e7-1d95-41f2-8016-e8d30289f7d9 None admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:41:42.822998 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c90fa6e7-1d95-41f2-8016-e8d30289f7d9 None admin] PATH_INFO: `/v3/services` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:41:42.824031 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c90fa6e7-1d95-41f2-8016-e8d30289f7d9 None admin] RBAC: Adding query filter params (type=network) {{(pid=92057) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} May 23 12:41:42.824536 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c90fa6e7-1d95-41f2-8016-e8d30289f7d9 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:41:42.826162 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c90fa6e7-1d95-41f2-8016-e8d30289f7d9 None admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:41:42.834856 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 353/717] 10.222.0.30 () {56 vars in 1112 bytes} [Mon May 23 12:41:42 2022] GET /identity/v3/services?type=network => generated 124 bytes in 178 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) May 23 12:41:42.849589 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db916ddb-f953-4db4-a8e8-f45e9dad39ab None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:41:42.850412 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db916ddb-f953-4db4-a8e8-f45e9dad39ab None admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:41:42.850729 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db916ddb-f953-4db4-a8e8-f45e9dad39ab None admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:41:42.851004 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db916ddb-f953-4db4-a8e8-f45e9dad39ab None admin] PATH_INFO: `/v3/services` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:41:42.851735 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db916ddb-f953-4db4-a8e8-f45e9dad39ab None admin] RBAC: Adding query filter params (name=block-storage) {{(pid=92056) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} May 23 12:41:42.852039 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db916ddb-f953-4db4-a8e8-f45e9dad39ab None admin] RBAC: Authorizing `identity:list_services()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:41:42.853475 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db916ddb-f953-4db4-a8e8-f45e9dad39ab None admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:41:42.871800 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 365/718] 10.222.0.30 () {56 vars in 1124 bytes} [Mon May 23 12:41:42 2022] GET /identity/v3/services?name=block-storage => generated 130 bytes in 315 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) May 23 12:41:42.877961 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c90fa6e7-1d95-41f2-8016-e8d30289f7d9 None admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:41:42.936089 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=100 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:41:42.999942 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c90fa6e7-1d95-41f2-8016-e8d30289f7d9 None admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:41:43.038649 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c6f62ab-7ffc-4f45-915c-95cb99f588f2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:41:43.039659 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c6f62ab-7ffc-4f45-915c-95cb99f588f2 None admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:41:43.040141 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c6f62ab-7ffc-4f45-915c-95cb99f588f2 None admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:41:43.040642 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c6f62ab-7ffc-4f45-915c-95cb99f588f2 None admin] PATH_INFO: `/v3/services` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:41:43.041634 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c6f62ab-7ffc-4f45-915c-95cb99f588f2 None admin] RBAC: Adding query filter params (type=block-storage) {{(pid=92057) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} May 23 12:41:43.042196 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c6f62ab-7ffc-4f45-915c-95cb99f588f2 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:41:43.043839 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c6f62ab-7ffc-4f45-915c-95cb99f588f2 None admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:41:43.050046 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 354/719] 10.222.0.30 () {56 vars in 1124 bytes} [Mon May 23 12:41:42 2022] GET /identity/v3/services?type=block-storage => generated 130 bytes in 172 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) May 23 12:41:44.308794 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00fb8932-0185-47a9-b806-8b0129d103f4 None None] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:41:44.309345 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00fb8932-0185-47a9-b806-8b0129d103f4 None None] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:41:44.309345 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00fb8932-0185-47a9-b806-8b0129d103f4 None None] PATH_INFO: `/` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:41:44.311066 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 366/720] 10.222.0.30 () {54 vars in 926 bytes} [Mon May 23 12:41:44 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) May 23 12:41:44.316528 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-373e77b0-f955-412c-963a-038a62fae4a4 None None] REQUEST_METHOD: `POST` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:41:44.317329 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-373e77b0-f955-412c-963a-038a62fae4a4 None None] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:41:44.317804 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-373e77b0-f955-412c-963a-038a62fae4a4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:41:44.336559 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.auth.core [None req-373e77b0-f955-412c-963a-038a62fae4a4 None None] MFA Rules not processed for user `0f7119f8fa5c4445977671140ded91a4`. Rule list: `[]` (Enabled: `True`). {{(pid=92057) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 12:41:44.338092 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.fernet_utils [None req-373e77b0-f955-412c-963a-038a62fae4a4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92057) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 12:41:44.422909 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 355/721] 10.222.0.30 () {58 vars in 1023 bytes} [Mon May 23 12:41:44 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 107 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) May 23 12:41:44.464482 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00fb8932-0185-47a9-b806-8b0129d103f4 None None] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:41:44.608879 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00fb8932-0185-47a9-b806-8b0129d103f4 None None] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:41:44.650673 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=100 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:41:44.660413 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6860fc90-cc9e-4e28-81ba-ec73a4eb2c6a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:41:44.661217 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6860fc90-cc9e-4e28-81ba-ec73a4eb2c6a None admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:41:44.661503 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6860fc90-cc9e-4e28-81ba-ec73a4eb2c6a None admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:41:44.661824 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6860fc90-cc9e-4e28-81ba-ec73a4eb2c6a None admin] PATH_INFO: `/v3/roles/admin` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:41:44.675729 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6860fc90-cc9e-4e28-81ba-ec73a4eb2c6a None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} May 23 12:41:44.689351 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application [None req-6860fc90-cc9e-4e28-81ba-ec73a4eb2c6a None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. May 23 12:41:44.689351 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application Traceback (most recent call last): May 23 12:41:44.689351 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request May 23 12:41:44.689351 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application rv = self.dispatch_request() May 23 12:41:44.689351 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request May 23 12:41:44.689351 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) May 23 12:41:44.689351 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper May 23 12:41:44.689351 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) May 23 12:41:44.689351 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view May 23 12:41:44.689351 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) May 23 12:41:44.689351 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request May 23 12:41:44.689351 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) May 23 12:41:44.689351 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get May 23 12:41:44.689351 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self._get_role(role_id) May 23 12:41:44.689351 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role May 23 12:41:44.689351 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') May 23 12:41:44.690405 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call May 23 12:41:44.690405 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( May 23 12:41:44.690405 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data May 23 12:41:44.690405 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) May 23 12:41:44.690405 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped May 23 12:41:44.690405 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) May 23 12:41:44.690405 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun May 23 12:41:44.690405 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) May 23 12:41:44.690405 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func May 23 12:41:44.690405 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self.get_or_create( May 23 12:41:44.690405 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create May 23 12:41:44.690405 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application with Lock( May 23 12:41:44.690405 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ May 23 12:41:44.690405 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self._enter() May 23 12:41:44.690405 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter May 23 12:41:44.690405 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) May 23 12:41:44.691285 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create May 23 12:41:44.691285 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self.creator() May 23 12:41:44.691285 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value May 23 12:41:44.691285 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application created_value = creator( May 23 12:41:44.691285 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role May 23 12:41:44.691285 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) May 23 12:41:44.691285 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role May 23 12:41:44.691285 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() May 23 12:41:44.691285 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role May 23 12:41:44.691285 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) May 23 12:41:44.691285 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. May 23 12:41:44.691285 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application May 23 12:41:44.691285 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 367/722] 10.222.0.30 () {56 vars in 1096 bytes} [Mon May 23 12:41:44 2022] GET /identity/v3/roles/admin => generated 83 bytes in 222 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) May 23 12:41:44.694463 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-373e77b0-f955-412c-963a-038a62fae4a4 None None] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:41:44.766253 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=100 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:41:44.833254 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-373e77b0-f955-412c-963a-038a62fae4a4 None None] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:41:44.838780 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bda69983-6e77-4ce5-a2d9-ad128514554a None None] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:41:44.839072 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bda69983-6e77-4ce5-a2d9-ad128514554a None None] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:41:44.839359 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bda69983-6e77-4ce5-a2d9-ad128514554a None None] PATH_INFO: `/` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:41:44.841792 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 368/723] 10.222.0.30 () {54 vars in 926 bytes} [Mon May 23 12:41:44 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) May 23 12:41:44.851299 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ca22979-a2df-4c92-9c45-4a485d78f650 None None] REQUEST_METHOD: `POST` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:41:44.851588 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ca22979-a2df-4c92-9c45-4a485d78f650 None None] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:41:44.851868 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ca22979-a2df-4c92-9c45-4a485d78f650 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:41:44.872883 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9529f812-f086-4168-b218-10ec80fd48d6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:41:44.873563 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9529f812-f086-4168-b218-10ec80fd48d6 None admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:41:44.873850 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9529f812-f086-4168-b218-10ec80fd48d6 None admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:41:44.874225 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9529f812-f086-4168-b218-10ec80fd48d6 None admin] PATH_INFO: `/v3/roles` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:41:44.874988 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9529f812-f086-4168-b218-10ec80fd48d6 None admin] RBAC: Adding query filter params (name=admin) {{(pid=92057) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} May 23 12:41:44.875305 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9529f812-f086-4168-b218-10ec80fd48d6 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:41:44.876824 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9529f812-f086-4168-b218-10ec80fd48d6 None admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:41:44.887134 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 356/724] 10.222.0.30 () {56 vars in 1099 bytes} [Mon May 23 12:41:44 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 194 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) May 23 12:41:44.892247 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9529f812-f086-4168-b218-10ec80fd48d6 None admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:41:44.894847 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.auth.core [None req-5ca22979-a2df-4c92-9c45-4a485d78f650 None None] MFA Rules not processed for user `0f7119f8fa5c4445977671140ded91a4`. Rule list: `[]` (Enabled: `True`). {{(pid=92056) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 12:41:44.896086 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.fernet_utils [None req-5ca22979-a2df-4c92-9c45-4a485d78f650 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92056) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 12:41:45.037708 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 369/725] 10.222.0.30 () {58 vars in 1023 bytes} [Mon May 23 12:41:44 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 187 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) May 23 12:41:45.076230 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ca22979-a2df-4c92-9c45-4a485d78f650 None None] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:41:45.081490 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=100 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:41:45.091044 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9529f812-f086-4168-b218-10ec80fd48d6 None admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:41:45.149406 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18fdea60-81f6-4beb-9f89-fe67ebd1968e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:41:45.150166 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18fdea60-81f6-4beb-9f89-fe67ebd1968e None admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:41:45.150450 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18fdea60-81f6-4beb-9f89-fe67ebd1968e None admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:41:45.150726 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18fdea60-81f6-4beb-9f89-fe67ebd1968e None admin] PATH_INFO: `/v3/domains/Default` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:41:45.164437 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18fdea60-81f6-4beb-9f89-fe67ebd1968e None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:41:45.165958 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18fdea60-81f6-4beb-9f89-fe67ebd1968e None admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:41:45.185061 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application [None req-18fdea60-81f6-4beb-9f89-fe67ebd1968e None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. May 23 12:41:45.185061 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application Traceback (most recent call last): May 23 12:41:45.185061 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request May 23 12:41:45.185061 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application rv = self.dispatch_request() May 23 12:41:45.185061 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request May 23 12:41:45.185061 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) May 23 12:41:45.185061 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper May 23 12:41:45.185061 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) May 23 12:41:45.185061 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view May 23 12:41:45.185061 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) May 23 12:41:45.185061 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request May 23 12:41:45.185061 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) May 23 12:41:45.185061 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get May 23 12:41:45.185061 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self._get_domain(domain_id) May 23 12:41:45.185061 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain May 23 12:41:45.186184 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) May 23 12:41:45.186184 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped May 23 12:41:45.186184 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) May 23 12:41:45.186184 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun May 23 12:41:45.186184 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) May 23 12:41:45.186184 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func May 23 12:41:45.186184 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self.get_or_create( May 23 12:41:45.186184 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create May 23 12:41:45.186184 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application with Lock( May 23 12:41:45.186184 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ May 23 12:41:45.186184 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self._enter() May 23 12:41:45.186184 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter May 23 12:41:45.186184 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) May 23 12:41:45.186184 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create May 23 12:41:45.186184 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self.creator() May 23 12:41:45.186184 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value May 23 12:41:45.186184 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application created_value = creator( May 23 12:41:45.187097 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain May 23 12:41:45.187097 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) May 23 12:41:45.187097 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. May 23 12:41:45.187097 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application May 23 12:41:45.187097 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 357/726] 10.222.0.30 () {56 vars in 1108 bytes} [Mon May 23 12:41:44 2022] GET /identity/v3/domains/Default => generated 87 bytes in 295 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) May 23 12:41:45.187953 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27cc2257-b0fd-403c-8100-f29c3ff1b640 None None] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:41:45.188237 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27cc2257-b0fd-403c-8100-f29c3ff1b640 None None] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:41:45.188514 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27cc2257-b0fd-403c-8100-f29c3ff1b640 None None] PATH_INFO: `/` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:41:45.199326 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 358/727] 10.222.0.30 () {54 vars in 926 bytes} [Mon May 23 12:41:45 2022] GET /identity => generated 270 bytes in 8 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) May 23 12:41:45.199326 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27cc2257-b0fd-403c-8100-f29c3ff1b640 None None] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:41:45.314920 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27cc2257-b0fd-403c-8100-f29c3ff1b640 None None] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:41:45.340758 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ca22979-a2df-4c92-9c45-4a485d78f650 None None] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:41:45.352147 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15063993-709d-4165-bec3-afed2f68f5d4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:41:45.352816 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15063993-709d-4165-bec3-afed2f68f5d4 None admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:41:45.353104 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15063993-709d-4165-bec3-afed2f68f5d4 None admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:41:45.353381 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15063993-709d-4165-bec3-afed2f68f5d4 None admin] PATH_INFO: `/v3/domains` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:41:45.354132 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15063993-709d-4165-bec3-afed2f68f5d4 None admin] RBAC: Adding query filter params (name=Default) {{(pid=92057) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} May 23 12:41:45.354440 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15063993-709d-4165-bec3-afed2f68f5d4 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:41:45.355825 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15063993-709d-4165-bec3-afed2f68f5d4 None admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:41:45.362581 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=100 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:41:45.373626 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 359/728] 10.222.0.30 () {56 vars in 1109 bytes} [Mon May 23 12:41:45 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 173 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) May 23 12:41:45.375013 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11c544e0-aee9-47a3-b9c5-e5768ddc5c1c None None] REQUEST_METHOD: `POST` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:41:45.375302 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11c544e0-aee9-47a3-b9c5-e5768ddc5c1c None None] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:41:45.375578 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11c544e0-aee9-47a3-b9c5-e5768ddc5c1c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:41:45.377918 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-362caebf-328c-4c6c-8dd2-065ad5159787 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:41:45.378806 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-362caebf-328c-4c6c-8dd2-065ad5159787 None admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:41:45.379222 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-362caebf-328c-4c6c-8dd2-065ad5159787 None admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:41:45.379498 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-362caebf-328c-4c6c-8dd2-065ad5159787 None admin] PATH_INFO: `/v3/roles/service` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:41:45.391210 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-362caebf-328c-4c6c-8dd2-065ad5159787 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} May 23 12:41:45.393177 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.auth.core [None req-11c544e0-aee9-47a3-b9c5-e5768ddc5c1c None None] MFA Rules not processed for user `0f7119f8fa5c4445977671140ded91a4`. Rule list: `[]` (Enabled: `True`). {{(pid=92057) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 12:41:45.394401 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.fernet_utils [None req-11c544e0-aee9-47a3-b9c5-e5768ddc5c1c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92057) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 12:41:45.398008 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=100 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:41:45.400464 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application [None req-362caebf-328c-4c6c-8dd2-065ad5159787 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. May 23 12:41:45.400464 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application Traceback (most recent call last): May 23 12:41:45.400464 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request May 23 12:41:45.400464 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application rv = self.dispatch_request() May 23 12:41:45.400464 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request May 23 12:41:45.400464 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) May 23 12:41:45.400464 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper May 23 12:41:45.400464 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) May 23 12:41:45.400464 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view May 23 12:41:45.400464 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) May 23 12:41:45.400464 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request May 23 12:41:45.400464 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) May 23 12:41:45.400464 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get May 23 12:41:45.400464 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self._get_role(role_id) May 23 12:41:45.400464 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role May 23 12:41:45.400464 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') May 23 12:41:45.401399 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call May 23 12:41:45.401399 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( May 23 12:41:45.401399 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data May 23 12:41:45.401399 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) May 23 12:41:45.401399 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped May 23 12:41:45.401399 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) May 23 12:41:45.401399 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun May 23 12:41:45.401399 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) May 23 12:41:45.401399 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func May 23 12:41:45.401399 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self.get_or_create( May 23 12:41:45.401399 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create May 23 12:41:45.401399 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application with Lock( May 23 12:41:45.401399 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ May 23 12:41:45.401399 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self._enter() May 23 12:41:45.401399 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter May 23 12:41:45.401399 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) May 23 12:41:45.402606 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create May 23 12:41:45.402606 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self.creator() May 23 12:41:45.402606 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value May 23 12:41:45.402606 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application created_value = creator( May 23 12:41:45.402606 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role May 23 12:41:45.402606 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) May 23 12:41:45.402606 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role May 23 12:41:45.402606 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() May 23 12:41:45.402606 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role May 23 12:41:45.402606 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) May 23 12:41:45.402606 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. May 23 12:41:45.402606 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application May 23 12:41:45.406906 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 370/729] 10.222.0.30 () {56 vars in 1102 bytes} [Mon May 23 12:41:45 2022] GET /identity/v3/roles/service => generated 85 bytes in 326 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) May 23 12:41:45.407438 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-362caebf-328c-4c6c-8dd2-065ad5159787 None admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:41:45.471953 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 360/730] 10.222.0.30 () {58 vars in 1023 bytes} [Mon May 23 12:41:45 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 98 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) May 23 12:41:45.472821 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11c544e0-aee9-47a3-b9c5-e5768ddc5c1c None None] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:41:45.522984 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-362caebf-328c-4c6c-8dd2-065ad5159787 None admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:41:45.560253 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d8d82c6-e873-4278-a99b-251056037370 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:41:45.561149 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d8d82c6-e873-4278-a99b-251056037370 None admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:41:45.561431 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d8d82c6-e873-4278-a99b-251056037370 None admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:41:45.561700 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d8d82c6-e873-4278-a99b-251056037370 None admin] PATH_INFO: `/v3/users/swift` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:41:45.569939 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d8d82c6-e873-4278-a99b-251056037370 None admin] RBAC: Authorizing `identity:get_user(user_id=swift)` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:41:45.571602 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d8d82c6-e873-4278-a99b-251056037370 None admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:41:45.579797 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11c544e0-aee9-47a3-b9c5-e5768ddc5c1c None None] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:41:45.586811 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application [None req-4d8d82c6-e873-4278-a99b-251056037370 None admin] Could not find user: swift.: keystone.exception.UserNotFound: Could not find user: swift. May 23 12:41:45.586811 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application Traceback (most recent call last): May 23 12:41:45.586811 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request May 23 12:41:45.586811 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application rv = self.dispatch_request() May 23 12:41:45.586811 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request May 23 12:41:45.586811 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) May 23 12:41:45.586811 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper May 23 12:41:45.586811 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) May 23 12:41:45.586811 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view May 23 12:41:45.586811 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) May 23 12:41:45.586811 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request May 23 12:41:45.586811 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) May 23 12:41:45.586811 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get May 23 12:41:45.586811 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self._get_user(user_id) May 23 12:41:45.586811 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user May 23 12:41:45.586811 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) May 23 12:41:45.588978 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped May 23 12:41:45.588978 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) May 23 12:41:45.588978 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper May 23 12:41:45.588978 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) May 23 12:41:45.588978 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper May 23 12:41:45.588978 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) May 23 12:41:45.588978 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun May 23 12:41:45.588978 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) May 23 12:41:45.588978 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func May 23 12:41:45.588978 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self.get_or_create( May 23 12:41:45.588978 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create May 23 12:41:45.588978 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application with Lock( May 23 12:41:45.588978 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ May 23 12:41:45.588978 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self._enter() May 23 12:41:45.588978 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter May 23 12:41:45.588978 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) May 23 12:41:45.588978 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create May 23 12:41:45.588978 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self.creator() May 23 12:41:45.591714 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value May 23 12:41:45.591714 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application created_value = creator( May 23 12:41:45.591714 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user May 23 12:41:45.591714 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) May 23 12:41:45.591714 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user May 23 12:41:45.591714 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) May 23 12:41:45.591714 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user May 23 12:41:45.591714 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) May 23 12:41:45.591714 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: swift. May 23 12:41:45.591714 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application May 23 12:41:45.591714 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 371/731] 10.222.0.30 () {56 vars in 1096 bytes} [Mon May 23 12:41:45 2022] GET /identity/v3/users/swift => generated 83 bytes in 181 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) May 23 12:41:45.591714 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d8d82c6-e873-4278-a99b-251056037370 None admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:41:45.622051 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15b576cc-c0f5-417b-8664-0e31325e9cb3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:41:45.622724 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15b576cc-c0f5-417b-8664-0e31325e9cb3 None admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:41:45.623009 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15b576cc-c0f5-417b-8664-0e31325e9cb3 None admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:41:45.623288 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15b576cc-c0f5-417b-8664-0e31325e9cb3 None admin] PATH_INFO: `/v3/roles` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:41:45.624035 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15b576cc-c0f5-417b-8664-0e31325e9cb3 None admin] RBAC: Adding query filter params (name=service) {{(pid=92057) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} May 23 12:41:45.624342 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15b576cc-c0f5-417b-8664-0e31325e9cb3 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:41:45.625745 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15b576cc-c0f5-417b-8664-0e31325e9cb3 None admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:41:45.635040 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=100 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:41:45.640053 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 361/732] 10.222.0.30 () {56 vars in 1103 bytes} [Mon May 23 12:41:45 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 163 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) May 23 12:41:45.640053 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15b576cc-c0f5-417b-8664-0e31325e9cb3 None admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:41:45.640559 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=100 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:41:45.710037 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d8d82c6-e873-4278-a99b-251056037370 None admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:41:45.749548 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4575cbcb-5600-4a98-8ff6-82f471c6ea55 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:41:45.750420 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4575cbcb-5600-4a98-8ff6-82f471c6ea55 None admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:41:45.750706 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4575cbcb-5600-4a98-8ff6-82f471c6ea55 None admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:41:45.750977 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4575cbcb-5600-4a98-8ff6-82f471c6ea55 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:41:45.752014 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15b576cc-c0f5-417b-8664-0e31325e9cb3 None admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:41:45.762997 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4575cbcb-5600-4a98-8ff6-82f471c6ea55 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} May 23 12:41:45.771840 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application [None req-4575cbcb-5600-4a98-8ff6-82f471c6ea55 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. May 23 12:41:45.771840 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application Traceback (most recent call last): May 23 12:41:45.771840 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request May 23 12:41:45.771840 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application rv = self.dispatch_request() May 23 12:41:45.771840 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request May 23 12:41:45.771840 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) May 23 12:41:45.771840 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper May 23 12:41:45.771840 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) May 23 12:41:45.771840 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view May 23 12:41:45.771840 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) May 23 12:41:45.771840 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request May 23 12:41:45.771840 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) May 23 12:41:45.771840 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get May 23 12:41:45.771840 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self._get_role(role_id) May 23 12:41:45.771840 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role May 23 12:41:45.771840 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') May 23 12:41:45.772782 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call May 23 12:41:45.772782 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( May 23 12:41:45.772782 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data May 23 12:41:45.772782 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) May 23 12:41:45.772782 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped May 23 12:41:45.772782 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) May 23 12:41:45.772782 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun May 23 12:41:45.772782 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) May 23 12:41:45.772782 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func May 23 12:41:45.772782 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self.get_or_create( May 23 12:41:45.772782 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create May 23 12:41:45.772782 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application with Lock( May 23 12:41:45.772782 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ May 23 12:41:45.772782 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self._enter() May 23 12:41:45.772782 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter May 23 12:41:45.772782 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) May 23 12:41:45.773636 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create May 23 12:41:45.773636 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self.creator() May 23 12:41:45.773636 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value May 23 12:41:45.773636 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application created_value = creator( May 23 12:41:45.773636 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role May 23 12:41:45.773636 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) May 23 12:41:45.773636 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role May 23 12:41:45.773636 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() May 23 12:41:45.773636 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role May 23 12:41:45.773636 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) May 23 12:41:45.773636 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. May 23 12:41:45.773636 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application May 23 12:41:45.773636 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 372/733] 10.222.0.30 () {56 vars in 1096 bytes} [Mon May 23 12:41:45 2022] GET /identity/v3/roles/admin => generated 83 bytes in 185 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) May 23 12:41:45.774392 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4575cbcb-5600-4a98-8ff6-82f471c6ea55 None admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:41:45.792450 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-871480aa-f580-4b3a-a7b6-af6c2dd42911 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:41:45.793496 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-871480aa-f580-4b3a-a7b6-af6c2dd42911 None admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:41:45.793929 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-871480aa-f580-4b3a-a7b6-af6c2dd42911 None admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:41:45.794355 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-871480aa-f580-4b3a-a7b6-af6c2dd42911 None admin] PATH_INFO: `/v3/users` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:41:45.795195 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-871480aa-f580-4b3a-a7b6-af6c2dd42911 None admin] RBAC: Adding query filter params (domain_id=default, name=swift) {{(pid=92057) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} May 23 12:41:45.795508 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-871480aa-f580-4b3a-a7b6-af6c2dd42911 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:41:45.796940 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-871480aa-f580-4b3a-a7b6-af6c2dd42911 None admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:41:45.815628 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 362/734] 10.222.0.30 () {56 vars in 1135 bytes} [Mon May 23 12:41:45 2022] GET /identity/v3/users?domain_id=default&name=swift => generated 370 bytes in 179 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) May 23 12:41:45.817005 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c201d0a-263c-4f38-8cd9-7f861e26b5b4 None None] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:41:45.817290 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c201d0a-263c-4f38-8cd9-7f861e26b5b4 None None] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:41:45.817565 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c201d0a-263c-4f38-8cd9-7f861e26b5b4 None None] PATH_INFO: `/` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:41:45.822016 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 363/735] 10.222.0.30 () {54 vars in 926 bytes} [Mon May 23 12:41:45 2022] GET /identity => generated 270 bytes in 6 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) May 23 12:41:45.822950 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c201d0a-263c-4f38-8cd9-7f861e26b5b4 None None] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:41:45.884951 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=100 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:41:45.887862 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=100 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:41:45.890962 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4575cbcb-5600-4a98-8ff6-82f471c6ea55 None admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:41:45.925892 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0cabaf6c-a57a-4be5-ad40-4cd8d0a30104 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:41:45.926689 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cabaf6c-a57a-4be5-ad40-4cd8d0a30104 None admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:41:45.926969 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cabaf6c-a57a-4be5-ad40-4cd8d0a30104 None admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:41:45.927260 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cabaf6c-a57a-4be5-ad40-4cd8d0a30104 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:41:45.928092 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c201d0a-263c-4f38-8cd9-7f861e26b5b4 None None] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:41:45.939189 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0cabaf6c-a57a-4be5-ad40-4cd8d0a30104 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:41:45.940686 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0cabaf6c-a57a-4be5-ad40-4cd8d0a30104 None admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:41:45.958204 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application [None req-0cabaf6c-a57a-4be5-ad40-4cd8d0a30104 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. May 23 12:41:45.958204 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application Traceback (most recent call last): May 23 12:41:45.958204 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request May 23 12:41:45.958204 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application rv = self.dispatch_request() May 23 12:41:45.958204 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request May 23 12:41:45.958204 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) May 23 12:41:45.958204 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper May 23 12:41:45.958204 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) May 23 12:41:45.958204 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view May 23 12:41:45.958204 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) May 23 12:41:45.958204 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request May 23 12:41:45.958204 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) May 23 12:41:45.958204 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get May 23 12:41:45.958204 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self._get_domain(domain_id) May 23 12:41:45.958204 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain May 23 12:41:45.959416 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) May 23 12:41:45.959416 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped May 23 12:41:45.959416 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) May 23 12:41:45.959416 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun May 23 12:41:45.959416 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) May 23 12:41:45.959416 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func May 23 12:41:45.959416 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self.get_or_create( May 23 12:41:45.959416 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create May 23 12:41:45.959416 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application with Lock( May 23 12:41:45.959416 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ May 23 12:41:45.959416 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self._enter() May 23 12:41:45.959416 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter May 23 12:41:45.959416 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) May 23 12:41:45.959416 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create May 23 12:41:45.959416 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self.creator() May 23 12:41:45.959416 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value May 23 12:41:45.959416 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application created_value = creator( May 23 12:41:45.960341 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain May 23 12:41:45.960341 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) May 23 12:41:45.960341 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. May 23 12:41:45.960341 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application May 23 12:41:45.960341 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 373/736] 10.222.0.30 () {56 vars in 1108 bytes} [Mon May 23 12:41:45 2022] GET /identity/v3/domains/Default => generated 87 bytes in 185 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) May 23 12:41:45.960341 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0cabaf6c-a57a-4be5-ad40-4cd8d0a30104 None admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:41:45.963702 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d449c871-d90c-4814-86c9-a24cf69b9fca None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:41:45.964376 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d449c871-d90c-4814-86c9-a24cf69b9fca None admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:41:45.964664 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d449c871-d90c-4814-86c9-a24cf69b9fca None admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:41:45.964942 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d449c871-d90c-4814-86c9-a24cf69b9fca None admin] PATH_INFO: `/v3/roles` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:41:45.965704 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d449c871-d90c-4814-86c9-a24cf69b9fca None admin] RBAC: Adding query filter params (name=admin) {{(pid=92057) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} May 23 12:41:45.966049 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d449c871-d90c-4814-86c9-a24cf69b9fca None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:41:45.967424 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d449c871-d90c-4814-86c9-a24cf69b9fca None admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:41:45.976355 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 364/737] 10.222.0.30 () {56 vars in 1099 bytes} [Mon May 23 12:41:45 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 154 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) May 23 12:41:45.977569 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2570514-2088-4845-89db-7161ab870f2d None None] REQUEST_METHOD: `POST` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:41:45.977854 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2570514-2088-4845-89db-7161ab870f2d None None] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:41:45.978172 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2570514-2088-4845-89db-7161ab870f2d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:41:45.994300 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.auth.core [None req-a2570514-2088-4845-89db-7161ab870f2d None None] MFA Rules not processed for user `0f7119f8fa5c4445977671140ded91a4`. Rule list: `[]` (Enabled: `True`). {{(pid=92057) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 12:41:45.995477 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.fernet_utils [None req-a2570514-2088-4845-89db-7161ab870f2d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92057) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 12:41:46.061529 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0cabaf6c-a57a-4be5-ad40-4cd8d0a30104 None admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:41:46.062006 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 365/738] 10.222.0.30 () {58 vars in 1023 bytes} [Mon May 23 12:41:45 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) May 23 12:41:46.063302 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4d03e7d-aaf9-4738-a892-fd0c98d9aa13 None None] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:41:46.063592 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4d03e7d-aaf9-4738-a892-fd0c98d9aa13 None None] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:41:46.068034 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4d03e7d-aaf9-4738-a892-fd0c98d9aa13 None None] PATH_INFO: `/` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:41:46.068034 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 366/739] 10.222.0.30 () {54 vars in 926 bytes} [Mon May 23 12:41:46 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) May 23 12:41:46.068034 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4d03e7d-aaf9-4738-a892-fd0c98d9aa13 None None] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:41:46.104165 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3dad3953-5ec1-4f1f-a7fa-798144cbecf0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:41:46.104820 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3dad3953-5ec1-4f1f-a7fa-798144cbecf0 None admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:41:46.105153 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3dad3953-5ec1-4f1f-a7fa-798144cbecf0 None admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:41:46.105379 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3dad3953-5ec1-4f1f-a7fa-798144cbecf0 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:41:46.126631 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=100 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:41:46.126785 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=100 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:41:46.173474 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3dad3953-5ec1-4f1f-a7fa-798144cbecf0 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:41:46.174237 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4d03e7d-aaf9-4738-a892-fd0c98d9aa13 None None] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:41:46.174974 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3dad3953-5ec1-4f1f-a7fa-798144cbecf0 None admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:41:46.217577 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a6d259d-e004-4549-a470-b489947f1766 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:41:46.218513 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a6d259d-e004-4549-a470-b489947f1766 None admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:41:46.218952 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a6d259d-e004-4549-a470-b489947f1766 None admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:41:46.219380 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a6d259d-e004-4549-a470-b489947f1766 None admin] PATH_INFO: `/v3/domains` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:41:46.220498 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a6d259d-e004-4549-a470-b489947f1766 None admin] RBAC: Adding query filter params (name=Default) {{(pid=92057) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} May 23 12:41:46.220976 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a6d259d-e004-4549-a470-b489947f1766 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:41:46.223169 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a6d259d-e004-4549-a470-b489947f1766 None admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:41:46.238053 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 367/740] 10.222.0.30 () {56 vars in 1109 bytes} [Mon May 23 12:41:46 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 172 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) May 23 12:41:46.238849 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a6d259d-e004-4549-a470-b489947f1766 None admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:41:46.288049 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 374/741] 10.222.0.30 () {58 vars in 1282 bytes} [Mon May 23 12:41:45 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 328 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) May 23 12:41:46.289352 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e09392c-cd53-4206-981e-54948e8fb268 None None] REQUEST_METHOD: `POST` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:41:46.289641 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e09392c-cd53-4206-981e-54948e8fb268 None None] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:41:46.289960 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e09392c-cd53-4206-981e-54948e8fb268 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:41:46.305645 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.auth.core [None req-2e09392c-cd53-4206-981e-54948e8fb268 None None] MFA Rules not processed for user `0f7119f8fa5c4445977671140ded91a4`. Rule list: `[]` (Enabled: `True`). {{(pid=92056) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 12:41:46.306866 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.fernet_utils [None req-2e09392c-cd53-4206-981e-54948e8fb268 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92056) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 12:41:46.338429 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a6d259d-e004-4549-a470-b489947f1766 None admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:41:46.351356 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=100 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:41:46.353477 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=100 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:41:46.371736 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40a6880a-5108-4f65-bfb1-05f5a86a9a58 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:41:46.372311 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 375/742] 10.222.0.30 () {58 vars in 1023 bytes} [Mon May 23 12:41:46 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 84 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) May 23 12:41:46.372517 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40a6880a-5108-4f65-bfb1-05f5a86a9a58 None admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:41:46.372797 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40a6880a-5108-4f65-bfb1-05f5a86a9a58 None admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:41:46.373095 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40a6880a-5108-4f65-bfb1-05f5a86a9a58 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:41:46.373192 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e09392c-cd53-4206-981e-54948e8fb268 None None] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:41:46.384130 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40a6880a-5108-4f65-bfb1-05f5a86a9a58 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:41:46.385616 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40a6880a-5108-4f65-bfb1-05f5a86a9a58 None admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:41:46.403937 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application [None req-40a6880a-5108-4f65-bfb1-05f5a86a9a58 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. May 23 12:41:46.403937 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application Traceback (most recent call last): May 23 12:41:46.403937 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request May 23 12:41:46.403937 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application rv = self.dispatch_request() May 23 12:41:46.403937 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request May 23 12:41:46.403937 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) May 23 12:41:46.403937 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper May 23 12:41:46.403937 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) May 23 12:41:46.403937 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view May 23 12:41:46.403937 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) May 23 12:41:46.403937 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request May 23 12:41:46.403937 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) May 23 12:41:46.403937 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get May 23 12:41:46.403937 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self._get_domain(domain_id) May 23 12:41:46.403937 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain May 23 12:41:46.405021 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) May 23 12:41:46.405021 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped May 23 12:41:46.405021 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) May 23 12:41:46.405021 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun May 23 12:41:46.405021 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) May 23 12:41:46.405021 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func May 23 12:41:46.405021 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self.get_or_create( May 23 12:41:46.405021 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create May 23 12:41:46.405021 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application with Lock( May 23 12:41:46.405021 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ May 23 12:41:46.405021 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self._enter() May 23 12:41:46.405021 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter May 23 12:41:46.405021 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) May 23 12:41:46.405021 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create May 23 12:41:46.405021 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self.creator() May 23 12:41:46.405021 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value May 23 12:41:46.405021 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application created_value = creator( May 23 12:41:46.405981 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain May 23 12:41:46.405981 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) May 23 12:41:46.405981 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. May 23 12:41:46.405981 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application May 23 12:41:46.405981 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 368/743] 10.222.0.30 () {56 vars in 1108 bytes} [Mon May 23 12:41:46 2022] GET /identity/v3/domains/Default => generated 87 bytes in 167 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) May 23 12:41:46.405981 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40a6880a-5108-4f65-bfb1-05f5a86a9a58 None admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:41:46.478129 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e09392c-cd53-4206-981e-54948e8fb268 None None] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:41:46.507361 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40a6880a-5108-4f65-bfb1-05f5a86a9a58 None admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:41:46.514048 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc91aabe-59a5-4f75-a050-ec0fd5787876 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:41:46.514701 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc91aabe-59a5-4f75-a050-ec0fd5787876 None admin] REQUEST_METHOD: `POST` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:41:46.514978 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc91aabe-59a5-4f75-a050-ec0fd5787876 None admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:41:46.515249 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc91aabe-59a5-4f75-a050-ec0fd5787876 None admin] PATH_INFO: `/v3/services` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:41:46.515961 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc91aabe-59a5-4f75-a050-ec0fd5787876 None admin] RBAC: Authorizing `identity:create_service()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:41:46.517313 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc91aabe-59a5-4f75-a050-ec0fd5787876 None admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:41:46.541696 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c7cddcf-adc3-4358-84d6-4f73b5f657c9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:41:46.542935 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c7cddcf-adc3-4358-84d6-4f73b5f657c9 None admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:41:46.543376 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c7cddcf-adc3-4358-84d6-4f73b5f657c9 None admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:41:46.543790 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c7cddcf-adc3-4358-84d6-4f73b5f657c9 None admin] PATH_INFO: `/v3/users/glance-swift` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:41:46.556273 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c7cddcf-adc3-4358-84d6-4f73b5f657c9 None admin] RBAC: Authorizing `identity:get_user(user_id=glance-swift)` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:41:46.558624 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c7cddcf-adc3-4358-84d6-4f73b5f657c9 None admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:41:46.581042 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application [None req-6c7cddcf-adc3-4358-84d6-4f73b5f657c9 None admin] Could not find user: glance-swift.: keystone.exception.UserNotFound: Could not find user: glance-swift. May 23 12:41:46.581042 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application Traceback (most recent call last): May 23 12:41:46.581042 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request May 23 12:41:46.581042 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application rv = self.dispatch_request() May 23 12:41:46.581042 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request May 23 12:41:46.581042 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) May 23 12:41:46.581042 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper May 23 12:41:46.581042 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) May 23 12:41:46.581042 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view May 23 12:41:46.581042 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) May 23 12:41:46.581042 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request May 23 12:41:46.581042 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) May 23 12:41:46.581042 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get May 23 12:41:46.581042 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self._get_user(user_id) May 23 12:41:46.581042 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user May 23 12:41:46.581042 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) May 23 12:41:46.582263 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped May 23 12:41:46.582263 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) May 23 12:41:46.582263 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper May 23 12:41:46.582263 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) May 23 12:41:46.582263 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper May 23 12:41:46.582263 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) May 23 12:41:46.582263 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun May 23 12:41:46.582263 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) May 23 12:41:46.582263 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func May 23 12:41:46.582263 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self.get_or_create( May 23 12:41:46.582263 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create May 23 12:41:46.582263 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application with Lock( May 23 12:41:46.582263 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ May 23 12:41:46.582263 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self._enter() May 23 12:41:46.582263 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter May 23 12:41:46.582263 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) May 23 12:41:46.582263 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create May 23 12:41:46.582263 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self.creator() May 23 12:41:46.583493 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value May 23 12:41:46.583493 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application created_value = creator( May 23 12:41:46.583493 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user May 23 12:41:46.583493 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) May 23 12:41:46.583493 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user May 23 12:41:46.583493 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) May 23 12:41:46.583493 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user May 23 12:41:46.583493 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) May 23 12:41:46.583493 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: glance-swift. May 23 12:41:46.583493 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application May 23 12:41:46.583493 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 369/744] 10.222.0.30 () {56 vars in 1117 bytes} [Mon May 23 12:41:46 2022] GET /identity/v3/users/glance-swift => generated 90 bytes in 174 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) May 23 12:41:46.583493 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c7cddcf-adc3-4358-84d6-4f73b5f657c9 None admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:41:46.607564 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=100 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:41:46.633754 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 376/745] 10.222.0.30 () {60 vars in 1141 bytes} [Mon May 23 12:41:46 2022] POST /identity/v3/services => generated 240 bytes in 261 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) May 23 12:41:46.634677 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc91aabe-59a5-4f75-a050-ec0fd5787876 None admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:41:46.686497 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:41:46.701864 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c7cddcf-adc3-4358-84d6-4f73b5f657c9 None admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:41:46.757520 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4565e52-b6e3-4b4d-9bbb-4da30e6cad79 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:41:46.758320 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4565e52-b6e3-4b4d-9bbb-4da30e6cad79 None admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:41:46.758608 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4565e52-b6e3-4b4d-9bbb-4da30e6cad79 None admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:41:46.758889 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4565e52-b6e3-4b4d-9bbb-4da30e6cad79 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:41:46.767847 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc91aabe-59a5-4f75-a050-ec0fd5787876 None admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:41:46.773862 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4565e52-b6e3-4b4d-9bbb-4da30e6cad79 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:41:46.775684 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4565e52-b6e3-4b4d-9bbb-4da30e6cad79 None admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:41:46.800981 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application [None req-e4565e52-b6e3-4b4d-9bbb-4da30e6cad79 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. May 23 12:41:46.800981 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application Traceback (most recent call last): May 23 12:41:46.800981 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request May 23 12:41:46.800981 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application rv = self.dispatch_request() May 23 12:41:46.800981 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request May 23 12:41:46.800981 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) May 23 12:41:46.800981 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper May 23 12:41:46.800981 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) May 23 12:41:46.800981 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view May 23 12:41:46.800981 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) May 23 12:41:46.800981 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request May 23 12:41:46.800981 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) May 23 12:41:46.800981 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get May 23 12:41:46.800981 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self._get_domain(domain_id) May 23 12:41:46.800981 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain May 23 12:41:46.802034 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) May 23 12:41:46.802034 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped May 23 12:41:46.802034 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) May 23 12:41:46.802034 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun May 23 12:41:46.802034 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) May 23 12:41:46.802034 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func May 23 12:41:46.802034 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self.get_or_create( May 23 12:41:46.802034 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create May 23 12:41:46.802034 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application with Lock( May 23 12:41:46.802034 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ May 23 12:41:46.802034 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self._enter() May 23 12:41:46.802034 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter May 23 12:41:46.802034 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) May 23 12:41:46.802034 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create May 23 12:41:46.802034 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self.creator() May 23 12:41:46.802034 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value May 23 12:41:46.802034 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application created_value = creator( May 23 12:41:46.803243 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain May 23 12:41:46.803243 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) May 23 12:41:46.803243 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. May 23 12:41:46.803243 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application May 23 12:41:46.803477 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 370/746] 10.222.0.30 () {56 vars in 1108 bytes} [Mon May 23 12:41:46 2022] GET /identity/v3/domains/Default => generated 87 bytes in 223 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) May 23 12:41:46.804283 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4565e52-b6e3-4b4d-9bbb-4da30e6cad79 None admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:41:46.812119 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5fb5b5b-2e39-476b-83d0-a2fec89a7777 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:41:46.812801 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5fb5b5b-2e39-476b-83d0-a2fec89a7777 None admin] REQUEST_METHOD: `POST` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:41:46.813085 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5fb5b5b-2e39-476b-83d0-a2fec89a7777 None admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:41:46.813365 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5fb5b5b-2e39-476b-83d0-a2fec89a7777 None admin] PATH_INFO: `/v3/services` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:41:46.814166 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5fb5b5b-2e39-476b-83d0-a2fec89a7777 None admin] RBAC: Authorizing `identity:create_service()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:41:46.815558 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5fb5b5b-2e39-476b-83d0-a2fec89a7777 None admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:41:46.877031 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=100 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:41:46.906992 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4565e52-b6e3-4b4d-9bbb-4da30e6cad79 None admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:41:46.921915 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 377/747] 10.222.0.30 () {60 vars in 1141 bytes} [Mon May 23 12:41:46 2022] POST /identity/v3/services => generated 251 bytes in 280 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) May 23 12:41:46.921915 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5fb5b5b-2e39-476b-83d0-a2fec89a7777 None admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:41:46.968337 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a219e8e3-12e6-4580-a554-7d424b07fe6d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:41:46.969328 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a219e8e3-12e6-4580-a554-7d424b07fe6d None admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:41:46.969794 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a219e8e3-12e6-4580-a554-7d424b07fe6d None admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:41:46.970957 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a219e8e3-12e6-4580-a554-7d424b07fe6d None admin] PATH_INFO: `/v3/domains` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:41:46.971923 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a219e8e3-12e6-4580-a554-7d424b07fe6d None admin] RBAC: Adding query filter params (name=Default) {{(pid=92057) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} May 23 12:41:46.972421 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a219e8e3-12e6-4580-a554-7d424b07fe6d None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:41:46.975805 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a219e8e3-12e6-4580-a554-7d424b07fe6d None admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:41:46.995589 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 371/748] 10.222.0.30 () {56 vars in 1109 bytes} [Mon May 23 12:41:46 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 191 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) May 23 12:41:46.997054 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a219e8e3-12e6-4580-a554-7d424b07fe6d None admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:41:47.047832 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:41:47.053925 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5fb5b5b-2e39-476b-83d0-a2fec89a7777 None admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:41:47.104543 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0e4c6df-1901-4e96-9cf8-ffc94acb4d6d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:41:47.105768 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0e4c6df-1901-4e96-9cf8-ffc94acb4d6d None admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:41:47.106730 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0e4c6df-1901-4e96-9cf8-ffc94acb4d6d None admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:41:47.107523 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0e4c6df-1901-4e96-9cf8-ffc94acb4d6d None admin] PATH_INFO: `/v3/users` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:41:47.108591 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0e4c6df-1901-4e96-9cf8-ffc94acb4d6d None admin] RBAC: Adding query filter params (domain_id=default, name=glance-swift) {{(pid=92056) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} May 23 12:41:47.109227 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0e4c6df-1901-4e96-9cf8-ffc94acb4d6d None admin] RBAC: Authorizing `identity:list_users()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:41:47.111201 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0e4c6df-1901-4e96-9cf8-ffc94acb4d6d None admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:41:47.133970 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 378/749] 10.222.0.30 () {56 vars in 1149 bytes} [Mon May 23 12:41:46 2022] GET /identity/v3/users?domain_id=default&name=glance-swift => generated 384 bytes in 219 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) May 23 12:41:47.134860 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0e4c6df-1901-4e96-9cf8-ffc94acb4d6d None admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:41:47.136091 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a219e8e3-12e6-4580-a554-7d424b07fe6d None admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:41:47.169550 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=100 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:41:47.178393 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6cf9e2b-aca9-4e8d-8807-0cdf7e0dc7a6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:41:47.179285 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6cf9e2b-aca9-4e8d-8807-0cdf7e0dc7a6 None admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:41:47.179727 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6cf9e2b-aca9-4e8d-8807-0cdf7e0dc7a6 None admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:41:47.182087 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6cf9e2b-aca9-4e8d-8807-0cdf7e0dc7a6 None admin] PATH_INFO: `/v3/domains` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:41:47.182087 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6cf9e2b-aca9-4e8d-8807-0cdf7e0dc7a6 None admin] RBAC: Adding query filter params (name=Default) {{(pid=92057) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} May 23 12:41:47.182087 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6cf9e2b-aca9-4e8d-8807-0cdf7e0dc7a6 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:41:47.182965 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6cf9e2b-aca9-4e8d-8807-0cdf7e0dc7a6 None admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:41:47.195751 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 372/750] 10.222.0.30 () {56 vars in 1109 bytes} [Mon May 23 12:41:46 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 199 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) May 23 12:41:47.196810 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6cf9e2b-aca9-4e8d-8807-0cdf7e0dc7a6 None admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:41:47.278403 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0e4c6df-1901-4e96-9cf8-ffc94acb4d6d None admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:41:47.314924 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a20b2fd-66c8-4604-bf7a-aa6f57b984ae None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:41:47.315945 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a20b2fd-66c8-4604-bf7a-aa6f57b984ae None admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:41:47.316387 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a20b2fd-66c8-4604-bf7a-aa6f57b984ae None admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:41:47.316870 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a20b2fd-66c8-4604-bf7a-aa6f57b984ae None admin] PATH_INFO: `/v3/users/nova` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:41:47.327234 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6cf9e2b-aca9-4e8d-8807-0cdf7e0dc7a6 None admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:41:47.328207 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a20b2fd-66c8-4604-bf7a-aa6f57b984ae None admin] RBAC: Authorizing `identity:get_user(user_id=nova)` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:41:47.329679 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a20b2fd-66c8-4604-bf7a-aa6f57b984ae None admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:41:47.354561 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application [None req-1a20b2fd-66c8-4604-bf7a-aa6f57b984ae None admin] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova. May 23 12:41:47.354561 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application Traceback (most recent call last): May 23 12:41:47.354561 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request May 23 12:41:47.354561 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application rv = self.dispatch_request() May 23 12:41:47.354561 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request May 23 12:41:47.354561 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) May 23 12:41:47.354561 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper May 23 12:41:47.354561 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) May 23 12:41:47.354561 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view May 23 12:41:47.354561 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) May 23 12:41:47.354561 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request May 23 12:41:47.354561 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) May 23 12:41:47.354561 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get May 23 12:41:47.354561 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self._get_user(user_id) May 23 12:41:47.354561 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user May 23 12:41:47.354561 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) May 23 12:41:47.356216 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped May 23 12:41:47.356216 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) May 23 12:41:47.356216 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper May 23 12:41:47.356216 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) May 23 12:41:47.356216 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper May 23 12:41:47.356216 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) May 23 12:41:47.356216 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun May 23 12:41:47.356216 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) May 23 12:41:47.356216 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func May 23 12:41:47.356216 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self.get_or_create( May 23 12:41:47.356216 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create May 23 12:41:47.356216 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application with Lock( May 23 12:41:47.356216 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ May 23 12:41:47.356216 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self._enter() May 23 12:41:47.356216 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter May 23 12:41:47.356216 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) May 23 12:41:47.356216 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create May 23 12:41:47.356216 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self.creator() May 23 12:41:47.357660 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value May 23 12:41:47.357660 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application created_value = creator( May 23 12:41:47.357660 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user May 23 12:41:47.357660 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) May 23 12:41:47.357660 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user May 23 12:41:47.357660 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) May 23 12:41:47.357660 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user May 23 12:41:47.357660 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) May 23 12:41:47.357660 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: nova. May 23 12:41:47.357660 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application May 23 12:41:47.357660 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 379/751] 10.222.0.30 () {56 vars in 1093 bytes} [Mon May 23 12:41:47 2022] GET /identity/v3/users/nova => generated 82 bytes in 222 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) May 23 12:41:47.357660 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a20b2fd-66c8-4604-bf7a-aa6f57b984ae None admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:41:47.374527 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-faefdb1b-a771-4acf-8f13-aa508b2f8a8c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:41:47.375303 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-faefdb1b-a771-4acf-8f13-aa508b2f8a8c None admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:41:47.375811 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-faefdb1b-a771-4acf-8f13-aa508b2f8a8c None admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:41:47.376091 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-faefdb1b-a771-4acf-8f13-aa508b2f8a8c None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:41:47.379264 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=100 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:41:47.456312 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=100 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:41:47.472180 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a20b2fd-66c8-4604-bf7a-aa6f57b984ae None admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:41:47.476289 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-faefdb1b-a771-4acf-8f13-aa508b2f8a8c None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:41:47.477155 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-faefdb1b-a771-4acf-8f13-aa508b2f8a8c None admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:41:47.526337 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2fb16ba5-4b09-4b7b-b75d-d4bfe59dcc65 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:41:47.527919 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fb16ba5-4b09-4b7b-b75d-d4bfe59dcc65 None admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:41:47.528353 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fb16ba5-4b09-4b7b-b75d-d4bfe59dcc65 None admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:41:47.528802 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fb16ba5-4b09-4b7b-b75d-d4bfe59dcc65 None admin] PATH_INFO: `/v3/projects/service` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:41:47.538607 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2fb16ba5-4b09-4b7b-b75d-d4bfe59dcc65 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:41:47.541105 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2fb16ba5-4b09-4b7b-b75d-d4bfe59dcc65 None admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:41:47.561073 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application [None req-2fb16ba5-4b09-4b7b-b75d-d4bfe59dcc65 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. May 23 12:41:47.561073 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application Traceback (most recent call last): May 23 12:41:47.561073 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request May 23 12:41:47.561073 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application rv = self.dispatch_request() May 23 12:41:47.561073 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request May 23 12:41:47.561073 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) May 23 12:41:47.561073 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper May 23 12:41:47.561073 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) May 23 12:41:47.561073 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view May 23 12:41:47.561073 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) May 23 12:41:47.561073 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request May 23 12:41:47.561073 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) May 23 12:41:47.561073 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get May 23 12:41:47.561073 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self._get_project(project_id) May 23 12:41:47.561073 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project May 23 12:41:47.562274 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) May 23 12:41:47.562274 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped May 23 12:41:47.562274 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) May 23 12:41:47.562274 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun May 23 12:41:47.562274 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) May 23 12:41:47.562274 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func May 23 12:41:47.562274 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self.get_or_create( May 23 12:41:47.562274 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create May 23 12:41:47.562274 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application with Lock( May 23 12:41:47.562274 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ May 23 12:41:47.562274 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self._enter() May 23 12:41:47.562274 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter May 23 12:41:47.562274 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) May 23 12:41:47.562274 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create May 23 12:41:47.562274 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self.creator() May 23 12:41:47.562274 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value May 23 12:41:47.562274 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application created_value = creator( May 23 12:41:47.563429 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project May 23 12:41:47.563429 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) May 23 12:41:47.563429 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project May 23 12:41:47.563429 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() May 23 12:41:47.563429 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project May 23 12:41:47.563429 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) May 23 12:41:47.563429 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. May 23 12:41:47.563429 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application May 23 12:41:47.571397 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 380/752] 10.222.0.30 () {56 vars in 1111 bytes} [Mon May 23 12:41:47 2022] GET /identity/v3/projects/service => generated 88 bytes in 209 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) May 23 12:41:47.571397 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2fb16ba5-4b09-4b7b-b75d-d4bfe59dcc65 None admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:41:47.600359 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 373/753] 10.222.0.30 () {58 vars in 1282 bytes} [Mon May 23 12:41:47 2022] GET /identity/v3/auth/tokens => generated 1002 bytes in 399 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) May 23 12:41:47.601786 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-faefdb1b-a771-4acf-8f13-aa508b2f8a8c None admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:41:47.667731 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=100 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:41:47.691156 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2fb16ba5-4b09-4b7b-b75d-d4bfe59dcc65 None admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:41:47.728244 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=100 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:41:47.736994 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef6e1ae7-0aa8-4fc3-b355-63fc935d6aee None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:41:47.737666 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef6e1ae7-0aa8-4fc3-b355-63fc935d6aee None admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:41:47.737987 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef6e1ae7-0aa8-4fc3-b355-63fc935d6aee None admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:41:47.738264 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef6e1ae7-0aa8-4fc3-b355-63fc935d6aee None admin] PATH_INFO: `/v3/users` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:41:47.739169 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef6e1ae7-0aa8-4fc3-b355-63fc935d6aee None admin] RBAC: Adding query filter params (domain_id=default, name=nova) {{(pid=92056) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} May 23 12:41:47.739400 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef6e1ae7-0aa8-4fc3-b355-63fc935d6aee None admin] RBAC: Authorizing `identity:list_users()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:41:47.740895 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef6e1ae7-0aa8-4fc3-b355-63fc935d6aee None admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:41:47.745731 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-faefdb1b-a771-4acf-8f13-aa508b2f8a8c None admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:41:47.764838 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 381/754] 10.222.0.30 () {56 vars in 1133 bytes} [Mon May 23 12:41:47 2022] GET /identity/v3/users?domain_id=default&name=nova => generated 368 bytes in 191 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) May 23 12:41:47.765833 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef6e1ae7-0aa8-4fc3-b355-63fc935d6aee None admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:41:47.790803 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1276dacb-cc59-47d5-b70d-88b67a399296 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:41:47.792545 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1276dacb-cc59-47d5-b70d-88b67a399296 None admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:41:47.793310 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1276dacb-cc59-47d5-b70d-88b67a399296 None admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:41:47.793754 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1276dacb-cc59-47d5-b70d-88b67a399296 None admin] PATH_INFO: `/v3/projects` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:41:47.795039 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1276dacb-cc59-47d5-b70d-88b67a399296 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=92057) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} May 23 12:41:47.795538 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1276dacb-cc59-47d5-b70d-88b67a399296 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:41:47.798131 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1276dacb-cc59-47d5-b70d-88b67a399296 None admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:41:47.820400 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 374/755] 10.222.0.30 () {56 vars in 1148 bytes} [Mon May 23 12:41:47 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 214 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) May 23 12:41:47.820400 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1276dacb-cc59-47d5-b70d-88b67a399296 None admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:41:47.912254 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef6e1ae7-0aa8-4fc3-b355-63fc935d6aee None admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:41:47.950985 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a03465c7-d8ae-45d0-9a90-3ff299aba86a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:41:47.951772 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a03465c7-d8ae-45d0-9a90-3ff299aba86a None admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:41:47.952051 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a03465c7-d8ae-45d0-9a90-3ff299aba86a None admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:41:47.952322 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a03465c7-d8ae-45d0-9a90-3ff299aba86a None admin] PATH_INFO: `/v3/domains/Default` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:41:47.954638 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=100 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:41:47.961691 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1276dacb-cc59-47d5-b70d-88b67a399296 None admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:41:47.964046 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a03465c7-d8ae-45d0-9a90-3ff299aba86a None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:41:47.965688 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a03465c7-d8ae-45d0-9a90-3ff299aba86a None admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:41:47.983689 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application [None req-a03465c7-d8ae-45d0-9a90-3ff299aba86a None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. May 23 12:41:47.983689 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application Traceback (most recent call last): May 23 12:41:47.983689 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request May 23 12:41:47.983689 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application rv = self.dispatch_request() May 23 12:41:47.983689 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request May 23 12:41:47.983689 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) May 23 12:41:47.983689 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper May 23 12:41:47.983689 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) May 23 12:41:47.983689 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view May 23 12:41:47.983689 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) May 23 12:41:47.983689 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request May 23 12:41:47.983689 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) May 23 12:41:47.983689 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get May 23 12:41:47.983689 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self._get_domain(domain_id) May 23 12:41:47.983689 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain May 23 12:41:47.984810 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) May 23 12:41:47.984810 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped May 23 12:41:47.984810 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) May 23 12:41:47.984810 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun May 23 12:41:47.984810 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) May 23 12:41:47.984810 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func May 23 12:41:47.984810 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self.get_or_create( May 23 12:41:47.984810 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create May 23 12:41:47.984810 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application with Lock( May 23 12:41:47.984810 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ May 23 12:41:47.984810 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self._enter() May 23 12:41:47.984810 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter May 23 12:41:47.984810 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) May 23 12:41:47.984810 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create May 23 12:41:47.984810 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self.creator() May 23 12:41:47.984810 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value May 23 12:41:47.984810 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application created_value = creator( May 23 12:41:47.986115 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain May 23 12:41:47.986115 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) May 23 12:41:47.986115 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. May 23 12:41:47.986115 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application May 23 12:41:47.987503 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 382/756] 10.222.0.30 () {56 vars in 1108 bytes} [Mon May 23 12:41:47 2022] GET /identity/v3/domains/Default => generated 87 bytes in 222 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) May 23 12:41:47.988445 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a03465c7-d8ae-45d0-9a90-3ff299aba86a None admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:41:48.008876 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15c2647a-c9ae-4d2e-9b21-b23f24613988 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:41:48.010247 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15c2647a-c9ae-4d2e-9b21-b23f24613988 None admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:41:48.010682 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15c2647a-c9ae-4d2e-9b21-b23f24613988 None admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:41:48.011382 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15c2647a-c9ae-4d2e-9b21-b23f24613988 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:41:48.026657 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15c2647a-c9ae-4d2e-9b21-b23f24613988 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:41:48.029561 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15c2647a-c9ae-4d2e-9b21-b23f24613988 None admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:41:48.040695 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=100 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:41:48.054695 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application [None req-15c2647a-c9ae-4d2e-9b21-b23f24613988 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. May 23 12:41:48.054695 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application Traceback (most recent call last): May 23 12:41:48.054695 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request May 23 12:41:48.054695 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application rv = self.dispatch_request() May 23 12:41:48.054695 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request May 23 12:41:48.054695 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) May 23 12:41:48.054695 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper May 23 12:41:48.054695 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) May 23 12:41:48.054695 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view May 23 12:41:48.054695 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) May 23 12:41:48.054695 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request May 23 12:41:48.054695 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) May 23 12:41:48.054695 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get May 23 12:41:48.054695 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self._get_domain(domain_id) May 23 12:41:48.054695 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain May 23 12:41:48.055840 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) May 23 12:41:48.055840 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped May 23 12:41:48.055840 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) May 23 12:41:48.055840 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun May 23 12:41:48.055840 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) May 23 12:41:48.055840 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func May 23 12:41:48.055840 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self.get_or_create( May 23 12:41:48.055840 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create May 23 12:41:48.055840 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application with Lock( May 23 12:41:48.055840 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ May 23 12:41:48.055840 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self._enter() May 23 12:41:48.055840 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter May 23 12:41:48.055840 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) May 23 12:41:48.055840 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create May 23 12:41:48.055840 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self.creator() May 23 12:41:48.055840 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value May 23 12:41:48.055840 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application created_value = creator( May 23 12:41:48.056956 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain May 23 12:41:48.056956 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) May 23 12:41:48.056956 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. May 23 12:41:48.056956 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application May 23 12:41:48.059475 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 375/757] 10.222.0.30 () {56 vars in 1108 bytes} [Mon May 23 12:41:47 2022] GET /identity/v3/domains/Default => generated 87 bytes in 240 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) May 23 12:41:48.060874 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15c2647a-c9ae-4d2e-9b21-b23f24613988 None admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:41:48.117498 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a03465c7-d8ae-45d0-9a90-3ff299aba86a None admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:41:48.164722 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75ae5a1b-d936-48f9-bb5e-bc60388a1f9f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:41:48.165452 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75ae5a1b-d936-48f9-bb5e-bc60388a1f9f None admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:41:48.165829 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75ae5a1b-d936-48f9-bb5e-bc60388a1f9f None admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:41:48.166223 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75ae5a1b-d936-48f9-bb5e-bc60388a1f9f None admin] PATH_INFO: `/v3/role_assignments` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:41:48.167146 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75ae5a1b-d936-48f9-bb5e-bc60388a1f9f None admin] RBAC: Adding query filter params (role.id=bc3c3c9d654940cbaa5177a6be23d1bc, scope.project.id=9c389ae65aaa4542bb5a5c4fd0bacf5b, user.id=138d91edb99f450e846765f9eef00d62) {{(pid=92056) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} May 23 12:41:48.167555 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75ae5a1b-d936-48f9-bb5e-bc60388a1f9f None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:41:48.169117 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75ae5a1b-d936-48f9-bb5e-bc60388a1f9f None admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:41:48.181918 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 383/758] 10.222.0.30 () {56 vars in 1374 bytes} [Mon May 23 12:41:47 2022] GET /identity/v3/role_assignments?user.id=138d91edb99f450e846765f9eef00d62&scope.project.id=9c389ae65aaa4542bb5a5c4fd0bacf5b&role.id=bc3c3c9d654940cbaa5177a6be23d1bc => generated 259 bytes in 187 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) May 23 12:41:48.181918 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75ae5a1b-d936-48f9-bb5e-bc60388a1f9f None admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:41:48.191586 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15c2647a-c9ae-4d2e-9b21-b23f24613988 None admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:41:48.263294 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88c42589-94a2-4e8e-80d8-899b7e5b823e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:41:48.263971 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88c42589-94a2-4e8e-80d8-899b7e5b823e None admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:41:48.264253 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88c42589-94a2-4e8e-80d8-899b7e5b823e None admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:41:48.264528 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88c42589-94a2-4e8e-80d8-899b7e5b823e None admin] PATH_INFO: `/v3/domains` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:41:48.265257 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-88c42589-94a2-4e8e-80d8-899b7e5b823e None admin] RBAC: Adding query filter params (name=Default) {{(pid=92057) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} May 23 12:41:48.265563 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-88c42589-94a2-4e8e-80d8-899b7e5b823e None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:41:48.268160 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-88c42589-94a2-4e8e-80d8-899b7e5b823e None admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:41:48.287790 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=100 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:41:48.294090 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 376/759] 10.222.0.30 () {56 vars in 1109 bytes} [Mon May 23 12:41:48 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 234 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) May 23 12:41:48.299340 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88c42589-94a2-4e8e-80d8-899b7e5b823e None admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:41:48.387043 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75ae5a1b-d936-48f9-bb5e-bc60388a1f9f None admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:41:48.415076 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=100 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:41:48.451820 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5077834-0840-457a-a688-246e746ee40c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:41:48.453132 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5077834-0840-457a-a688-246e746ee40c None admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:41:48.453621 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5077834-0840-457a-a688-246e746ee40c None admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:41:48.454278 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5077834-0840-457a-a688-246e746ee40c None admin] PATH_INFO: `/v3/domains` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:41:48.455233 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e5077834-0840-457a-a688-246e746ee40c None admin] RBAC: Adding query filter params (name=Default) {{(pid=92056) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} May 23 12:41:48.455740 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e5077834-0840-457a-a688-246e746ee40c None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:41:48.457746 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e5077834-0840-457a-a688-246e746ee40c None admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:41:48.475874 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 384/760] 10.222.0.30 () {56 vars in 1109 bytes} [Mon May 23 12:41:48 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 300 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) May 23 12:41:48.481680 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5077834-0840-457a-a688-246e746ee40c None admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:41:48.496647 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88c42589-94a2-4e8e-80d8-899b7e5b823e None admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:41:48.601481 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5077834-0840-457a-a688-246e746ee40c None admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:41:48.613559 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=100 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:41:48.627227 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f9996e2-ac1f-448d-9c0a-db1320543d9f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:41:48.635688 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f9996e2-ac1f-448d-9c0a-db1320543d9f None admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:41:48.635688 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f9996e2-ac1f-448d-9c0a-db1320543d9f None admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:41:48.635688 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f9996e2-ac1f-448d-9c0a-db1320543d9f None admin] PATH_INFO: `/v3/projects/service` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:41:48.642747 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36a4f16a-a8a8-4d6b-8c38-b4fd68941565 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:41:48.643847 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36a4f16a-a8a8-4d6b-8c38-b4fd68941565 None admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:41:48.644330 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36a4f16a-a8a8-4d6b-8c38-b4fd68941565 None admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:41:48.644817 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36a4f16a-a8a8-4d6b-8c38-b4fd68941565 None admin] PATH_INFO: `/v3/projects/service` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:41:48.654251 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36a4f16a-a8a8-4d6b-8c38-b4fd68941565 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:41:48.654734 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f9996e2-ac1f-448d-9c0a-db1320543d9f None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:41:48.656240 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f9996e2-ac1f-448d-9c0a-db1320543d9f None admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:41:48.656513 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36a4f16a-a8a8-4d6b-8c38-b4fd68941565 None admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:41:48.674382 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application [None req-36a4f16a-a8a8-4d6b-8c38-b4fd68941565 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. May 23 12:41:48.674382 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application Traceback (most recent call last): May 23 12:41:48.674382 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request May 23 12:41:48.674382 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application rv = self.dispatch_request() May 23 12:41:48.674382 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request May 23 12:41:48.674382 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) May 23 12:41:48.674382 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper May 23 12:41:48.674382 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) May 23 12:41:48.674382 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view May 23 12:41:48.674382 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) May 23 12:41:48.674382 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request May 23 12:41:48.674382 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) May 23 12:41:48.674382 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get May 23 12:41:48.674382 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self._get_project(project_id) May 23 12:41:48.674382 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project May 23 12:41:48.675344 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) May 23 12:41:48.675344 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped May 23 12:41:48.675344 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) May 23 12:41:48.675344 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun May 23 12:41:48.675344 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) May 23 12:41:48.675344 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func May 23 12:41:48.675344 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self.get_or_create( May 23 12:41:48.675344 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create May 23 12:41:48.675344 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application with Lock( May 23 12:41:48.675344 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ May 23 12:41:48.675344 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self._enter() May 23 12:41:48.675344 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter May 23 12:41:48.675344 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) May 23 12:41:48.675344 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create May 23 12:41:48.675344 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self.creator() May 23 12:41:48.675344 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value May 23 12:41:48.675344 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application created_value = creator( May 23 12:41:48.676296 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project May 23 12:41:48.676296 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) May 23 12:41:48.676296 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project May 23 12:41:48.676296 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() May 23 12:41:48.676296 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project May 23 12:41:48.676296 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) May 23 12:41:48.676296 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. May 23 12:41:48.676296 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application May 23 12:41:48.679915 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 385/761] 10.222.0.30 () {56 vars in 1111 bytes} [Mon May 23 12:41:48 2022] GET /identity/v3/projects/service => generated 88 bytes in 199 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) May 23 12:41:48.688633 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36a4f16a-a8a8-4d6b-8c38-b4fd68941565 None admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:41:48.692337 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application [None req-2f9996e2-ac1f-448d-9c0a-db1320543d9f None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. May 23 12:41:48.692337 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application Traceback (most recent call last): May 23 12:41:48.692337 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request May 23 12:41:48.692337 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application rv = self.dispatch_request() May 23 12:41:48.692337 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request May 23 12:41:48.692337 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) May 23 12:41:48.692337 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper May 23 12:41:48.692337 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) May 23 12:41:48.692337 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view May 23 12:41:48.692337 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) May 23 12:41:48.692337 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request May 23 12:41:48.692337 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) May 23 12:41:48.692337 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get May 23 12:41:48.692337 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self._get_project(project_id) May 23 12:41:48.692337 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project May 23 12:41:48.693775 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) May 23 12:41:48.693775 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped May 23 12:41:48.693775 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) May 23 12:41:48.693775 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun May 23 12:41:48.693775 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) May 23 12:41:48.693775 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func May 23 12:41:48.693775 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self.get_or_create( May 23 12:41:48.693775 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create May 23 12:41:48.693775 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application with Lock( May 23 12:41:48.693775 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ May 23 12:41:48.693775 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self._enter() May 23 12:41:48.693775 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter May 23 12:41:48.693775 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) May 23 12:41:48.693775 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create May 23 12:41:48.693775 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self.creator() May 23 12:41:48.693775 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value May 23 12:41:48.693775 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application created_value = creator( May 23 12:41:48.694726 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project May 23 12:41:48.694726 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) May 23 12:41:48.694726 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project May 23 12:41:48.694726 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() May 23 12:41:48.694726 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project May 23 12:41:48.694726 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) May 23 12:41:48.694726 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. May 23 12:41:48.694726 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application May 23 12:41:48.694726 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 377/762] 10.222.0.30 () {56 vars in 1111 bytes} [Mon May 23 12:41:48 2022] GET /identity/v3/projects/service => generated 88 bytes in 392 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) May 23 12:41:48.702890 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f9996e2-ac1f-448d-9c0a-db1320543d9f None admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:41:48.857108 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36a4f16a-a8a8-4d6b-8c38-b4fd68941565 None admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:41:48.895752 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eeb5a520-0eae-46ce-8577-9518f86a70ce None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:41:48.896862 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eeb5a520-0eae-46ce-8577-9518f86a70ce None admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:41:48.897374 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eeb5a520-0eae-46ce-8577-9518f86a70ce None admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:41:48.897851 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eeb5a520-0eae-46ce-8577-9518f86a70ce None admin] PATH_INFO: `/v3/projects` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:41:48.898921 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eeb5a520-0eae-46ce-8577-9518f86a70ce None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=92056) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} May 23 12:41:48.899454 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eeb5a520-0eae-46ce-8577-9518f86a70ce None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:41:48.901153 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eeb5a520-0eae-46ce-8577-9518f86a70ce None admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:41:48.916079 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 386/763] 10.222.0.30 () {56 vars in 1148 bytes} [Mon May 23 12:41:48 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 228 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) May 23 12:41:48.922174 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eeb5a520-0eae-46ce-8577-9518f86a70ce None admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:41:48.958243 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=100 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:41:49.040564 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eeb5a520-0eae-46ce-8577-9518f86a70ce None admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:41:49.062118 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f9996e2-ac1f-448d-9c0a-db1320543d9f None admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:41:49.082745 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c0ac8e3-df43-476a-a7ab-370fd0de6611 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:41:49.083699 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c0ac8e3-df43-476a-a7ab-370fd0de6611 None admin] REQUEST_METHOD: `PUT` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:41:49.083961 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c0ac8e3-df43-476a-a7ab-370fd0de6611 None admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:41:49.084232 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c0ac8e3-df43-476a-a7ab-370fd0de6611 None admin] PATH_INFO: `/v3/projects/9c389ae65aaa4542bb5a5c4fd0bacf5b/users/cbe9931d94d44cc7ace0e83a7e5015fc/roles/bc3c3c9d654940cbaa5177a6be23d1bc` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:41:49.086182 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c0ac8e3-df43-476a-a7ab-370fd0de6611 None admin] RBAC: Authorizing `identity:create_grant(project_id=9c389ae65aaa4542bb5a5c4fd0bacf5b, user_id=cbe9931d94d44cc7ace0e83a7e5015fc, role_id=bc3c3c9d654940cbaa5177a6be23d1bc)` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:41:49.087606 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c0ac8e3-df43-476a-a7ab-370fd0de6611 None admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:41:49.119248 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=100 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:41:49.136536 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08deb6ae-a6b9-4869-88e3-ce91cd869ede None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:41:49.137626 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08deb6ae-a6b9-4869-88e3-ce91cd869ede None admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:41:49.138158 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08deb6ae-a6b9-4869-88e3-ce91cd869ede None admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:41:49.138630 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08deb6ae-a6b9-4869-88e3-ce91cd869ede None admin] PATH_INFO: `/v3/projects` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:41:49.139711 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08deb6ae-a6b9-4869-88e3-ce91cd869ede None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=92057) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} May 23 12:41:49.140239 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08deb6ae-a6b9-4869-88e3-ce91cd869ede None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:41:49.142003 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08deb6ae-a6b9-4869-88e3-ce91cd869ede None admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:41:49.156722 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 378/764] 10.222.0.30 () {56 vars in 1148 bytes} [Mon May 23 12:41:48 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 454 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) May 23 12:41:49.158607 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38dd93e2-e7ad-4d84-a13e-bceb967835c9 None None] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:41:49.159057 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38dd93e2-e7ad-4d84-a13e-bceb967835c9 None None] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:41:49.159492 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38dd93e2-e7ad-4d84-a13e-bceb967835c9 None None] PATH_INFO: `/` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:41:49.161775 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 379/765] 10.222.0.30 () {54 vars in 926 bytes} [Mon May 23 12:41:49 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) May 23 12:41:49.164708 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38dd93e2-e7ad-4d84-a13e-bceb967835c9 None None] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:41:49.169343 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 387/766] 10.222.0.30 () {58 vars in 1439 bytes} [Mon May 23 12:41:48 2022] PUT /identity/v3/projects/9c389ae65aaa4542bb5a5c4fd0bacf5b/users/cbe9931d94d44cc7ace0e83a7e5015fc/roles/bc3c3c9d654940cbaa5177a6be23d1bc => generated 0 bytes in 247 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) May 23 12:41:49.172930 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3596510-e7fa-489b-91cd-7ba4aff75f66 None None] REQUEST_METHOD: `POST` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:41:49.173221 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3596510-e7fa-489b-91cd-7ba4aff75f66 None None] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:41:49.173549 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3596510-e7fa-489b-91cd-7ba4aff75f66 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:41:49.208649 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.auth.core [None req-e3596510-e7fa-489b-91cd-7ba4aff75f66 None None] MFA Rules not processed for user `0f7119f8fa5c4445977671140ded91a4`. Rule list: `[]` (Enabled: `True`). {{(pid=92056) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 12:41:49.210265 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.fernet_utils [None req-e3596510-e7fa-489b-91cd-7ba4aff75f66 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92056) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 12:41:49.299239 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38dd93e2-e7ad-4d84-a13e-bceb967835c9 None None] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:41:49.343172 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9bf7c8f-d8e4-416f-b19f-5268eff8c612 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:41:49.344284 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9bf7c8f-d8e4-416f-b19f-5268eff8c612 None admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:41:49.344757 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9bf7c8f-d8e4-416f-b19f-5268eff8c612 None admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:41:49.345411 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9bf7c8f-d8e4-416f-b19f-5268eff8c612 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:41:49.346481 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9bf7c8f-d8e4-416f-b19f-5268eff8c612 None admin] RBAC: Adding query filter params (role.id=41ab28c1ae3d4941bd09fb5d35f1dc54, scope.project.id=9c389ae65aaa4542bb5a5c4fd0bacf5b, user.id=6d21a56126fd4ccd837c33ccc08ab609) {{(pid=92057) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} May 23 12:41:49.347108 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9bf7c8f-d8e4-416f-b19f-5268eff8c612 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:41:49.348797 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9bf7c8f-d8e4-416f-b19f-5268eff8c612 None admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:41:49.357728 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 380/767] 10.222.0.30 () {56 vars in 1374 bytes} [Mon May 23 12:41:49 2022] GET /identity/v3/role_assignments?user.id=6d21a56126fd4ccd837c33ccc08ab609&scope.project.id=9c389ae65aaa4542bb5a5c4fd0bacf5b&role.id=41ab28c1ae3d4941bd09fb5d35f1dc54 => generated 259 bytes in 193 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) May 23 12:41:49.365932 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34c75a2c-cf80-4743-acc1-6a3c0afc464f None None] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:41:49.366335 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34c75a2c-cf80-4743-acc1-6a3c0afc464f None None] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:41:49.366526 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34c75a2c-cf80-4743-acc1-6a3c0afc464f None None] PATH_INFO: `/` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:41:49.368134 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 381/768] 10.222.0.30 () {54 vars in 926 bytes} [Mon May 23 12:41:49 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) May 23 12:41:49.374097 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-403c9b69-af47-4b8e-b116-1eea7866d843 None None] REQUEST_METHOD: `POST` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:41:49.374457 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-403c9b69-af47-4b8e-b116-1eea7866d843 None None] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:41:49.374649 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-403c9b69-af47-4b8e-b116-1eea7866d843 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:41:49.378959 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 388/769] 10.222.0.30 () {58 vars in 1023 bytes} [Mon May 23 12:41:49 2022] POST /identity/v3/auth/tokens => generated 1002 bytes in 207 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) May 23 12:41:49.405730 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3596510-e7fa-489b-91cd-7ba4aff75f66 None None] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:41:49.418261 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.auth.core [None req-403c9b69-af47-4b8e-b116-1eea7866d843 None None] MFA Rules not processed for user `0f7119f8fa5c4445977671140ded91a4`. Rule list: `[]` (Enabled: `True`). {{(pid=92057) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 12:41:49.419482 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.fernet_utils [None req-403c9b69-af47-4b8e-b116-1eea7866d843 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92057) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 12:41:49.448627 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:41:49.496090 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=100 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:41:49.530625 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 382/770] 10.222.0.30 () {58 vars in 1023 bytes} [Mon May 23 12:41:49 2022] POST /identity/v3/auth/tokens => generated 1002 bytes in 157 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) May 23 12:41:49.546583 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3596510-e7fa-489b-91cd-7ba4aff75f66 None None] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:41:49.556189 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-403c9b69-af47-4b8e-b116-1eea7866d843 None None] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:41:49.587123 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-594375af-b9fa-4609-a989-1edcac0edaaf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:41:49.587332 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-594375af-b9fa-4609-a989-1edcac0edaaf None admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:41:49.587645 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-594375af-b9fa-4609-a989-1edcac0edaaf None admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:41:49.587987 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-594375af-b9fa-4609-a989-1edcac0edaaf None admin] PATH_INFO: `/v3/services/network` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:41:49.588732 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-594375af-b9fa-4609-a989-1edcac0edaaf None admin] RBAC: Authorizing `identity:get_service(service_id=network)` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:41:49.590842 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-594375af-b9fa-4609-a989-1edcac0edaaf None admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:41:49.606018 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application [None req-594375af-b9fa-4609-a989-1edcac0edaaf None admin] Could not find service: network.: keystone.exception.ServiceNotFound: Could not find service: network. May 23 12:41:49.606018 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application Traceback (most recent call last): May 23 12:41:49.606018 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service May 23 12:41:49.606018 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) May 23 12:41:49.606018 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service May 23 12:41:49.606018 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() May 23 12:41:49.606018 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service May 23 12:41:49.606018 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) May 23 12:41:49.606018 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: network. May 23 12:41:49.606018 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application May 23 12:41:49.606018 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: May 23 12:41:49.606018 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application May 23 12:41:49.606018 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application Traceback (most recent call last): May 23 12:41:49.606018 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request May 23 12:41:49.606018 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application rv = self.dispatch_request() May 23 12:41:49.606018 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request May 23 12:41:49.606018 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) May 23 12:41:49.607081 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper May 23 12:41:49.607081 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) May 23 12:41:49.607081 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view May 23 12:41:49.607081 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) May 23 12:41:49.607081 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request May 23 12:41:49.607081 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) May 23 12:41:49.607081 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get May 23 12:41:49.607081 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self._get_service(service_id) May 23 12:41:49.607081 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service May 23 12:41:49.607081 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) May 23 12:41:49.607081 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped May 23 12:41:49.607081 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) May 23 12:41:49.607081 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun May 23 12:41:49.607081 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) May 23 12:41:49.607081 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func May 23 12:41:49.607081 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self.get_or_create( May 23 12:41:49.608392 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create May 23 12:41:49.608392 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application with Lock( May 23 12:41:49.608392 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ May 23 12:41:49.608392 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self._enter() May 23 12:41:49.608392 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter May 23 12:41:49.608392 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) May 23 12:41:49.608392 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create May 23 12:41:49.608392 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self.creator() May 23 12:41:49.608392 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value May 23 12:41:49.608392 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application created_value = creator( May 23 12:41:49.608392 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service May 23 12:41:49.608392 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) May 23 12:41:49.608392 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: network. May 23 12:41:49.608392 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application May 23 12:41:49.608392 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 389/771] 10.222.0.30 () {56 vars in 1111 bytes} [Mon May 23 12:41:49 2022] GET /identity/v3/services/network => generated 88 bytes in 203 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) May 23 12:41:49.613612 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-594375af-b9fa-4609-a989-1edcac0edaaf None admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:41:49.671010 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-403c9b69-af47-4b8e-b116-1eea7866d843 None None] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:41:49.707353 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-006d2e93-5048-4101-be31-79d9db0e5200 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:41:49.708235 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-006d2e93-5048-4101-be31-79d9db0e5200 None admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:41:49.708486 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-006d2e93-5048-4101-be31-79d9db0e5200 None admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:41:49.708764 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-006d2e93-5048-4101-be31-79d9db0e5200 None admin] PATH_INFO: `/v3/services/block-storage` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:41:49.709424 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-006d2e93-5048-4101-be31-79d9db0e5200 None admin] RBAC: Authorizing `identity:get_service(service_id=block-storage)` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:41:49.711148 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-006d2e93-5048-4101-be31-79d9db0e5200 None admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:41:49.723675 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application [None req-006d2e93-5048-4101-be31-79d9db0e5200 None admin] Could not find service: block-storage.: keystone.exception.ServiceNotFound: Could not find service: block-storage. May 23 12:41:49.723675 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application Traceback (most recent call last): May 23 12:41:49.723675 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service May 23 12:41:49.723675 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) May 23 12:41:49.723675 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service May 23 12:41:49.723675 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() May 23 12:41:49.723675 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service May 23 12:41:49.723675 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) May 23 12:41:49.723675 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: block-storage. May 23 12:41:49.723675 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application May 23 12:41:49.723675 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: May 23 12:41:49.723675 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application May 23 12:41:49.723675 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application Traceback (most recent call last): May 23 12:41:49.723675 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request May 23 12:41:49.723675 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application rv = self.dispatch_request() May 23 12:41:49.723675 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request May 23 12:41:49.725358 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) May 23 12:41:49.725358 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper May 23 12:41:49.725358 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) May 23 12:41:49.725358 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view May 23 12:41:49.725358 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) May 23 12:41:49.725358 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request May 23 12:41:49.725358 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) May 23 12:41:49.725358 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get May 23 12:41:49.725358 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self._get_service(service_id) May 23 12:41:49.725358 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service May 23 12:41:49.725358 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) May 23 12:41:49.725358 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped May 23 12:41:49.725358 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) May 23 12:41:49.725358 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun May 23 12:41:49.725358 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) May 23 12:41:49.725358 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func May 23 12:41:49.726256 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self.get_or_create( May 23 12:41:49.726256 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create May 23 12:41:49.726256 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application with Lock( May 23 12:41:49.726256 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ May 23 12:41:49.726256 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self._enter() May 23 12:41:49.726256 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter May 23 12:41:49.726256 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) May 23 12:41:49.726256 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create May 23 12:41:49.726256 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self.creator() May 23 12:41:49.726256 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value May 23 12:41:49.726256 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application created_value = creator( May 23 12:41:49.726256 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service May 23 12:41:49.726256 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) May 23 12:41:49.726256 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: block-storage. May 23 12:41:49.726256 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application May 23 12:41:49.726256 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 383/772] 10.222.0.30 () {56 vars in 1129 bytes} [Mon May 23 12:41:49 2022] GET /identity/v3/services/block-storage => generated 94 bytes in 170 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) May 23 12:41:49.730745 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=100 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:41:49.731600 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-006d2e93-5048-4101-be31-79d9db0e5200 None admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:41:49.742174 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-594375af-b9fa-4609-a989-1edcac0edaaf None admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:41:49.774422 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=100 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:41:49.825796 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-736fb954-4c95-4a8a-aba9-fdcd0a734d7f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:41:49.826562 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-736fb954-4c95-4a8a-aba9-fdcd0a734d7f None admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:41:49.826896 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-736fb954-4c95-4a8a-aba9-fdcd0a734d7f None admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:41:49.827114 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-736fb954-4c95-4a8a-aba9-fdcd0a734d7f None admin] PATH_INFO: `/v3/services` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:41:49.827821 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-736fb954-4c95-4a8a-aba9-fdcd0a734d7f None admin] RBAC: Adding query filter params (name=network) {{(pid=92056) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} May 23 12:41:49.828127 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-736fb954-4c95-4a8a-aba9-fdcd0a734d7f None admin] RBAC: Authorizing `identity:list_services()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:41:49.829492 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-736fb954-4c95-4a8a-aba9-fdcd0a734d7f None admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:41:49.839634 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-006d2e93-5048-4101-be31-79d9db0e5200 None admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:41:49.845016 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 390/773] 10.222.0.30 () {56 vars in 1112 bytes} [Mon May 23 12:41:49 2022] GET /identity/v3/services?name=network => generated 124 bytes in 232 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) May 23 12:41:49.854921 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-736fb954-4c95-4a8a-aba9-fdcd0a734d7f None admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:41:49.880794 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c29d096-2e0e-47e0-b52e-2dc0fc2439a2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:41:49.881216 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c29d096-2e0e-47e0-b52e-2dc0fc2439a2 None admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:41:49.881675 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c29d096-2e0e-47e0-b52e-2dc0fc2439a2 None admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:41:49.881978 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c29d096-2e0e-47e0-b52e-2dc0fc2439a2 None admin] PATH_INFO: `/v3/services` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:41:49.882689 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c29d096-2e0e-47e0-b52e-2dc0fc2439a2 None admin] RBAC: Adding query filter params (name=block-storage) {{(pid=92057) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} May 23 12:41:49.883007 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c29d096-2e0e-47e0-b52e-2dc0fc2439a2 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:41:49.885176 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c29d096-2e0e-47e0-b52e-2dc0fc2439a2 None admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:41:49.903666 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 384/774] 10.222.0.30 () {56 vars in 1124 bytes} [Mon May 23 12:41:49 2022] GET /identity/v3/services?name=block-storage => generated 130 bytes in 171 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) May 23 12:41:49.906342 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c29d096-2e0e-47e0-b52e-2dc0fc2439a2 None admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:41:50.047152 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-736fb954-4c95-4a8a-aba9-fdcd0a734d7f None admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:41:50.087746 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7492c621-b982-41e6-8e18-b6c018ae7251 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:41:50.088838 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7492c621-b982-41e6-8e18-b6c018ae7251 None admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:41:50.089130 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7492c621-b982-41e6-8e18-b6c018ae7251 None admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:41:50.089411 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7492c621-b982-41e6-8e18-b6c018ae7251 None admin] PATH_INFO: `/v3/services` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:41:50.090160 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7492c621-b982-41e6-8e18-b6c018ae7251 None admin] RBAC: Adding query filter params (type=network) {{(pid=92056) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} May 23 12:41:50.090470 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7492c621-b982-41e6-8e18-b6c018ae7251 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:41:50.091848 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7492c621-b982-41e6-8e18-b6c018ae7251 None admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:41:50.098700 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 391/775] 10.222.0.30 () {56 vars in 1112 bytes} [Mon May 23 12:41:49 2022] GET /identity/v3/services?type=network => generated 350 bytes in 244 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) May 23 12:41:50.113203 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7492c621-b982-41e6-8e18-b6c018ae7251 None admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:41:50.128901 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c29d096-2e0e-47e0-b52e-2dc0fc2439a2 None admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:41:50.129488 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=100 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:41:50.131494 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=100 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:41:50.208843 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f323c9fe-51e0-4481-bef3-366ea525de5a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:41:50.209684 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f323c9fe-51e0-4481-bef3-366ea525de5a None admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:41:50.210093 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f323c9fe-51e0-4481-bef3-366ea525de5a None admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:41:50.210678 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f323c9fe-51e0-4481-bef3-366ea525de5a None admin] PATH_INFO: `/v3/services` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:41:50.211080 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f323c9fe-51e0-4481-bef3-366ea525de5a None admin] RBAC: Adding query filter params (type=block-storage) {{(pid=92057) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} May 23 12:41:50.212055 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f323c9fe-51e0-4481-bef3-366ea525de5a None admin] RBAC: Authorizing `identity:list_services()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:41:50.212743 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f323c9fe-51e0-4481-bef3-366ea525de5a None admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:41:50.218035 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 385/776] 10.222.0.30 () {56 vars in 1124 bytes} [Mon May 23 12:41:49 2022] GET /identity/v3/services?type=block-storage => generated 367 bytes in 312 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) May 23 12:41:50.226280 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7492c621-b982-41e6-8e18-b6c018ae7251 None admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:41:50.235788 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f323c9fe-51e0-4481-bef3-366ea525de5a None admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:41:50.263659 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b9b2977-d7b1-468d-808a-e43b4b58cebb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:41:50.264380 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b9b2977-d7b1-468d-808a-e43b4b58cebb None admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:41:50.264676 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b9b2977-d7b1-468d-808a-e43b4b58cebb None admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:41:50.264958 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b9b2977-d7b1-468d-808a-e43b4b58cebb None admin] PATH_INFO: `/v3/endpoints` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:41:50.265763 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b9b2977-d7b1-468d-808a-e43b4b58cebb None admin] RBAC: Adding query filter params (interface=public, service_id=65044bd5de014257b47a32f6b4c93376, region_id=RegionOne) {{(pid=92056) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} May 23 12:41:50.266110 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b9b2977-d7b1-468d-808a-e43b4b58cebb None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:41:50.267629 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b9b2977-d7b1-468d-808a-e43b4b58cebb None admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:41:50.276701 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 392/777] 10.222.0.30 () {56 vars in 1251 bytes} [Mon May 23 12:41:50 2022] GET /identity/v3/endpoints?service_id=65044bd5de014257b47a32f6b4c93376&interface=public®ion_id=RegionOne => generated 194 bytes in 164 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) May 23 12:41:50.282883 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b9b2977-d7b1-468d-808a-e43b4b58cebb None admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:41:50.394833 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f323c9fe-51e0-4481-bef3-366ea525de5a None admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:41:50.438254 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=100 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:41:50.459663 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b9b2977-d7b1-468d-808a-e43b4b58cebb None admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:41:50.474526 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be253e07-9620-477d-a1a8-c9bed935974b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:41:50.476615 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be253e07-9620-477d-a1a8-c9bed935974b None admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:41:50.476899 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be253e07-9620-477d-a1a8-c9bed935974b None admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:41:50.477172 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be253e07-9620-477d-a1a8-c9bed935974b None admin] PATH_INFO: `/v3/endpoints` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:41:50.477975 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be253e07-9620-477d-a1a8-c9bed935974b None admin] RBAC: Adding query filter params (interface=public, service_id=85d4721ccd6149f2a5aa7191f0cc4803, region_id=RegionOne) {{(pid=92057) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} May 23 12:41:50.478287 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be253e07-9620-477d-a1a8-c9bed935974b None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:41:50.479658 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be253e07-9620-477d-a1a8-c9bed935974b None admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:41:50.500993 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 386/778] 10.222.0.30 () {56 vars in 1251 bytes} [Mon May 23 12:41:50 2022] GET /identity/v3/endpoints?service_id=85d4721ccd6149f2a5aa7191f0cc4803&interface=public®ion_id=RegionOne => generated 194 bytes in 265 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) May 23 12:41:50.506349 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be253e07-9620-477d-a1a8-c9bed935974b None admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:41:50.521994 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e210da9-4191-479c-9112-016930068e50 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:41:50.522661 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e210da9-4191-479c-9112-016930068e50 None admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:41:50.523198 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e210da9-4191-479c-9112-016930068e50 None admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:41:50.523471 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e210da9-4191-479c-9112-016930068e50 None admin] PATH_INFO: `/v3/services` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:41:50.525821 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e210da9-4191-479c-9112-016930068e50 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:41:50.527730 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e210da9-4191-479c-9112-016930068e50 None admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:41:50.537567 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 393/779] 10.222.0.30 () {56 vars in 1087 bytes} [Mon May 23 12:41:50 2022] GET /identity/v3/services => generated 772 bytes in 255 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) May 23 12:41:50.554105 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=100 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:41:50.655314 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be253e07-9620-477d-a1a8-c9bed935974b None admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:41:50.713547 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa5529f0-27f7-459f-a4c3-58ba961f897d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:41:50.714259 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa5529f0-27f7-459f-a4c3-58ba961f897d None admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:41:50.714543 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa5529f0-27f7-459f-a4c3-58ba961f897d None admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:41:50.714821 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa5529f0-27f7-459f-a4c3-58ba961f897d None admin] PATH_INFO: `/v3/services` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:41:50.715508 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa5529f0-27f7-459f-a4c3-58ba961f897d None admin] RBAC: Authorizing `identity:list_services()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:41:50.716909 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa5529f0-27f7-459f-a4c3-58ba961f897d None admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:41:50.726793 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 387/780] 10.222.0.30 () {56 vars in 1087 bytes} [Mon May 23 12:41:50 2022] GET /identity/v3/services => generated 772 bytes in 222 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) May 23 12:41:50.861850 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3fa972a-e45e-4018-9add-78b766144e96 None None] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:41:50.862243 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3fa972a-e45e-4018-9add-78b766144e96 None None] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:41:50.862453 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3fa972a-e45e-4018-9add-78b766144e96 None None] PATH_INFO: `/` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:41:50.864323 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 394/781] 10.222.0.30 () {54 vars in 926 bytes} [Mon May 23 12:41:50 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) May 23 12:41:50.870448 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8dae0fbe-a8bc-44df-afd8-5aeffce24223 None None] REQUEST_METHOD: `POST` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:41:50.870759 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8dae0fbe-a8bc-44df-afd8-5aeffce24223 None None] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:41:50.871009 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8dae0fbe-a8bc-44df-afd8-5aeffce24223 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:41:50.893403 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.auth.core [None req-8dae0fbe-a8bc-44df-afd8-5aeffce24223 None None] MFA Rules not processed for user `0f7119f8fa5c4445977671140ded91a4`. Rule list: `[]` (Enabled: `True`). {{(pid=92057) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 12:41:50.894648 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.fernet_utils [None req-8dae0fbe-a8bc-44df-afd8-5aeffce24223 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92057) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 12:41:50.981314 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 388/782] 10.222.0.30 () {58 vars in 1023 bytes} [Mon May 23 12:41:50 2022] POST /identity/v3/auth/tokens => generated 1002 bytes in 112 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) May 23 12:41:51.008765 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3fa972a-e45e-4018-9add-78b766144e96 None None] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:41:51.132124 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3fa972a-e45e-4018-9add-78b766144e96 None None] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:41:51.177627 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-910b065f-f385-469a-bb0b-554980ae9ab0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:41:51.178464 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-910b065f-f385-469a-bb0b-554980ae9ab0 None admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:41:51.178741 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-910b065f-f385-469a-bb0b-554980ae9ab0 None admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:41:51.179009 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-910b065f-f385-469a-bb0b-554980ae9ab0 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:41:51.190530 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=100 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:41:51.194950 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-910b065f-f385-469a-bb0b-554980ae9ab0 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} May 23 12:41:51.202652 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application [None req-910b065f-f385-469a-bb0b-554980ae9ab0 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. May 23 12:41:51.202652 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application Traceback (most recent call last): May 23 12:41:51.202652 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request May 23 12:41:51.202652 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application rv = self.dispatch_request() May 23 12:41:51.202652 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request May 23 12:41:51.202652 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) May 23 12:41:51.202652 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper May 23 12:41:51.202652 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) May 23 12:41:51.202652 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view May 23 12:41:51.202652 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) May 23 12:41:51.202652 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request May 23 12:41:51.202652 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) May 23 12:41:51.202652 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get May 23 12:41:51.202652 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self._get_role(role_id) May 23 12:41:51.202652 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role May 23 12:41:51.202652 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') May 23 12:41:51.204830 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call May 23 12:41:51.204830 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( May 23 12:41:51.204830 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data May 23 12:41:51.204830 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) May 23 12:41:51.204830 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped May 23 12:41:51.204830 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) May 23 12:41:51.204830 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun May 23 12:41:51.204830 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) May 23 12:41:51.204830 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func May 23 12:41:51.204830 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self.get_or_create( May 23 12:41:51.204830 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create May 23 12:41:51.204830 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application with Lock( May 23 12:41:51.204830 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ May 23 12:41:51.204830 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self._enter() May 23 12:41:51.204830 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter May 23 12:41:51.204830 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) May 23 12:41:51.210839 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create May 23 12:41:51.210839 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self.creator() May 23 12:41:51.210839 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value May 23 12:41:51.210839 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application created_value = creator( May 23 12:41:51.210839 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role May 23 12:41:51.210839 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) May 23 12:41:51.210839 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role May 23 12:41:51.210839 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() May 23 12:41:51.210839 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role May 23 12:41:51.210839 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) May 23 12:41:51.210839 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. May 23 12:41:51.210839 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application May 23 12:41:51.210839 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 395/783] 10.222.0.30 () {56 vars in 1096 bytes} [Mon May 23 12:41:51 2022] GET /identity/v3/roles/admin => generated 83 bytes in 196 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) May 23 12:41:51.212545 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8dae0fbe-a8bc-44df-afd8-5aeffce24223 None None] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:41:51.260084 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=100 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:41:51.341759 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8dae0fbe-a8bc-44df-afd8-5aeffce24223 None None] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:41:51.390804 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2431280-dd38-4879-937d-8b0eeb942229 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:41:51.391841 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2431280-dd38-4879-937d-8b0eeb942229 None admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:41:51.392330 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2431280-dd38-4879-937d-8b0eeb942229 None admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:41:51.392806 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2431280-dd38-4879-937d-8b0eeb942229 None admin] PATH_INFO: `/v3/roles` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:41:51.393749 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2431280-dd38-4879-937d-8b0eeb942229 None admin] RBAC: Adding query filter params (name=admin) {{(pid=92057) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} May 23 12:41:51.394315 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2431280-dd38-4879-937d-8b0eeb942229 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:41:51.396019 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2431280-dd38-4879-937d-8b0eeb942229 None admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:41:51.409782 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 389/784] 10.222.0.30 () {56 vars in 1099 bytes} [Mon May 23 12:41:51 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 200 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) May 23 12:41:51.415534 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-910b065f-f385-469a-bb0b-554980ae9ab0 None admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:41:51.564761 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-910b065f-f385-469a-bb0b-554980ae9ab0 None admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:41:51.613358 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15c2f1d9-6651-41e0-8949-07bd8ef0ae7b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:41:51.614186 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15c2f1d9-6651-41e0-8949-07bd8ef0ae7b None admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:41:51.615259 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15c2f1d9-6651-41e0-8949-07bd8ef0ae7b None admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:41:51.615543 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15c2f1d9-6651-41e0-8949-07bd8ef0ae7b None admin] PATH_INFO: `/v3/domains/Default` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:41:51.630928 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15c2f1d9-6651-41e0-8949-07bd8ef0ae7b None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:41:51.632687 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15c2f1d9-6651-41e0-8949-07bd8ef0ae7b None admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:41:51.657704 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application [None req-15c2f1d9-6651-41e0-8949-07bd8ef0ae7b None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. May 23 12:41:51.657704 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application Traceback (most recent call last): May 23 12:41:51.657704 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request May 23 12:41:51.657704 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application rv = self.dispatch_request() May 23 12:41:51.657704 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request May 23 12:41:51.657704 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) May 23 12:41:51.657704 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper May 23 12:41:51.657704 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) May 23 12:41:51.657704 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view May 23 12:41:51.657704 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) May 23 12:41:51.657704 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request May 23 12:41:51.657704 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) May 23 12:41:51.657704 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get May 23 12:41:51.657704 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self._get_domain(domain_id) May 23 12:41:51.657704 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain May 23 12:41:51.658714 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) May 23 12:41:51.658714 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped May 23 12:41:51.658714 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) May 23 12:41:51.658714 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun May 23 12:41:51.658714 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) May 23 12:41:51.658714 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func May 23 12:41:51.658714 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self.get_or_create( May 23 12:41:51.658714 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create May 23 12:41:51.658714 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application with Lock( May 23 12:41:51.658714 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ May 23 12:41:51.658714 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self._enter() May 23 12:41:51.658714 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter May 23 12:41:51.658714 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) May 23 12:41:51.658714 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create May 23 12:41:51.658714 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self.creator() May 23 12:41:51.658714 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value May 23 12:41:51.658714 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application created_value = creator( May 23 12:41:51.659644 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain May 23 12:41:51.659644 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) May 23 12:41:51.659644 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. May 23 12:41:51.659644 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application May 23 12:41:51.662510 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 396/785] 10.222.0.30 () {56 vars in 1108 bytes} [Mon May 23 12:41:51 2022] GET /identity/v3/domains/Default => generated 87 bytes in 247 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) May 23 12:41:51.667899 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2431280-dd38-4879-937d-8b0eeb942229 None admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:41:51.806674 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=100 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:41:51.813483 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2431280-dd38-4879-937d-8b0eeb942229 None admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:41:51.848755 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f98f31a-ad56-4d02-b5cc-6a0080a76532 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:41:51.849323 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f98f31a-ad56-4d02-b5cc-6a0080a76532 None admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:41:51.849612 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f98f31a-ad56-4d02-b5cc-6a0080a76532 None admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:41:51.849906 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f98f31a-ad56-4d02-b5cc-6a0080a76532 None admin] PATH_INFO: `/v3/domains` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:41:51.850695 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f98f31a-ad56-4d02-b5cc-6a0080a76532 None admin] RBAC: Adding query filter params (name=Default) {{(pid=92057) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} May 23 12:41:51.851014 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f98f31a-ad56-4d02-b5cc-6a0080a76532 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:41:51.852399 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f98f31a-ad56-4d02-b5cc-6a0080a76532 None admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:41:51.865277 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 390/786] 10.222.0.30 () {56 vars in 1109 bytes} [Mon May 23 12:41:51 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 198 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) May 23 12:41:51.870754 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15c2f1d9-6651-41e0-8949-07bd8ef0ae7b None admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:41:51.947108 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=100 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:41:52.009871 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15c2f1d9-6651-41e0-8949-07bd8ef0ae7b None admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:41:52.051009 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2266e91c-4fee-40b2-8a30-23c812680b42 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:41:52.052336 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2266e91c-4fee-40b2-8a30-23c812680b42 None admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:41:52.052664 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2266e91c-4fee-40b2-8a30-23c812680b42 None admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:41:52.052981 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2266e91c-4fee-40b2-8a30-23c812680b42 None admin] PATH_INFO: `/v3/users/swift` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:41:52.062254 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2266e91c-4fee-40b2-8a30-23c812680b42 None admin] RBAC: Authorizing `identity:get_user(user_id=swift)` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:41:52.063752 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2266e91c-4fee-40b2-8a30-23c812680b42 None admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:41:52.081051 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application [None req-2266e91c-4fee-40b2-8a30-23c812680b42 None admin] Could not find user: swift.: keystone.exception.UserNotFound: Could not find user: swift. May 23 12:41:52.081051 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application Traceback (most recent call last): May 23 12:41:52.081051 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request May 23 12:41:52.081051 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application rv = self.dispatch_request() May 23 12:41:52.081051 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request May 23 12:41:52.081051 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) May 23 12:41:52.081051 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper May 23 12:41:52.081051 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) May 23 12:41:52.081051 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view May 23 12:41:52.081051 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) May 23 12:41:52.081051 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request May 23 12:41:52.081051 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) May 23 12:41:52.081051 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get May 23 12:41:52.081051 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self._get_user(user_id) May 23 12:41:52.081051 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user May 23 12:41:52.081051 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) May 23 12:41:52.082124 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped May 23 12:41:52.082124 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) May 23 12:41:52.082124 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper May 23 12:41:52.082124 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) May 23 12:41:52.082124 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper May 23 12:41:52.082124 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) May 23 12:41:52.082124 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun May 23 12:41:52.082124 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) May 23 12:41:52.082124 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func May 23 12:41:52.082124 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self.get_or_create( May 23 12:41:52.082124 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create May 23 12:41:52.082124 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application with Lock( May 23 12:41:52.082124 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ May 23 12:41:52.082124 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self._enter() May 23 12:41:52.082124 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter May 23 12:41:52.082124 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) May 23 12:41:52.082124 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create May 23 12:41:52.082124 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self.creator() May 23 12:41:52.083737 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value May 23 12:41:52.083737 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application created_value = creator( May 23 12:41:52.083737 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user May 23 12:41:52.083737 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) May 23 12:41:52.083737 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user May 23 12:41:52.083737 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) May 23 12:41:52.083737 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user May 23 12:41:52.083737 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) May 23 12:41:52.083737 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: swift. May 23 12:41:52.083737 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application May 23 12:41:52.083737 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 397/787] 10.222.0.30 () {56 vars in 1096 bytes} [Mon May 23 12:41:51 2022] GET /identity/v3/users/swift => generated 83 bytes in 212 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) May 23 12:41:52.086846 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96f3bd38-cd3e-4465-8058-6b4ebe5b7191 None None] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:41:52.088885 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96f3bd38-cd3e-4465-8058-6b4ebe5b7191 None None] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:41:52.088885 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96f3bd38-cd3e-4465-8058-6b4ebe5b7191 None None] PATH_INFO: `/` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:41:52.089229 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 391/788] 10.222.0.30 () {54 vars in 926 bytes} [Mon May 23 12:41:52 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) May 23 12:41:52.090221 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2266e91c-4fee-40b2-8a30-23c812680b42 None admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:41:52.095180 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64d3d7e1-ee2d-4754-87d1-fefdbaa32e32 None None] REQUEST_METHOD: `POST` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:41:52.095467 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64d3d7e1-ee2d-4754-87d1-fefdbaa32e32 None None] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:41:52.095745 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64d3d7e1-ee2d-4754-87d1-fefdbaa32e32 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:41:52.127109 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.auth.core [None req-64d3d7e1-ee2d-4754-87d1-fefdbaa32e32 None None] MFA Rules not processed for user `0f7119f8fa5c4445977671140ded91a4`. Rule list: `[]` (Enabled: `True`). {{(pid=92057) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 12:41:52.128277 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.fernet_utils [None req-64d3d7e1-ee2d-4754-87d1-fefdbaa32e32 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92057) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 12:41:52.224139 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2266e91c-4fee-40b2-8a30-23c812680b42 None admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:41:52.238246 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=100 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:41:52.271197 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c636773-8542-4079-b6f9-dd8d3816d899 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:41:52.272020 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c636773-8542-4079-b6f9-dd8d3816d899 None admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:41:52.272252 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c636773-8542-4079-b6f9-dd8d3816d899 None admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:41:52.272355 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 392/789] 10.222.0.30 () {58 vars in 1023 bytes} [Mon May 23 12:41:52 2022] POST /identity/v3/auth/tokens => generated 1002 bytes in 178 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) May 23 12:41:52.272535 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c636773-8542-4079-b6f9-dd8d3816d899 None admin] PATH_INFO: `/v3/users` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:41:52.273913 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c636773-8542-4079-b6f9-dd8d3816d899 None admin] RBAC: Adding query filter params (domain_id=default, name=swift) {{(pid=92056) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} May 23 12:41:52.273913 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c636773-8542-4079-b6f9-dd8d3816d899 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:41:52.277013 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c636773-8542-4079-b6f9-dd8d3816d899 None admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:41:52.286291 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-302bf24d-2c43-46c5-a786-3f22e097dbaf None None] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:41:52.286580 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-302bf24d-2c43-46c5-a786-3f22e097dbaf None None] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:41:52.286896 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-302bf24d-2c43-46c5-a786-3f22e097dbaf None None] PATH_INFO: `/` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:41:52.288537 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 393/790] 10.222.0.30 () {54 vars in 926 bytes} [Mon May 23 12:41:52 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) May 23 12:41:52.293321 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ee39ac3-c59b-4ac0-9893-c5eab154e1df None None] REQUEST_METHOD: `POST` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:41:52.293607 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ee39ac3-c59b-4ac0-9893-c5eab154e1df None None] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:41:52.293906 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ee39ac3-c59b-4ac0-9893-c5eab154e1df None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:41:52.313448 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 398/791] 10.222.0.30 () {56 vars in 1135 bytes} [Mon May 23 12:41:52 2022] GET /identity/v3/users?domain_id=default&name=swift => generated 370 bytes in 224 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) May 23 12:41:52.313971 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.auth.core [None req-6ee39ac3-c59b-4ac0-9893-c5eab154e1df None None] MFA Rules not processed for user `0f7119f8fa5c4445977671140ded91a4`. Rule list: `[]` (Enabled: `True`). {{(pid=92057) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 12:41:52.314303 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c636773-8542-4079-b6f9-dd8d3816d899 None admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:41:52.315163 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.fernet_utils [None req-6ee39ac3-c59b-4ac0-9893-c5eab154e1df None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92057) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 12:41:52.398045 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 394/792] 10.222.0.30 () {58 vars in 1023 bytes} [Mon May 23 12:41:52 2022] POST /identity/v3/auth/tokens => generated 1002 bytes in 105 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) May 23 12:41:52.398940 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ee39ac3-c59b-4ac0-9893-c5eab154e1df None None] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:41:52.407489 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=100 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:41:52.428933 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c636773-8542-4079-b6f9-dd8d3816d899 None admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:41:52.465119 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-753cc688-29b8-420a-93cd-44f1495b7939 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:41:52.466282 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-753cc688-29b8-420a-93cd-44f1495b7939 None admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:41:52.466604 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-753cc688-29b8-420a-93cd-44f1495b7939 None admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:41:52.466979 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-753cc688-29b8-420a-93cd-44f1495b7939 None admin] PATH_INFO: `/v3/roles/service` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:41:52.479563 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-753cc688-29b8-420a-93cd-44f1495b7939 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} May 23 12:41:52.487021 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application [None req-753cc688-29b8-420a-93cd-44f1495b7939 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. May 23 12:41:52.487021 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application Traceback (most recent call last): May 23 12:41:52.487021 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request May 23 12:41:52.487021 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application rv = self.dispatch_request() May 23 12:41:52.487021 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request May 23 12:41:52.487021 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) May 23 12:41:52.487021 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper May 23 12:41:52.487021 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) May 23 12:41:52.487021 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view May 23 12:41:52.487021 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) May 23 12:41:52.487021 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request May 23 12:41:52.487021 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) May 23 12:41:52.487021 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get May 23 12:41:52.487021 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self._get_role(role_id) May 23 12:41:52.487021 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role May 23 12:41:52.487021 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') May 23 12:41:52.488139 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call May 23 12:41:52.488139 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( May 23 12:41:52.488139 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data May 23 12:41:52.488139 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) May 23 12:41:52.488139 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped May 23 12:41:52.488139 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) May 23 12:41:52.488139 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun May 23 12:41:52.488139 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) May 23 12:41:52.488139 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func May 23 12:41:52.488139 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self.get_or_create( May 23 12:41:52.488139 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create May 23 12:41:52.488139 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application with Lock( May 23 12:41:52.488139 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ May 23 12:41:52.488139 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self._enter() May 23 12:41:52.488139 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter May 23 12:41:52.488139 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) May 23 12:41:52.489016 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create May 23 12:41:52.489016 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self.creator() May 23 12:41:52.489016 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value May 23 12:41:52.489016 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application created_value = creator( May 23 12:41:52.489016 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role May 23 12:41:52.489016 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) May 23 12:41:52.489016 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role May 23 12:41:52.489016 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() May 23 12:41:52.489016 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role May 23 12:41:52.489016 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) May 23 12:41:52.489016 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. May 23 12:41:52.489016 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application May 23 12:41:52.490319 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 399/793] 10.222.0.30 () {56 vars in 1102 bytes} [Mon May 23 12:41:52 2022] GET /identity/v3/roles/service => generated 85 bytes in 175 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) May 23 12:41:52.496069 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-753cc688-29b8-420a-93cd-44f1495b7939 None admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:41:52.515824 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ee39ac3-c59b-4ac0-9893-c5eab154e1df None None] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:41:52.529386 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=100 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:41:52.556052 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f487b90d-4e2e-40cc-a5cd-33463ce373b9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:41:52.557142 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f487b90d-4e2e-40cc-a5cd-33463ce373b9 None admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:41:52.557645 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f487b90d-4e2e-40cc-a5cd-33463ce373b9 None admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:41:52.558129 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f487b90d-4e2e-40cc-a5cd-33463ce373b9 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:41:52.569385 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f487b90d-4e2e-40cc-a5cd-33463ce373b9 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:41:52.571144 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f487b90d-4e2e-40cc-a5cd-33463ce373b9 None admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:41:52.590190 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application [None req-f487b90d-4e2e-40cc-a5cd-33463ce373b9 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. May 23 12:41:52.590190 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application Traceback (most recent call last): May 23 12:41:52.590190 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request May 23 12:41:52.590190 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application rv = self.dispatch_request() May 23 12:41:52.590190 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request May 23 12:41:52.590190 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) May 23 12:41:52.590190 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper May 23 12:41:52.590190 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) May 23 12:41:52.590190 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view May 23 12:41:52.590190 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) May 23 12:41:52.590190 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request May 23 12:41:52.590190 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) May 23 12:41:52.590190 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get May 23 12:41:52.590190 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self._get_domain(domain_id) May 23 12:41:52.590190 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain May 23 12:41:52.591148 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) May 23 12:41:52.591148 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped May 23 12:41:52.591148 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) May 23 12:41:52.591148 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun May 23 12:41:52.591148 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) May 23 12:41:52.591148 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func May 23 12:41:52.591148 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self.get_or_create( May 23 12:41:52.591148 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create May 23 12:41:52.591148 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application with Lock( May 23 12:41:52.591148 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ May 23 12:41:52.591148 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self._enter() May 23 12:41:52.591148 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter May 23 12:41:52.591148 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) May 23 12:41:52.591148 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create May 23 12:41:52.591148 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self.creator() May 23 12:41:52.591148 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value May 23 12:41:52.591148 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application created_value = creator( May 23 12:41:52.592091 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain May 23 12:41:52.592091 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) May 23 12:41:52.592091 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. May 23 12:41:52.592091 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application May 23 12:41:52.594459 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 395/794] 10.222.0.30 () {56 vars in 1108 bytes} [Mon May 23 12:41:52 2022] GET /identity/v3/domains/Default => generated 87 bytes in 196 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) May 23 12:41:52.595434 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f487b90d-4e2e-40cc-a5cd-33463ce373b9 None admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:41:52.604444 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-753cc688-29b8-420a-93cd-44f1495b7939 None admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:41:52.640796 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8eaf3708-3733-4326-81d1-e2161ee388ed None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:41:52.641757 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8eaf3708-3733-4326-81d1-e2161ee388ed None admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:41:52.641907 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8eaf3708-3733-4326-81d1-e2161ee388ed None admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:41:52.642182 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8eaf3708-3733-4326-81d1-e2161ee388ed None admin] PATH_INFO: `/v3/roles/admin` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:41:52.653658 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8eaf3708-3733-4326-81d1-e2161ee388ed None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} May 23 12:41:52.660587 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=100 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:41:52.662475 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application [None req-8eaf3708-3733-4326-81d1-e2161ee388ed None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. May 23 12:41:52.662475 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application Traceback (most recent call last): May 23 12:41:52.662475 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request May 23 12:41:52.662475 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application rv = self.dispatch_request() May 23 12:41:52.662475 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request May 23 12:41:52.662475 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) May 23 12:41:52.662475 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper May 23 12:41:52.662475 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) May 23 12:41:52.662475 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view May 23 12:41:52.662475 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) May 23 12:41:52.662475 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request May 23 12:41:52.662475 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) May 23 12:41:52.662475 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get May 23 12:41:52.662475 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self._get_role(role_id) May 23 12:41:52.662475 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role May 23 12:41:52.662475 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') May 23 12:41:52.663451 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call May 23 12:41:52.663451 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( May 23 12:41:52.663451 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data May 23 12:41:52.663451 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) May 23 12:41:52.663451 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped May 23 12:41:52.663451 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) May 23 12:41:52.663451 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun May 23 12:41:52.663451 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) May 23 12:41:52.663451 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func May 23 12:41:52.663451 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self.get_or_create( May 23 12:41:52.663451 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create May 23 12:41:52.663451 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application with Lock( May 23 12:41:52.663451 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ May 23 12:41:52.663451 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self._enter() May 23 12:41:52.663451 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter May 23 12:41:52.663451 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) May 23 12:41:52.664355 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create May 23 12:41:52.664355 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self.creator() May 23 12:41:52.664355 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value May 23 12:41:52.664355 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application created_value = creator( May 23 12:41:52.664355 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role May 23 12:41:52.664355 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) May 23 12:41:52.664355 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role May 23 12:41:52.664355 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() May 23 12:41:52.664355 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role May 23 12:41:52.664355 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) May 23 12:41:52.664355 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. May 23 12:41:52.664355 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application May 23 12:41:52.664355 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 400/795] 10.222.0.30 () {56 vars in 1096 bytes} [Mon May 23 12:41:52 2022] GET /identity/v3/roles/admin => generated 83 bytes in 174 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) May 23 12:41:52.665676 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8eaf3708-3733-4326-81d1-e2161ee388ed None admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:41:52.712027 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f487b90d-4e2e-40cc-a5cd-33463ce373b9 None admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:41:52.753805 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd800c2a-1919-4985-a861-a8dd03cf799a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:41:52.754880 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd800c2a-1919-4985-a861-a8dd03cf799a None admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:41:52.755396 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd800c2a-1919-4985-a861-a8dd03cf799a None admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:41:52.755873 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd800c2a-1919-4985-a861-a8dd03cf799a None admin] PATH_INFO: `/v3/roles` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:41:52.756933 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd800c2a-1919-4985-a861-a8dd03cf799a None admin] RBAC: Adding query filter params (name=service) {{(pid=92057) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} May 23 12:41:52.757585 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd800c2a-1919-4985-a861-a8dd03cf799a None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:41:52.759437 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd800c2a-1919-4985-a861-a8dd03cf799a None admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:41:52.769030 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 396/796] 10.222.0.30 () {56 vars in 1103 bytes} [Mon May 23 12:41:52 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 174 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) May 23 12:41:52.770205 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd800c2a-1919-4985-a861-a8dd03cf799a None admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:41:52.780393 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=100 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:41:52.791585 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8eaf3708-3733-4326-81d1-e2161ee388ed None admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:41:52.828818 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e17e7ca-a778-4baa-bd8f-e5699e37d164 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:41:52.829488 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e17e7ca-a778-4baa-bd8f-e5699e37d164 None admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:41:52.829775 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e17e7ca-a778-4baa-bd8f-e5699e37d164 None admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:41:52.830100 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e17e7ca-a778-4baa-bd8f-e5699e37d164 None admin] PATH_INFO: `/v3/domains` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:41:52.831130 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e17e7ca-a778-4baa-bd8f-e5699e37d164 None admin] RBAC: Adding query filter params (name=Default) {{(pid=92056) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} May 23 12:41:52.831130 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e17e7ca-a778-4baa-bd8f-e5699e37d164 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:41:52.832496 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e17e7ca-a778-4baa-bd8f-e5699e37d164 None admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:41:52.845400 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 401/797] 10.222.0.30 () {56 vars in 1109 bytes} [Mon May 23 12:41:52 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 181 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) May 23 12:41:52.846451 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e17e7ca-a778-4baa-bd8f-e5699e37d164 None admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:41:52.883527 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd800c2a-1919-4985-a861-a8dd03cf799a None admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:41:52.905543 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=100 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:41:52.920916 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24b987de-7718-432c-98f7-0d9f8371c1cd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:41:52.922001 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24b987de-7718-432c-98f7-0d9f8371c1cd None admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:41:52.922495 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24b987de-7718-432c-98f7-0d9f8371c1cd None admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:41:52.923004 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24b987de-7718-432c-98f7-0d9f8371c1cd None admin] PATH_INFO: `/v3/roles` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:41:52.923946 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24b987de-7718-432c-98f7-0d9f8371c1cd None admin] RBAC: Adding query filter params (name=admin) {{(pid=92057) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} May 23 12:41:52.924458 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24b987de-7718-432c-98f7-0d9f8371c1cd None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:41:52.926078 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24b987de-7718-432c-98f7-0d9f8371c1cd None admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:41:52.936349 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 397/798] 10.222.0.30 () {56 vars in 1099 bytes} [Mon May 23 12:41:52 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 166 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) May 23 12:41:52.937557 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24b987de-7718-432c-98f7-0d9f8371c1cd None admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:41:52.956397 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e17e7ca-a778-4baa-bd8f-e5699e37d164 None admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:41:52.994996 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-185b1f9c-0630-4121-84bd-a850e7cd5836 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:41:52.995981 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-185b1f9c-0630-4121-84bd-a850e7cd5836 None admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:41:52.996200 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-185b1f9c-0630-4121-84bd-a850e7cd5836 None admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:41:52.996476 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-185b1f9c-0630-4121-84bd-a850e7cd5836 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:41:53.008716 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-185b1f9c-0630-4121-84bd-a850e7cd5836 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:41:53.010550 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-185b1f9c-0630-4121-84bd-a850e7cd5836 None admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:41:53.036908 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application [None req-185b1f9c-0630-4121-84bd-a850e7cd5836 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. May 23 12:41:53.036908 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application Traceback (most recent call last): May 23 12:41:53.036908 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request May 23 12:41:53.036908 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application rv = self.dispatch_request() May 23 12:41:53.036908 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request May 23 12:41:53.036908 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) May 23 12:41:53.036908 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper May 23 12:41:53.036908 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) May 23 12:41:53.036908 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view May 23 12:41:53.036908 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) May 23 12:41:53.036908 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request May 23 12:41:53.036908 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) May 23 12:41:53.036908 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get May 23 12:41:53.036908 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self._get_domain(domain_id) May 23 12:41:53.036908 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain May 23 12:41:53.037955 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) May 23 12:41:53.037955 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped May 23 12:41:53.037955 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) May 23 12:41:53.037955 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun May 23 12:41:53.037955 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) May 23 12:41:53.037955 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func May 23 12:41:53.037955 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self.get_or_create( May 23 12:41:53.037955 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create May 23 12:41:53.037955 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application with Lock( May 23 12:41:53.037955 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ May 23 12:41:53.037955 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self._enter() May 23 12:41:53.037955 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter May 23 12:41:53.037955 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) May 23 12:41:53.037955 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create May 23 12:41:53.037955 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self.creator() May 23 12:41:53.037955 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value May 23 12:41:53.037955 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application created_value = creator( May 23 12:41:53.039859 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain May 23 12:41:53.039859 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) May 23 12:41:53.039859 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. May 23 12:41:53.039859 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application May 23 12:41:53.045899 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 402/799] 10.222.0.30 () {56 vars in 1108 bytes} [Mon May 23 12:41:52 2022] GET /identity/v3/domains/Default => generated 87 bytes in 200 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) May 23 12:41:53.047460 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-185b1f9c-0630-4121-84bd-a850e7cd5836 None admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:41:53.050968 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=100 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:41:53.055952 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24b987de-7718-432c-98f7-0d9f8371c1cd None admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:41:53.096178 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb5022cf-cc92-452a-9016-1c5b3cafa54c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:41:53.097295 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb5022cf-cc92-452a-9016-1c5b3cafa54c None admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:41:53.097745 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb5022cf-cc92-452a-9016-1c5b3cafa54c None admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:41:53.098206 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb5022cf-cc92-452a-9016-1c5b3cafa54c None admin] PATH_INFO: `/v3/projects/service` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:41:53.106296 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb5022cf-cc92-452a-9016-1c5b3cafa54c None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:41:53.108162 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb5022cf-cc92-452a-9016-1c5b3cafa54c None admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:41:53.123529 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application [None req-cb5022cf-cc92-452a-9016-1c5b3cafa54c None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. May 23 12:41:53.123529 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application Traceback (most recent call last): May 23 12:41:53.123529 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request May 23 12:41:53.123529 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application rv = self.dispatch_request() May 23 12:41:53.123529 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request May 23 12:41:53.123529 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) May 23 12:41:53.123529 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper May 23 12:41:53.123529 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) May 23 12:41:53.123529 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view May 23 12:41:53.123529 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) May 23 12:41:53.123529 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request May 23 12:41:53.123529 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) May 23 12:41:53.123529 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get May 23 12:41:53.123529 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self._get_project(project_id) May 23 12:41:53.123529 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project May 23 12:41:53.124565 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) May 23 12:41:53.124565 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped May 23 12:41:53.124565 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) May 23 12:41:53.124565 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun May 23 12:41:53.124565 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) May 23 12:41:53.124565 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func May 23 12:41:53.124565 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self.get_or_create( May 23 12:41:53.124565 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create May 23 12:41:53.124565 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application with Lock( May 23 12:41:53.124565 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ May 23 12:41:53.124565 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self._enter() May 23 12:41:53.124565 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter May 23 12:41:53.124565 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) May 23 12:41:53.124565 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create May 23 12:41:53.124565 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self.creator() May 23 12:41:53.124565 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value May 23 12:41:53.124565 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application created_value = creator( May 23 12:41:53.125530 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project May 23 12:41:53.125530 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) May 23 12:41:53.125530 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project May 23 12:41:53.125530 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() May 23 12:41:53.125530 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project May 23 12:41:53.125530 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) May 23 12:41:53.125530 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. May 23 12:41:53.125530 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application May 23 12:41:53.128141 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 398/800] 10.222.0.30 () {56 vars in 1111 bytes} [Mon May 23 12:41:52 2022] GET /identity/v3/projects/service => generated 88 bytes in 191 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) May 23 12:41:53.129103 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb5022cf-cc92-452a-9016-1c5b3cafa54c None admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:41:53.184721 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-185b1f9c-0630-4121-84bd-a850e7cd5836 None admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:41:53.190245 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=100 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:41:53.229339 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3f9b173-e85b-4139-80fd-56679282a981 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:41:53.230829 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3f9b173-e85b-4139-80fd-56679282a981 None admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:41:53.231125 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3f9b173-e85b-4139-80fd-56679282a981 None admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:41:53.231405 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3f9b173-e85b-4139-80fd-56679282a981 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:41:53.244383 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3f9b173-e85b-4139-80fd-56679282a981 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:41:53.246133 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3f9b173-e85b-4139-80fd-56679282a981 None admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:41:53.260573 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb5022cf-cc92-452a-9016-1c5b3cafa54c None admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:41:53.267408 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application [None req-d3f9b173-e85b-4139-80fd-56679282a981 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. May 23 12:41:53.267408 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application Traceback (most recent call last): May 23 12:41:53.267408 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request May 23 12:41:53.267408 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application rv = self.dispatch_request() May 23 12:41:53.267408 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request May 23 12:41:53.267408 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) May 23 12:41:53.267408 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper May 23 12:41:53.267408 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) May 23 12:41:53.267408 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view May 23 12:41:53.267408 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) May 23 12:41:53.267408 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request May 23 12:41:53.267408 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) May 23 12:41:53.267408 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get May 23 12:41:53.267408 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self._get_domain(domain_id) May 23 12:41:53.267408 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain May 23 12:41:53.268332 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) May 23 12:41:53.268332 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped May 23 12:41:53.268332 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) May 23 12:41:53.268332 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun May 23 12:41:53.268332 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) May 23 12:41:53.268332 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func May 23 12:41:53.268332 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self.get_or_create( May 23 12:41:53.268332 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create May 23 12:41:53.268332 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application with Lock( May 23 12:41:53.268332 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ May 23 12:41:53.268332 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self._enter() May 23 12:41:53.268332 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter May 23 12:41:53.268332 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) May 23 12:41:53.268332 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create May 23 12:41:53.268332 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self.creator() May 23 12:41:53.268332 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value May 23 12:41:53.268332 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application created_value = creator( May 23 12:41:53.269952 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain May 23 12:41:53.269952 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) May 23 12:41:53.269952 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. May 23 12:41:53.269952 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application May 23 12:41:53.269952 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 403/801] 10.222.0.30 () {56 vars in 1108 bytes} [Mon May 23 12:41:53 2022] GET /identity/v3/domains/Default => generated 87 bytes in 222 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) May 23 12:41:53.269952 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3f9b173-e85b-4139-80fd-56679282a981 None admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:41:53.296366 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6306d05b-3ee7-4a5b-85aa-b56431501486 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:41:53.297039 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6306d05b-3ee7-4a5b-85aa-b56431501486 None admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:41:53.297333 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6306d05b-3ee7-4a5b-85aa-b56431501486 None admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:41:53.297611 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6306d05b-3ee7-4a5b-85aa-b56431501486 None admin] PATH_INFO: `/v3/domains` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:41:53.298374 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6306d05b-3ee7-4a5b-85aa-b56431501486 None admin] RBAC: Adding query filter params (name=Default) {{(pid=92057) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} May 23 12:41:53.298692 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6306d05b-3ee7-4a5b-85aa-b56431501486 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:41:53.300118 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6306d05b-3ee7-4a5b-85aa-b56431501486 None admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:41:53.317917 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 399/802] 10.222.0.30 () {56 vars in 1109 bytes} [Mon May 23 12:41:53 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 185 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) May 23 12:41:53.317917 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3a0ee66-251e-4496-8ea2-1f4c041d4880 None None] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:41:53.317917 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3a0ee66-251e-4496-8ea2-1f4c041d4880 None None] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:41:53.317917 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3a0ee66-251e-4496-8ea2-1f4c041d4880 None None] PATH_INFO: `/` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:41:53.317917 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 400/803] 10.222.0.30 () {54 vars in 926 bytes} [Mon May 23 12:41:53 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) May 23 12:41:53.319216 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3a0ee66-251e-4496-8ea2-1f4c041d4880 None None] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:41:53.340145 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=100 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:41:53.397966 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3f9b173-e85b-4139-80fd-56679282a981 None admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:41:53.432139 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3a0ee66-251e-4496-8ea2-1f4c041d4880 None None] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:41:53.433856 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=100 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:41:53.434217 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11baa4c1-4c02-48b4-89fa-cac8b09773b8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:41:53.434946 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11baa4c1-4c02-48b4-89fa-cac8b09773b8 None admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:41:53.435172 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11baa4c1-4c02-48b4-89fa-cac8b09773b8 None admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:41:53.435448 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11baa4c1-4c02-48b4-89fa-cac8b09773b8 None admin] PATH_INFO: `/v3/projects` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:41:53.436238 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11baa4c1-4c02-48b4-89fa-cac8b09773b8 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=92056) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} May 23 12:41:53.436551 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11baa4c1-4c02-48b4-89fa-cac8b09773b8 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:41:53.438013 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11baa4c1-4c02-48b4-89fa-cac8b09773b8 None admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:41:53.454397 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 404/804] 10.222.0.30 () {56 vars in 1148 bytes} [Mon May 23 12:41:53 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 185 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) May 23 12:41:53.455117 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11baa4c1-4c02-48b4-89fa-cac8b09773b8 None admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:41:53.469413 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-519eebb7-3674-4b53-872d-e04dae7688a9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:41:53.470103 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-519eebb7-3674-4b53-872d-e04dae7688a9 None admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:41:53.470384 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-519eebb7-3674-4b53-872d-e04dae7688a9 None admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:41:53.470658 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-519eebb7-3674-4b53-872d-e04dae7688a9 None admin] PATH_INFO: `/v3/domains` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:41:53.471379 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-519eebb7-3674-4b53-872d-e04dae7688a9 None admin] RBAC: Adding query filter params (name=Default) {{(pid=92057) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} May 23 12:41:53.471679 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-519eebb7-3674-4b53-872d-e04dae7688a9 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:41:53.473044 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-519eebb7-3674-4b53-872d-e04dae7688a9 None admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:41:53.485728 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 401/805] 10.222.0.30 () {56 vars in 1109 bytes} [Mon May 23 12:41:53 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 167 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) May 23 12:41:53.486972 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-539157e3-8111-4e21-b19b-bf37ce5aefed None None] REQUEST_METHOD: `POST` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:41:53.487255 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-539157e3-8111-4e21-b19b-bf37ce5aefed None None] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:41:53.487529 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-539157e3-8111-4e21-b19b-bf37ce5aefed None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:41:53.504564 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.auth.core [None req-539157e3-8111-4e21-b19b-bf37ce5aefed None None] MFA Rules not processed for user `0f7119f8fa5c4445977671140ded91a4`. Rule list: `[]` (Enabled: `True`). {{(pid=92057) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 12:41:53.505931 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.fernet_utils [None req-539157e3-8111-4e21-b19b-bf37ce5aefed None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92057) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 12:41:53.564033 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11baa4c1-4c02-48b4-89fa-cac8b09773b8 None admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:41:53.571993 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 402/806] 10.222.0.30 () {58 vars in 1023 bytes} [Mon May 23 12:41:53 2022] POST /identity/v3/auth/tokens => generated 1002 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) May 23 12:41:53.572809 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-539157e3-8111-4e21-b19b-bf37ce5aefed None None] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:41:53.582779 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=100 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:41:53.605012 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-873c3697-4637-4e61-baf6-17393a81743b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:41:53.605814 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-873c3697-4637-4e61-baf6-17393a81743b None admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:41:53.606122 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-873c3697-4637-4e61-baf6-17393a81743b None admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:41:53.606394 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-873c3697-4637-4e61-baf6-17393a81743b None admin] PATH_INFO: `/v3/users/glance-swift` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:41:53.613534 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-873c3697-4637-4e61-baf6-17393a81743b None admin] RBAC: Authorizing `identity:get_user(user_id=glance-swift)` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:41:53.615326 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-873c3697-4637-4e61-baf6-17393a81743b None admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:41:53.631876 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application [None req-873c3697-4637-4e61-baf6-17393a81743b None admin] Could not find user: glance-swift.: keystone.exception.UserNotFound: Could not find user: glance-swift. May 23 12:41:53.631876 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application Traceback (most recent call last): May 23 12:41:53.631876 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request May 23 12:41:53.631876 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application rv = self.dispatch_request() May 23 12:41:53.631876 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request May 23 12:41:53.631876 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) May 23 12:41:53.631876 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper May 23 12:41:53.631876 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) May 23 12:41:53.631876 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view May 23 12:41:53.631876 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) May 23 12:41:53.631876 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request May 23 12:41:53.631876 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) May 23 12:41:53.631876 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get May 23 12:41:53.631876 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self._get_user(user_id) May 23 12:41:53.631876 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user May 23 12:41:53.631876 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) May 23 12:41:53.632844 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped May 23 12:41:53.632844 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) May 23 12:41:53.632844 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper May 23 12:41:53.632844 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) May 23 12:41:53.632844 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper May 23 12:41:53.632844 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) May 23 12:41:53.632844 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun May 23 12:41:53.632844 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) May 23 12:41:53.632844 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func May 23 12:41:53.632844 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self.get_or_create( May 23 12:41:53.632844 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create May 23 12:41:53.632844 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application with Lock( May 23 12:41:53.632844 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ May 23 12:41:53.632844 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self._enter() May 23 12:41:53.632844 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter May 23 12:41:53.632844 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) May 23 12:41:53.632844 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create May 23 12:41:53.632844 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self.creator() May 23 12:41:53.633835 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value May 23 12:41:53.633835 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application created_value = creator( May 23 12:41:53.633835 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user May 23 12:41:53.633835 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) May 23 12:41:53.633835 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user May 23 12:41:53.633835 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) May 23 12:41:53.633835 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user May 23 12:41:53.633835 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) May 23 12:41:53.633835 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: glance-swift. May 23 12:41:53.633835 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application May 23 12:41:53.633835 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 405/807] 10.222.0.30 () {56 vars in 1117 bytes} [Mon May 23 12:41:53 2022] GET /identity/v3/users/glance-swift => generated 90 bytes in 178 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) May 23 12:41:53.634467 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-873c3697-4637-4e61-baf6-17393a81743b None admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:41:53.668325 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=100 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:41:53.677332 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-539157e3-8111-4e21-b19b-bf37ce5aefed None None] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:41:53.711967 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fdcf6365-6f16-4b1a-8173-4d4b27eaf636 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:41:53.712933 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdcf6365-6f16-4b1a-8173-4d4b27eaf636 None admin] REQUEST_METHOD: `PUT` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:41:53.713451 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdcf6365-6f16-4b1a-8173-4d4b27eaf636 None admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:41:53.713580 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdcf6365-6f16-4b1a-8173-4d4b27eaf636 None admin] PATH_INFO: `/v3/projects/9c389ae65aaa4542bb5a5c4fd0bacf5b/users/138d91edb99f450e846765f9eef00d62/roles/bc3c3c9d654940cbaa5177a6be23d1bc` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:41:53.715561 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fdcf6365-6f16-4b1a-8173-4d4b27eaf636 None admin] RBAC: Authorizing `identity:create_grant(project_id=9c389ae65aaa4542bb5a5c4fd0bacf5b, user_id=138d91edb99f450e846765f9eef00d62, role_id=bc3c3c9d654940cbaa5177a6be23d1bc)` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:41:53.717005 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fdcf6365-6f16-4b1a-8173-4d4b27eaf636 None admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:41:53.734382 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-873c3697-4637-4e61-baf6-17393a81743b None admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:41:53.768542 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0514b790-e103-468e-9a9f-81b685b4c95d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:41:53.769324 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0514b790-e103-468e-9a9f-81b685b4c95d None admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:41:53.769614 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0514b790-e103-468e-9a9f-81b685b4c95d None admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:41:53.769919 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0514b790-e103-468e-9a9f-81b685b4c95d None admin] PATH_INFO: `/v3/users/nova` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:41:53.777175 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0514b790-e103-468e-9a9f-81b685b4c95d None admin] RBAC: Authorizing `identity:get_user(user_id=nova)` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:41:53.778643 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0514b790-e103-468e-9a9f-81b685b4c95d None admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:41:53.793140 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application [None req-0514b790-e103-468e-9a9f-81b685b4c95d None admin] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova. May 23 12:41:53.793140 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application Traceback (most recent call last): May 23 12:41:53.793140 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request May 23 12:41:53.793140 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application rv = self.dispatch_request() May 23 12:41:53.793140 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request May 23 12:41:53.793140 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) May 23 12:41:53.793140 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper May 23 12:41:53.793140 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) May 23 12:41:53.793140 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view May 23 12:41:53.793140 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) May 23 12:41:53.793140 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request May 23 12:41:53.793140 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) May 23 12:41:53.793140 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get May 23 12:41:53.793140 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self._get_user(user_id) May 23 12:41:53.793140 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user May 23 12:41:53.793140 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) May 23 12:41:53.794591 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped May 23 12:41:53.794591 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) May 23 12:41:53.794591 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper May 23 12:41:53.794591 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) May 23 12:41:53.794591 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper May 23 12:41:53.794591 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) May 23 12:41:53.794591 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun May 23 12:41:53.794591 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) May 23 12:41:53.794591 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func May 23 12:41:53.794591 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self.get_or_create( May 23 12:41:53.794591 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create May 23 12:41:53.794591 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application with Lock( May 23 12:41:53.794591 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ May 23 12:41:53.794591 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self._enter() May 23 12:41:53.794591 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter May 23 12:41:53.794591 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) May 23 12:41:53.794591 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create May 23 12:41:53.794591 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self.creator() May 23 12:41:53.795566 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value May 23 12:41:53.795566 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application created_value = creator( May 23 12:41:53.795566 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user May 23 12:41:53.795566 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) May 23 12:41:53.795566 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user May 23 12:41:53.795566 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) May 23 12:41:53.795566 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user May 23 12:41:53.795566 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) May 23 12:41:53.795566 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: nova. May 23 12:41:53.795566 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application May 23 12:41:53.795566 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 406/808] 10.222.0.30 () {56 vars in 1093 bytes} [Mon May 23 12:41:53 2022] GET /identity/v3/users/nova => generated 82 bytes in 160 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) May 23 12:41:53.796252 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fff92b0-df50-4f54-897a-e7b6784bdb07 None None] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:41:53.796252 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fff92b0-df50-4f54-897a-e7b6784bdb07 None None] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:41:53.796252 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fff92b0-df50-4f54-897a-e7b6784bdb07 None None] PATH_INFO: `/` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:41:53.797389 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 407/809] 10.222.0.30 () {54 vars in 926 bytes} [Mon May 23 12:41:53 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) May 23 12:41:53.798154 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2fff92b0-df50-4f54-897a-e7b6784bdb07 None None] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:41:53.826184 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 403/810] 10.222.0.30 () {58 vars in 1439 bytes} [Mon May 23 12:41:53 2022] PUT /identity/v3/projects/9c389ae65aaa4542bb5a5c4fd0bacf5b/users/138d91edb99f450e846765f9eef00d62/roles/bc3c3c9d654940cbaa5177a6be23d1bc => generated 0 bytes in 242 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) May 23 12:41:53.826184 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fdcf6365-6f16-4b1a-8173-4d4b27eaf636 None admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:41:53.835744 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=100 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:41:53.912986 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2fff92b0-df50-4f54-897a-e7b6784bdb07 None None] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:41:53.931839 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fdcf6365-6f16-4b1a-8173-4d4b27eaf636 None admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:41:53.953664 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8676eb9-0730-46c5-bf33-2cfbbc2b90b0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:41:53.954627 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8676eb9-0730-46c5-bf33-2cfbbc2b90b0 None admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:41:53.954991 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8676eb9-0730-46c5-bf33-2cfbbc2b90b0 None admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:41:53.955332 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8676eb9-0730-46c5-bf33-2cfbbc2b90b0 None admin] PATH_INFO: `/v3/services/network` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:41:53.956219 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8676eb9-0730-46c5-bf33-2cfbbc2b90b0 None admin] RBAC: Authorizing `identity:get_service(service_id=network)` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:41:53.958678 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8676eb9-0730-46c5-bf33-2cfbbc2b90b0 None admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:41:53.971374 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75ad4c10-a8ae-4343-aa8e-4d27d5caa4a4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:41:53.972051 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75ad4c10-a8ae-4343-aa8e-4d27d5caa4a4 None admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:41:53.972178 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application [None req-a8676eb9-0730-46c5-bf33-2cfbbc2b90b0 None admin] Could not find service: network.: keystone.exception.ServiceNotFound: Could not find service: network. May 23 12:41:53.972178 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application Traceback (most recent call last): May 23 12:41:53.972178 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service May 23 12:41:53.972178 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) May 23 12:41:53.972178 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service May 23 12:41:53.972178 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() May 23 12:41:53.972178 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service May 23 12:41:53.972178 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) May 23 12:41:53.972178 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: network. May 23 12:41:53.972178 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application May 23 12:41:53.972178 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: May 23 12:41:53.972178 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application May 23 12:41:53.972178 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application Traceback (most recent call last): May 23 12:41:53.972178 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request May 23 12:41:53.972178 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application rv = self.dispatch_request() May 23 12:41:53.972178 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request May 23 12:41:53.972178 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) May 23 12:41:53.973118 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper May 23 12:41:53.973118 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) May 23 12:41:53.973118 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view May 23 12:41:53.973118 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) May 23 12:41:53.973118 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request May 23 12:41:53.973118 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) May 23 12:41:53.973118 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get May 23 12:41:53.973118 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self._get_service(service_id) May 23 12:41:53.973118 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service May 23 12:41:53.973118 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) May 23 12:41:53.973118 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped May 23 12:41:53.973118 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) May 23 12:41:53.973118 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun May 23 12:41:53.973118 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) May 23 12:41:53.973118 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func May 23 12:41:53.973118 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self.get_or_create( May 23 12:41:53.974105 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create May 23 12:41:53.974105 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application with Lock( May 23 12:41:53.974105 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ May 23 12:41:53.974105 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self._enter() May 23 12:41:53.974105 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter May 23 12:41:53.974105 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) May 23 12:41:53.974105 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create May 23 12:41:53.974105 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self.creator() May 23 12:41:53.974105 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value May 23 12:41:53.974105 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application created_value = creator( May 23 12:41:53.974105 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service May 23 12:41:53.974105 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) May 23 12:41:53.974105 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: network. May 23 12:41:53.974105 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application May 23 12:41:53.975048 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75ad4c10-a8ae-4343-aa8e-4d27d5caa4a4 None admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:41:53.975048 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75ad4c10-a8ae-4343-aa8e-4d27d5caa4a4 None admin] PATH_INFO: `/v3/users` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:41:53.975048 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75ad4c10-a8ae-4343-aa8e-4d27d5caa4a4 None admin] RBAC: Adding query filter params (domain_id=default, name=glance-swift) {{(pid=92057) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} May 23 12:41:53.975048 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75ad4c10-a8ae-4343-aa8e-4d27d5caa4a4 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:41:53.975877 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75ad4c10-a8ae-4343-aa8e-4d27d5caa4a4 None admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:41:53.978483 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 408/811] 10.222.0.30 () {56 vars in 1111 bytes} [Mon May 23 12:41:53 2022] GET /identity/v3/services/network => generated 88 bytes in 180 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) May 23 12:41:53.979553 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8676eb9-0730-46c5-bf33-2cfbbc2b90b0 None admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:41:54.000424 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 404/812] 10.222.0.30 () {56 vars in 1149 bytes} [Mon May 23 12:41:53 2022] GET /identity/v3/users?domain_id=default&name=glance-swift => generated 384 bytes in 186 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) May 23 12:41:54.001733 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61cdbdc3-e1a3-471d-99dc-4e9b5456717f None None] REQUEST_METHOD: `POST` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:41:54.003268 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61cdbdc3-e1a3-471d-99dc-4e9b5456717f None None] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:41:54.003563 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61cdbdc3-e1a3-471d-99dc-4e9b5456717f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:41:54.021867 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.auth.core [None req-61cdbdc3-e1a3-471d-99dc-4e9b5456717f None None] MFA Rules not processed for user `0f7119f8fa5c4445977671140ded91a4`. Rule list: `[]` (Enabled: `True`). {{(pid=92057) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 12:41:54.023116 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.fernet_utils [None req-61cdbdc3-e1a3-471d-99dc-4e9b5456717f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92057) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 12:41:54.029869 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:41:54.085262 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=100 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:41:54.089177 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8676eb9-0730-46c5-bf33-2cfbbc2b90b0 None admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:41:54.095786 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 405/813] 10.222.0.30 () {58 vars in 1023 bytes} [Mon May 23 12:41:54 2022] POST /identity/v3/auth/tokens => generated 1002 bytes in 95 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) May 23 12:41:54.097114 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61cdbdc3-e1a3-471d-99dc-4e9b5456717f None None] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:41:54.133047 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5788a645-e189-4182-93ed-463f930d6063 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:41:54.133729 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5788a645-e189-4182-93ed-463f930d6063 None admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:41:54.134045 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5788a645-e189-4182-93ed-463f930d6063 None admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:41:54.134327 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5788a645-e189-4182-93ed-463f930d6063 None admin] PATH_INFO: `/v3/users` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:41:54.135175 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5788a645-e189-4182-93ed-463f930d6063 None admin] RBAC: Adding query filter params (domain_id=default, name=nova) {{(pid=92056) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} May 23 12:41:54.135488 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5788a645-e189-4182-93ed-463f930d6063 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:41:54.136936 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5788a645-e189-4182-93ed-463f930d6063 None admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:41:54.156089 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 409/814] 10.222.0.30 () {56 vars in 1133 bytes} [Mon May 23 12:41:53 2022] GET /identity/v3/users?domain_id=default&name=nova => generated 368 bytes in 177 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) May 23 12:41:54.158734 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5788a645-e189-4182-93ed-463f930d6063 None admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:41:54.205864 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61cdbdc3-e1a3-471d-99dc-4e9b5456717f None None] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:41:54.243154 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35a8f718-43cb-4e0e-b940-061b4ce3d5df None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:41:54.243815 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35a8f718-43cb-4e0e-b940-061b4ce3d5df None admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:41:54.244098 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35a8f718-43cb-4e0e-b940-061b4ce3d5df None admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:41:54.244373 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35a8f718-43cb-4e0e-b940-061b4ce3d5df None admin] PATH_INFO: `/v3/services` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:41:54.245073 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35a8f718-43cb-4e0e-b940-061b4ce3d5df None admin] RBAC: Adding query filter params (name=network) {{(pid=92057) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} May 23 12:41:54.245379 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35a8f718-43cb-4e0e-b940-061b4ce3d5df None admin] RBAC: Authorizing `identity:list_services()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:41:54.246787 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35a8f718-43cb-4e0e-b940-061b4ce3d5df None admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:41:54.251330 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 406/815] 10.222.0.30 () {56 vars in 1112 bytes} [Mon May 23 12:41:54 2022] GET /identity/v3/services?name=network => generated 124 bytes in 155 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) May 23 12:41:54.252103 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35a8f718-43cb-4e0e-b940-061b4ce3d5df None admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:41:54.258617 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=100 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:41:54.276426 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5788a645-e189-4182-93ed-463f930d6063 None admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:41:54.314819 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba2d3abd-7ea1-44d7-a0f7-4ca02f7f7820 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:41:54.315605 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba2d3abd-7ea1-44d7-a0f7-4ca02f7f7820 None admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:41:54.315888 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba2d3abd-7ea1-44d7-a0f7-4ca02f7f7820 None admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:41:54.316164 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba2d3abd-7ea1-44d7-a0f7-4ca02f7f7820 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:41:54.334028 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba2d3abd-7ea1-44d7-a0f7-4ca02f7f7820 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:41:54.335483 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba2d3abd-7ea1-44d7-a0f7-4ca02f7f7820 None admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:41:54.356069 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application [None req-ba2d3abd-7ea1-44d7-a0f7-4ca02f7f7820 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. May 23 12:41:54.356069 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application Traceback (most recent call last): May 23 12:41:54.356069 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request May 23 12:41:54.356069 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application rv = self.dispatch_request() May 23 12:41:54.356069 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request May 23 12:41:54.356069 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) May 23 12:41:54.356069 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper May 23 12:41:54.356069 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) May 23 12:41:54.356069 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view May 23 12:41:54.356069 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) May 23 12:41:54.356069 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request May 23 12:41:54.356069 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) May 23 12:41:54.356069 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get May 23 12:41:54.356069 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self._get_domain(domain_id) May 23 12:41:54.356069 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain May 23 12:41:54.357031 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) May 23 12:41:54.357031 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped May 23 12:41:54.357031 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) May 23 12:41:54.357031 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun May 23 12:41:54.357031 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) May 23 12:41:54.357031 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func May 23 12:41:54.357031 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self.get_or_create( May 23 12:41:54.357031 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create May 23 12:41:54.357031 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application with Lock( May 23 12:41:54.357031 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ May 23 12:41:54.357031 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self._enter() May 23 12:41:54.357031 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter May 23 12:41:54.357031 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) May 23 12:41:54.357031 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create May 23 12:41:54.357031 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self.creator() May 23 12:41:54.357031 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value May 23 12:41:54.357031 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application created_value = creator( May 23 12:41:54.358394 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain May 23 12:41:54.358394 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) May 23 12:41:54.358394 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. May 23 12:41:54.358394 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application May 23 12:41:54.358640 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35a8f718-43cb-4e0e-b940-061b4ce3d5df None admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:41:54.363060 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 410/816] 10.222.0.30 () {56 vars in 1108 bytes} [Mon May 23 12:41:54 2022] GET /identity/v3/domains/Default => generated 87 bytes in 206 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) May 23 12:41:54.364053 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba2d3abd-7ea1-44d7-a0f7-4ca02f7f7820 None admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:41:54.376595 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=100 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:41:54.394045 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b9c9492-704e-4644-af12-5059b713015d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:41:54.394851 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b9c9492-704e-4644-af12-5059b713015d None admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:41:54.395127 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b9c9492-704e-4644-af12-5059b713015d None admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:41:54.395949 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b9c9492-704e-4644-af12-5059b713015d None admin] PATH_INFO: `/v3/services/block-storage` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:41:54.396594 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b9c9492-704e-4644-af12-5059b713015d None admin] RBAC: Authorizing `identity:get_service(service_id=block-storage)` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:41:54.398016 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b9c9492-704e-4644-af12-5059b713015d None admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:41:54.410555 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application [None req-3b9c9492-704e-4644-af12-5059b713015d None admin] Could not find service: block-storage.: keystone.exception.ServiceNotFound: Could not find service: block-storage. May 23 12:41:54.410555 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application Traceback (most recent call last): May 23 12:41:54.410555 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service May 23 12:41:54.410555 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) May 23 12:41:54.410555 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service May 23 12:41:54.410555 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() May 23 12:41:54.410555 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service May 23 12:41:54.410555 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) May 23 12:41:54.410555 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: block-storage. May 23 12:41:54.410555 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application May 23 12:41:54.410555 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: May 23 12:41:54.410555 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application May 23 12:41:54.410555 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application Traceback (most recent call last): May 23 12:41:54.410555 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request May 23 12:41:54.410555 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application rv = self.dispatch_request() May 23 12:41:54.410555 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request May 23 12:41:54.412122 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) May 23 12:41:54.412122 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper May 23 12:41:54.412122 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) May 23 12:41:54.412122 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view May 23 12:41:54.412122 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) May 23 12:41:54.412122 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request May 23 12:41:54.412122 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) May 23 12:41:54.412122 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get May 23 12:41:54.412122 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self._get_service(service_id) May 23 12:41:54.412122 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service May 23 12:41:54.412122 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) May 23 12:41:54.412122 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped May 23 12:41:54.412122 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) May 23 12:41:54.412122 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun May 23 12:41:54.412122 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) May 23 12:41:54.412122 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func May 23 12:41:54.413088 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self.get_or_create( May 23 12:41:54.413088 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create May 23 12:41:54.413088 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application with Lock( May 23 12:41:54.413088 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ May 23 12:41:54.413088 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self._enter() May 23 12:41:54.413088 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter May 23 12:41:54.413088 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) May 23 12:41:54.413088 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create May 23 12:41:54.413088 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self.creator() May 23 12:41:54.413088 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value May 23 12:41:54.413088 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application created_value = creator( May 23 12:41:54.413088 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service May 23 12:41:54.413088 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) May 23 12:41:54.413088 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: block-storage. May 23 12:41:54.413088 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application May 23 12:41:54.413088 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 407/817] 10.222.0.30 () {56 vars in 1129 bytes} [Mon May 23 12:41:54 2022] GET /identity/v3/services/block-storage => generated 94 bytes in 160 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) May 23 12:41:54.414034 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b9c9492-704e-4644-af12-5059b713015d None admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:41:54.476853 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba2d3abd-7ea1-44d7-a0f7-4ca02f7f7820 None admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:41:54.480775 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=100 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:41:54.513851 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b9c9492-704e-4644-af12-5059b713015d None admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:41:54.518999 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95c4f54d-4eec-4c19-92b7-dc5a8884b4f9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:41:54.519654 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95c4f54d-4eec-4c19-92b7-dc5a8884b4f9 None admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:41:54.519934 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95c4f54d-4eec-4c19-92b7-dc5a8884b4f9 None admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:41:54.520205 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95c4f54d-4eec-4c19-92b7-dc5a8884b4f9 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:41:54.553050 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10df907e-4cd1-44f4-94f7-e4e2ad8cac30 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:41:54.553715 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10df907e-4cd1-44f4-94f7-e4e2ad8cac30 None admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:41:54.554036 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10df907e-4cd1-44f4-94f7-e4e2ad8cac30 None admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:41:54.554316 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10df907e-4cd1-44f4-94f7-e4e2ad8cac30 None admin] PATH_INFO: `/v3/services` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:41:54.555010 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10df907e-4cd1-44f4-94f7-e4e2ad8cac30 None admin] RBAC: Adding query filter params (type=network) {{(pid=92057) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} May 23 12:41:54.555314 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10df907e-4cd1-44f4-94f7-e4e2ad8cac30 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:41:54.556691 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10df907e-4cd1-44f4-94f7-e4e2ad8cac30 None admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:41:54.562144 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 408/818] 10.222.0.30 () {56 vars in 1112 bytes} [Mon May 23 12:41:54 2022] GET /identity/v3/services?type=network => generated 350 bytes in 149 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) May 23 12:41:54.562943 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10df907e-4cd1-44f4-94f7-e4e2ad8cac30 None admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:41:54.607264 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95c4f54d-4eec-4c19-92b7-dc5a8884b4f9 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:41:54.609268 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95c4f54d-4eec-4c19-92b7-dc5a8884b4f9 None admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:41:54.620093 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=100 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:41:54.671894 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10df907e-4cd1-44f4-94f7-e4e2ad8cac30 None admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:41:54.706317 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=100 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:41:54.712498 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25a679d3-50ab-4e64-b23b-eb1b08c3af5a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:41:54.713167 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25a679d3-50ab-4e64-b23b-eb1b08c3af5a None admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:41:54.713671 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25a679d3-50ab-4e64-b23b-eb1b08c3af5a None admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:41:54.713784 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25a679d3-50ab-4e64-b23b-eb1b08c3af5a None admin] PATH_INFO: `/v3/domains` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:41:54.714458 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25a679d3-50ab-4e64-b23b-eb1b08c3af5a None admin] RBAC: Adding query filter params (name=Default) {{(pid=92057) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} May 23 12:41:54.714761 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25a679d3-50ab-4e64-b23b-eb1b08c3af5a None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:41:54.716218 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25a679d3-50ab-4e64-b23b-eb1b08c3af5a None admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:41:54.726346 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 411/819] 10.222.0.30 () {58 vars in 1282 bytes} [Mon May 23 12:41:54 2022] GET /identity/v3/auth/tokens => generated 1002 bytes in 358 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) May 23 12:41:54.726346 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95c4f54d-4eec-4c19-92b7-dc5a8884b4f9 None admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:41:54.736630 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 409/820] 10.222.0.30 () {56 vars in 1109 bytes} [Mon May 23 12:41:54 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 170 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) May 23 12:41:54.736630 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25a679d3-50ab-4e64-b23b-eb1b08c3af5a None admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:41:54.834379 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95c4f54d-4eec-4c19-92b7-dc5a8884b4f9 None admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:41:54.843960 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25a679d3-50ab-4e64-b23b-eb1b08c3af5a None admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:41:54.844243 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=100 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:41:54.870929 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e89b2918-fcf7-4ef2-bb7b-c290f1d3dd5b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:41:54.871679 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e89b2918-fcf7-4ef2-bb7b-c290f1d3dd5b None admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:41:54.872064 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e89b2918-fcf7-4ef2-bb7b-c290f1d3dd5b None admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:41:54.872422 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e89b2918-fcf7-4ef2-bb7b-c290f1d3dd5b None admin] PATH_INFO: `/v3/services` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:41:54.873131 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e89b2918-fcf7-4ef2-bb7b-c290f1d3dd5b None admin] RBAC: Adding query filter params (name=block-storage) {{(pid=92056) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} May 23 12:41:54.873440 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e89b2918-fcf7-4ef2-bb7b-c290f1d3dd5b None admin] RBAC: Authorizing `identity:list_services()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:41:54.874848 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e89b2918-fcf7-4ef2-bb7b-c290f1d3dd5b None admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:41:54.884908 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 412/821] 10.222.0.30 () {56 vars in 1124 bytes} [Mon May 23 12:41:54 2022] GET /identity/v3/services?name=block-storage => generated 130 bytes in 159 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) May 23 12:41:54.884908 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e89b2918-fcf7-4ef2-bb7b-c290f1d3dd5b None admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:41:54.886145 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20c4b775-d2d8-4018-9018-f29fb7621e21 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:41:54.886868 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20c4b775-d2d8-4018-9018-f29fb7621e21 None admin] REQUEST_METHOD: `POST` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:41:54.887212 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20c4b775-d2d8-4018-9018-f29fb7621e21 None admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:41:54.887725 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20c4b775-d2d8-4018-9018-f29fb7621e21 None admin] PATH_INFO: `/v3/endpoints` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:41:54.889374 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20c4b775-d2d8-4018-9018-f29fb7621e21 None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:41:54.890922 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20c4b775-d2d8-4018-9018-f29fb7621e21 None admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:41:54.985925 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 410/822] 10.222.0.30 () {60 vars in 1144 bytes} [Mon May 23 12:41:54 2022] POST /identity/v3/endpoints => generated 327 bytes in 244 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) May 23 12:41:54.985925 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20c4b775-d2d8-4018-9018-f29fb7621e21 None admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:41:55.001637 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e89b2918-fcf7-4ef2-bb7b-c290f1d3dd5b None admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:41:55.052339 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:41:55.059079 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0cb81f54-b680-430b-854d-fae57ea6af2b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:41:55.060242 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cb81f54-b680-430b-854d-fae57ea6af2b None admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:41:55.060531 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cb81f54-b680-430b-854d-fae57ea6af2b None admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:41:55.061096 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cb81f54-b680-430b-854d-fae57ea6af2b None admin] PATH_INFO: `/v3/domains/Default` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:41:55.077295 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0cb81f54-b680-430b-854d-fae57ea6af2b None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:41:55.078809 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0cb81f54-b680-430b-854d-fae57ea6af2b None admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:41:55.105956 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application [None req-0cb81f54-b680-430b-854d-fae57ea6af2b None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. May 23 12:41:55.105956 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application Traceback (most recent call last): May 23 12:41:55.105956 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request May 23 12:41:55.105956 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application rv = self.dispatch_request() May 23 12:41:55.105956 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request May 23 12:41:55.105956 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) May 23 12:41:55.105956 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper May 23 12:41:55.105956 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) May 23 12:41:55.105956 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view May 23 12:41:55.105956 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) May 23 12:41:55.105956 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request May 23 12:41:55.105956 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) May 23 12:41:55.105956 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get May 23 12:41:55.105956 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self._get_domain(domain_id) May 23 12:41:55.105956 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain May 23 12:41:55.113004 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) May 23 12:41:55.113004 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped May 23 12:41:55.113004 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) May 23 12:41:55.113004 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun May 23 12:41:55.113004 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) May 23 12:41:55.113004 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func May 23 12:41:55.113004 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self.get_or_create( May 23 12:41:55.113004 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create May 23 12:41:55.113004 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application with Lock( May 23 12:41:55.113004 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ May 23 12:41:55.113004 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self._enter() May 23 12:41:55.113004 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter May 23 12:41:55.113004 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) May 23 12:41:55.113004 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create May 23 12:41:55.113004 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self.creator() May 23 12:41:55.113004 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value May 23 12:41:55.113004 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application created_value = creator( May 23 12:41:55.113974 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain May 23 12:41:55.113974 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) May 23 12:41:55.113974 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. May 23 12:41:55.113974 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application May 23 12:41:55.113974 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 413/823] 10.222.0.30 () {56 vars in 1108 bytes} [Mon May 23 12:41:54 2022] GET /identity/v3/domains/Default => generated 87 bytes in 227 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) May 23 12:41:55.113974 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0cb81f54-b680-430b-854d-fae57ea6af2b None admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:41:55.114402 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20c4b775-d2d8-4018-9018-f29fb7621e21 None admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:41:55.139603 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=100 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:41:55.152037 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c48d1a3-9780-450f-9798-cc70e75fb4e1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:41:55.152961 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c48d1a3-9780-450f-9798-cc70e75fb4e1 None admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:41:55.153309 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c48d1a3-9780-450f-9798-cc70e75fb4e1 None admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:41:55.153647 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c48d1a3-9780-450f-9798-cc70e75fb4e1 None admin] PATH_INFO: `/v3/projects/service` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:41:55.162584 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c48d1a3-9780-450f-9798-cc70e75fb4e1 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:41:55.164212 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c48d1a3-9780-450f-9798-cc70e75fb4e1 None admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:41:55.180897 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application [None req-9c48d1a3-9780-450f-9798-cc70e75fb4e1 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. May 23 12:41:55.180897 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application Traceback (most recent call last): May 23 12:41:55.180897 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request May 23 12:41:55.180897 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application rv = self.dispatch_request() May 23 12:41:55.180897 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request May 23 12:41:55.180897 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) May 23 12:41:55.180897 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper May 23 12:41:55.180897 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) May 23 12:41:55.180897 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view May 23 12:41:55.180897 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) May 23 12:41:55.180897 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request May 23 12:41:55.180897 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) May 23 12:41:55.180897 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get May 23 12:41:55.180897 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self._get_project(project_id) May 23 12:41:55.180897 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project May 23 12:41:55.182750 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) May 23 12:41:55.182750 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped May 23 12:41:55.182750 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) May 23 12:41:55.182750 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun May 23 12:41:55.182750 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) May 23 12:41:55.182750 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func May 23 12:41:55.182750 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self.get_or_create( May 23 12:41:55.182750 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create May 23 12:41:55.182750 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application with Lock( May 23 12:41:55.182750 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ May 23 12:41:55.182750 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self._enter() May 23 12:41:55.182750 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter May 23 12:41:55.182750 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) May 23 12:41:55.182750 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create May 23 12:41:55.182750 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self.creator() May 23 12:41:55.182750 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value May 23 12:41:55.182750 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application created_value = creator( May 23 12:41:55.184902 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project May 23 12:41:55.184902 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) May 23 12:41:55.184902 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project May 23 12:41:55.184902 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() May 23 12:41:55.184902 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project May 23 12:41:55.184902 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) May 23 12:41:55.184902 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. May 23 12:41:55.184902 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application May 23 12:41:55.184902 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 411/824] 10.222.0.30 () {56 vars in 1111 bytes} [Mon May 23 12:41:54 2022] GET /identity/v3/projects/service => generated 88 bytes in 206 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) May 23 12:41:55.184902 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c48d1a3-9780-450f-9798-cc70e75fb4e1 None admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:41:55.243001 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0cb81f54-b680-430b-854d-fae57ea6af2b None admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:41:55.293686 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49505144-68b2-4a82-9e12-485b51c913b3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:41:55.294306 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49505144-68b2-4a82-9e12-485b51c913b3 None admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:41:55.294585 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49505144-68b2-4a82-9e12-485b51c913b3 None admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:41:55.294855 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49505144-68b2-4a82-9e12-485b51c913b3 None admin] PATH_INFO: `/v3/services` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:41:55.295557 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49505144-68b2-4a82-9e12-485b51c913b3 None admin] RBAC: Adding query filter params (type=block-storage) {{(pid=92056) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} May 23 12:41:55.295861 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49505144-68b2-4a82-9e12-485b51c913b3 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:41:55.297233 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49505144-68b2-4a82-9e12-485b51c913b3 None admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:41:55.305960 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 414/825] 10.222.0.30 () {56 vars in 1124 bytes} [Mon May 23 12:41:55 2022] GET /identity/v3/services?type=block-storage => generated 367 bytes in 196 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) May 23 12:41:55.306785 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49505144-68b2-4a82-9e12-485b51c913b3 None admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:41:55.321573 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c48d1a3-9780-450f-9798-cc70e75fb4e1 None admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:41:55.331576 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=100 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:41:55.359799 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99344278-745e-4d5f-ae54-28133a6282e0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:41:55.360648 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99344278-745e-4d5f-ae54-28133a6282e0 None admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:41:55.361022 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99344278-745e-4d5f-ae54-28133a6282e0 None admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:41:55.361347 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99344278-745e-4d5f-ae54-28133a6282e0 None admin] PATH_INFO: `/v3/domains` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:41:55.362329 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99344278-745e-4d5f-ae54-28133a6282e0 None admin] RBAC: Adding query filter params (name=Default) {{(pid=92057) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} May 23 12:41:55.362747 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99344278-745e-4d5f-ae54-28133a6282e0 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:41:55.365617 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99344278-745e-4d5f-ae54-28133a6282e0 None admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:41:55.377934 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 412/826] 10.222.0.30 () {56 vars in 1109 bytes} [Mon May 23 12:41:55 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 194 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) May 23 12:41:55.378843 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99344278-745e-4d5f-ae54-28133a6282e0 None admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:41:55.398524 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=100 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:41:55.415208 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49505144-68b2-4a82-9e12-485b51c913b3 None admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:41:55.454214 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90183984-d324-4d26-bdf1-04c0f3323d9f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:41:55.455682 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90183984-d324-4d26-bdf1-04c0f3323d9f None admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:41:55.455968 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90183984-d324-4d26-bdf1-04c0f3323d9f None admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:41:55.456243 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90183984-d324-4d26-bdf1-04c0f3323d9f None admin] PATH_INFO: `/v3/projects` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:41:55.457022 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90183984-d324-4d26-bdf1-04c0f3323d9f None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=92056) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} May 23 12:41:55.457327 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90183984-d324-4d26-bdf1-04c0f3323d9f None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:41:55.458997 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90183984-d324-4d26-bdf1-04c0f3323d9f None admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:41:55.474110 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 415/827] 10.222.0.30 () {56 vars in 1148 bytes} [Mon May 23 12:41:55 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 168 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) May 23 12:41:55.475128 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90183984-d324-4d26-bdf1-04c0f3323d9f None admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:41:55.486962 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99344278-745e-4d5f-ae54-28133a6282e0 None admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:41:55.527698 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7191095a-5dcc-4199-81d7-7a7d6c941863 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:41:55.528350 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7191095a-5dcc-4199-81d7-7a7d6c941863 None admin] REQUEST_METHOD: `POST` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:41:55.528637 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7191095a-5dcc-4199-81d7-7a7d6c941863 None admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:41:55.528915 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7191095a-5dcc-4199-81d7-7a7d6c941863 None admin] PATH_INFO: `/v3/endpoints` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:41:55.529769 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7191095a-5dcc-4199-81d7-7a7d6c941863 None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:41:55.531298 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7191095a-5dcc-4199-81d7-7a7d6c941863 None admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:41:55.581064 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90183984-d324-4d26-bdf1-04c0f3323d9f None admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:41:55.615480 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c281248f-3d47-4762-a938-c7bdad237546 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:41:55.616337 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c281248f-3d47-4762-a938-c7bdad237546 None admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:41:55.616703 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c281248f-3d47-4762-a938-c7bdad237546 None admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:41:55.616966 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c281248f-3d47-4762-a938-c7bdad237546 None admin] PATH_INFO: `/v3/projects/service` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:41:55.630464 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 413/828] 10.222.0.30 () {60 vars in 1144 bytes} [Mon May 23 12:41:55 2022] POST /identity/v3/endpoints => generated 346 bytes in 244 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) May 23 12:41:55.630464 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7191095a-5dcc-4199-81d7-7a7d6c941863 None admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:41:55.637253 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c281248f-3d47-4762-a938-c7bdad237546 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:41:55.637253 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c281248f-3d47-4762-a938-c7bdad237546 None admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:41:55.646861 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application [None req-c281248f-3d47-4762-a938-c7bdad237546 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. May 23 12:41:55.646861 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application Traceback (most recent call last): May 23 12:41:55.646861 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request May 23 12:41:55.646861 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application rv = self.dispatch_request() May 23 12:41:55.646861 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request May 23 12:41:55.646861 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) May 23 12:41:55.646861 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper May 23 12:41:55.646861 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) May 23 12:41:55.646861 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view May 23 12:41:55.646861 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) May 23 12:41:55.646861 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request May 23 12:41:55.646861 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) May 23 12:41:55.646861 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get May 23 12:41:55.646861 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self._get_project(project_id) May 23 12:41:55.646861 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project May 23 12:41:55.647858 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) May 23 12:41:55.647858 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped May 23 12:41:55.647858 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) May 23 12:41:55.647858 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun May 23 12:41:55.647858 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) May 23 12:41:55.647858 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func May 23 12:41:55.647858 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self.get_or_create( May 23 12:41:55.647858 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create May 23 12:41:55.647858 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application with Lock( May 23 12:41:55.647858 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ May 23 12:41:55.647858 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self._enter() May 23 12:41:55.647858 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter May 23 12:41:55.647858 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) May 23 12:41:55.647858 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create May 23 12:41:55.647858 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self.creator() May 23 12:41:55.647858 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value May 23 12:41:55.647858 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application created_value = creator( May 23 12:41:55.648777 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project May 23 12:41:55.648777 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) May 23 12:41:55.648777 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project May 23 12:41:55.648777 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() May 23 12:41:55.648777 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project May 23 12:41:55.648777 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) May 23 12:41:55.648777 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. May 23 12:41:55.648777 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application May 23 12:41:55.648777 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 416/829] 10.222.0.30 () {56 vars in 1111 bytes} [Mon May 23 12:41:55 2022] GET /identity/v3/projects/service => generated 88 bytes in 173 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) May 23 12:41:55.654333 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c281248f-3d47-4762-a938-c7bdad237546 None admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:41:55.658243 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=100 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:41:55.664899 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:41:55.783947 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7191095a-5dcc-4199-81d7-7a7d6c941863 None admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:41:55.797651 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c281248f-3d47-4762-a938-c7bdad237546 None admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:41:55.837298 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-896c600f-9f7d-4594-8b8c-e7beec04e992 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:41:55.838688 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-896c600f-9f7d-4594-8b8c-e7beec04e992 None admin] REQUEST_METHOD: `PUT` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:41:55.838688 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-896c600f-9f7d-4594-8b8c-e7beec04e992 None admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:41:55.838688 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-896c600f-9f7d-4594-8b8c-e7beec04e992 None admin] PATH_INFO: `/v3/projects/9c389ae65aaa4542bb5a5c4fd0bacf5b/users/6d21a56126fd4ccd837c33ccc08ab609/roles/41ab28c1ae3d4941bd09fb5d35f1dc54` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:41:55.852294 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0fef687-92a8-4b2b-8393-f1a7416a752b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:41:55.852958 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0fef687-92a8-4b2b-8393-f1a7416a752b None admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:41:55.853394 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0fef687-92a8-4b2b-8393-f1a7416a752b None admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:41:55.853671 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0fef687-92a8-4b2b-8393-f1a7416a752b None admin] PATH_INFO: `/v3/projects` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:41:55.854638 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f0fef687-92a8-4b2b-8393-f1a7416a752b None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=92056) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} May 23 12:41:55.854946 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f0fef687-92a8-4b2b-8393-f1a7416a752b None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:41:55.856892 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f0fef687-92a8-4b2b-8393-f1a7416a752b None admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:41:55.859708 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-896c600f-9f7d-4594-8b8c-e7beec04e992 None admin] RBAC: Authorizing `identity:create_grant(project_id=9c389ae65aaa4542bb5a5c4fd0bacf5b, user_id=6d21a56126fd4ccd837c33ccc08ab609, role_id=41ab28c1ae3d4941bd09fb5d35f1dc54)` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:41:55.861157 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-896c600f-9f7d-4594-8b8c-e7beec04e992 None admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:41:55.871262 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 417/830] 10.222.0.30 () {56 vars in 1148 bytes} [Mon May 23 12:41:55 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 217 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) May 23 12:41:55.877541 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0fef687-92a8-4b2b-8393-f1a7416a752b None admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:41:55.933718 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 414/831] 10.222.0.30 () {58 vars in 1439 bytes} [Mon May 23 12:41:55 2022] PUT /identity/v3/projects/9c389ae65aaa4542bb5a5c4fd0bacf5b/users/6d21a56126fd4ccd837c33ccc08ab609/roles/41ab28c1ae3d4941bd09fb5d35f1dc54 => generated 0 bytes in 311 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) May 23 12:41:55.963195 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=100 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:41:56.013561 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0fef687-92a8-4b2b-8393-f1a7416a752b None admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:41:56.057253 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96b6b107-9e22-4e00-904b-7b60b67ed80c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:41:56.057940 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96b6b107-9e22-4e00-904b-7b60b67ed80c None admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:41:56.058349 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96b6b107-9e22-4e00-904b-7b60b67ed80c None admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:41:56.058633 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96b6b107-9e22-4e00-904b-7b60b67ed80c None admin] PATH_INFO: `/v3/role_assignments` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:41:56.059542 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96b6b107-9e22-4e00-904b-7b60b67ed80c None admin] RBAC: Adding query filter params (role.id=bc3c3c9d654940cbaa5177a6be23d1bc, scope.project.id=9c389ae65aaa4542bb5a5c4fd0bacf5b, user.id=cbe9931d94d44cc7ace0e83a7e5015fc) {{(pid=92056) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} May 23 12:41:56.059944 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96b6b107-9e22-4e00-904b-7b60b67ed80c None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:41:56.061445 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96b6b107-9e22-4e00-904b-7b60b67ed80c None admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:41:56.069783 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 418/832] 10.222.0.30 () {56 vars in 1374 bytes} [Mon May 23 12:41:55 2022] GET /identity/v3/role_assignments?user.id=cbe9931d94d44cc7ace0e83a7e5015fc&scope.project.id=9c389ae65aaa4542bb5a5c4fd0bacf5b&role.id=bc3c3c9d654940cbaa5177a6be23d1bc => generated 608 bytes in 193 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) May 23 12:41:56.143924 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c0cbcdf-d826-4cdf-94d4-30d235cd2160 None None] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:41:56.144275 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c0cbcdf-d826-4cdf-94d4-30d235cd2160 None None] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:41:56.144643 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c0cbcdf-d826-4cdf-94d4-30d235cd2160 None None] PATH_INFO: `/` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:41:56.147326 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 415/833] 10.222.0.30 () {54 vars in 926 bytes} [Mon May 23 12:41:56 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) May 23 12:41:56.153853 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83faef20-5f45-49eb-a1e1-23814b2cd47c None None] REQUEST_METHOD: `POST` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:41:56.154217 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83faef20-5f45-49eb-a1e1-23814b2cd47c None None] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:41:56.154496 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83faef20-5f45-49eb-a1e1-23814b2cd47c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:41:56.175751 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.auth.core [None req-83faef20-5f45-49eb-a1e1-23814b2cd47c None None] MFA Rules not processed for user `0f7119f8fa5c4445977671140ded91a4`. Rule list: `[]` (Enabled: `True`). {{(pid=92056) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 12:41:56.177730 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.fernet_utils [None req-83faef20-5f45-49eb-a1e1-23814b2cd47c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92056) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 12:41:56.255861 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 419/834] 10.222.0.30 () {58 vars in 1023 bytes} [Mon May 23 12:41:56 2022] POST /identity/v3/auth/tokens => generated 1151 bytes in 103 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) May 23 12:41:56.280722 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c0cbcdf-d826-4cdf-94d4-30d235cd2160 None None] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:41:56.355992 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:41:56.389188 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c0cbcdf-d826-4cdf-94d4-30d235cd2160 None None] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:41:56.428055 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b211b440-0f54-426f-901f-203bee1a76e4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:41:56.429182 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b211b440-0f54-426f-901f-203bee1a76e4 None admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:41:56.429670 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b211b440-0f54-426f-901f-203bee1a76e4 None admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:41:56.430171 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b211b440-0f54-426f-901f-203bee1a76e4 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:41:56.441214 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b211b440-0f54-426f-901f-203bee1a76e4 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} May 23 12:41:56.449906 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application [None req-b211b440-0f54-426f-901f-203bee1a76e4 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. May 23 12:41:56.449906 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application Traceback (most recent call last): May 23 12:41:56.449906 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request May 23 12:41:56.449906 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application rv = self.dispatch_request() May 23 12:41:56.449906 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request May 23 12:41:56.449906 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) May 23 12:41:56.449906 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper May 23 12:41:56.449906 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) May 23 12:41:56.449906 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view May 23 12:41:56.449906 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) May 23 12:41:56.449906 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request May 23 12:41:56.449906 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) May 23 12:41:56.449906 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get May 23 12:41:56.449906 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self._get_role(role_id) May 23 12:41:56.449906 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role May 23 12:41:56.449906 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') May 23 12:41:56.450860 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call May 23 12:41:56.450860 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( May 23 12:41:56.450860 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data May 23 12:41:56.450860 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) May 23 12:41:56.450860 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped May 23 12:41:56.450860 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) May 23 12:41:56.450860 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun May 23 12:41:56.450860 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) May 23 12:41:56.450860 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func May 23 12:41:56.450860 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self.get_or_create( May 23 12:41:56.450860 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create May 23 12:41:56.450860 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application with Lock( May 23 12:41:56.450860 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ May 23 12:41:56.450860 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self._enter() May 23 12:41:56.450860 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter May 23 12:41:56.450860 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) May 23 12:41:56.451747 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create May 23 12:41:56.451747 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self.creator() May 23 12:41:56.451747 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value May 23 12:41:56.451747 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application created_value = creator( May 23 12:41:56.451747 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role May 23 12:41:56.451747 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) May 23 12:41:56.451747 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role May 23 12:41:56.451747 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() May 23 12:41:56.451747 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role May 23 12:41:56.451747 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) May 23 12:41:56.451747 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. May 23 12:41:56.451747 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application May 23 12:41:56.454602 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 416/835] 10.222.0.30 () {56 vars in 1096 bytes} [Mon May 23 12:41:56 2022] GET /identity/v3/roles/admin => generated 83 bytes in 174 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) May 23 12:41:56.461350 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83faef20-5f45-49eb-a1e1-23814b2cd47c None None] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:41:56.526337 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=100 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:41:56.570723 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83faef20-5f45-49eb-a1e1-23814b2cd47c None None] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:41:56.606196 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4af9d61a-3cb4-4132-a1fb-43c20ef4f4f7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:41:56.606979 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4af9d61a-3cb4-4132-a1fb-43c20ef4f4f7 None admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:41:56.607266 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4af9d61a-3cb4-4132-a1fb-43c20ef4f4f7 None admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:41:56.607544 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4af9d61a-3cb4-4132-a1fb-43c20ef4f4f7 None admin] PATH_INFO: `/v3/roles` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:41:56.608295 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4af9d61a-3cb4-4132-a1fb-43c20ef4f4f7 None admin] RBAC: Adding query filter params (name=admin) {{(pid=92056) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} May 23 12:41:56.608605 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4af9d61a-3cb4-4132-a1fb-43c20ef4f4f7 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:41:56.610161 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4af9d61a-3cb4-4132-a1fb-43c20ef4f4f7 None admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:41:56.619927 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 420/836] 10.222.0.30 () {56 vars in 1099 bytes} [Mon May 23 12:41:56 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 159 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) May 23 12:41:56.625389 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b211b440-0f54-426f-901f-203bee1a76e4 None admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:41:56.732044 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b211b440-0f54-426f-901f-203bee1a76e4 None admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:41:56.762290 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=100 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:41:56.768698 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-057a06cc-b7c1-4f0b-bbfa-48ccea009204 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:41:56.769563 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-057a06cc-b7c1-4f0b-bbfa-48ccea009204 None admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:41:56.769851 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-057a06cc-b7c1-4f0b-bbfa-48ccea009204 None admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:41:56.770238 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-057a06cc-b7c1-4f0b-bbfa-48ccea009204 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:41:56.781598 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-057a06cc-b7c1-4f0b-bbfa-48ccea009204 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:41:56.783098 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-057a06cc-b7c1-4f0b-bbfa-48ccea009204 None admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:41:56.800758 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application [None req-057a06cc-b7c1-4f0b-bbfa-48ccea009204 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. May 23 12:41:56.800758 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application Traceback (most recent call last): May 23 12:41:56.800758 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request May 23 12:41:56.800758 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application rv = self.dispatch_request() May 23 12:41:56.800758 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request May 23 12:41:56.800758 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) May 23 12:41:56.800758 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper May 23 12:41:56.800758 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) May 23 12:41:56.800758 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view May 23 12:41:56.800758 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) May 23 12:41:56.800758 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request May 23 12:41:56.800758 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) May 23 12:41:56.800758 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get May 23 12:41:56.800758 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self._get_domain(domain_id) May 23 12:41:56.800758 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain May 23 12:41:56.803327 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) May 23 12:41:56.803327 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped May 23 12:41:56.803327 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) May 23 12:41:56.803327 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun May 23 12:41:56.803327 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) May 23 12:41:56.803327 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func May 23 12:41:56.803327 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self.get_or_create( May 23 12:41:56.803327 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create May 23 12:41:56.803327 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application with Lock( May 23 12:41:56.803327 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ May 23 12:41:56.803327 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self._enter() May 23 12:41:56.803327 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter May 23 12:41:56.803327 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) May 23 12:41:56.803327 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create May 23 12:41:56.803327 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self.creator() May 23 12:41:56.803327 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value May 23 12:41:56.803327 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application created_value = creator( May 23 12:41:56.804272 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain May 23 12:41:56.804272 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) May 23 12:41:56.804272 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. May 23 12:41:56.804272 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application May 23 12:41:56.804272 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 417/837] 10.222.0.30 () {56 vars in 1108 bytes} [Mon May 23 12:41:56 2022] GET /identity/v3/domains/Default => generated 87 bytes in 178 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) May 23 12:41:56.808372 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4af9d61a-3cb4-4132-a1fb-43c20ef4f4f7 None admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:41:56.917282 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4af9d61a-3cb4-4132-a1fb-43c20ef4f4f7 None admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:41:56.938742 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=100 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:41:56.955870 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e86ad82-2ab3-43c1-b498-1f00f6322faf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:41:56.955870 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e86ad82-2ab3-43c1-b498-1f00f6322faf None admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:41:56.955870 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e86ad82-2ab3-43c1-b498-1f00f6322faf None admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:41:56.955870 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e86ad82-2ab3-43c1-b498-1f00f6322faf None admin] PATH_INFO: `/v3/domains` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:41:56.956246 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e86ad82-2ab3-43c1-b498-1f00f6322faf None admin] RBAC: Adding query filter params (name=Default) {{(pid=92056) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} May 23 12:41:56.956246 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e86ad82-2ab3-43c1-b498-1f00f6322faf None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:41:56.956622 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e86ad82-2ab3-43c1-b498-1f00f6322faf None admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:41:56.969575 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 421/838] 10.222.0.30 () {56 vars in 1109 bytes} [Mon May 23 12:41:56 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 162 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) May 23 12:41:56.976069 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-057a06cc-b7c1-4f0b-bbfa-48ccea009204 None admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:41:57.084454 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-057a06cc-b7c1-4f0b-bbfa-48ccea009204 None admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:41:57.119966 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fce2bc54-bdd4-421b-9411-60107b0b8f88 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:41:57.120758 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fce2bc54-bdd4-421b-9411-60107b0b8f88 None admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:41:57.121042 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fce2bc54-bdd4-421b-9411-60107b0b8f88 None admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:41:57.121316 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fce2bc54-bdd4-421b-9411-60107b0b8f88 None admin] PATH_INFO: `/v3/users/swift` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:41:57.129078 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fce2bc54-bdd4-421b-9411-60107b0b8f88 None admin] RBAC: Authorizing `identity:get_user(user_id=swift)` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:41:57.131336 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fce2bc54-bdd4-421b-9411-60107b0b8f88 None admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:41:57.154468 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application [None req-fce2bc54-bdd4-421b-9411-60107b0b8f88 None admin] Could not find user: swift.: keystone.exception.UserNotFound: Could not find user: swift. May 23 12:41:57.154468 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application Traceback (most recent call last): May 23 12:41:57.154468 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request May 23 12:41:57.154468 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application rv = self.dispatch_request() May 23 12:41:57.154468 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request May 23 12:41:57.154468 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) May 23 12:41:57.154468 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper May 23 12:41:57.154468 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) May 23 12:41:57.154468 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view May 23 12:41:57.154468 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) May 23 12:41:57.154468 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request May 23 12:41:57.154468 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) May 23 12:41:57.154468 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get May 23 12:41:57.154468 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self._get_user(user_id) May 23 12:41:57.154468 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user May 23 12:41:57.154468 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) May 23 12:41:57.155499 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped May 23 12:41:57.155499 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) May 23 12:41:57.155499 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper May 23 12:41:57.155499 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) May 23 12:41:57.155499 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper May 23 12:41:57.155499 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) May 23 12:41:57.155499 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun May 23 12:41:57.155499 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) May 23 12:41:57.155499 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func May 23 12:41:57.155499 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self.get_or_create( May 23 12:41:57.155499 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create May 23 12:41:57.155499 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application with Lock( May 23 12:41:57.155499 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ May 23 12:41:57.155499 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self._enter() May 23 12:41:57.155499 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter May 23 12:41:57.155499 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) May 23 12:41:57.155499 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create May 23 12:41:57.155499 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self.creator() May 23 12:41:57.156489 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value May 23 12:41:57.156489 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application created_value = creator( May 23 12:41:57.156489 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user May 23 12:41:57.156489 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) May 23 12:41:57.156489 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user May 23 12:41:57.156489 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) May 23 12:41:57.156489 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user May 23 12:41:57.156489 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) May 23 12:41:57.156489 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: swift. May 23 12:41:57.156489 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application May 23 12:41:57.156489 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 418/839] 10.222.0.30 () {56 vars in 1096 bytes} [Mon May 23 12:41:56 2022] GET /identity/v3/users/swift => generated 83 bytes in 181 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) May 23 12:41:57.162682 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e86ad82-2ab3-43c1-b498-1f00f6322faf None admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:41:57.288452 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e86ad82-2ab3-43c1-b498-1f00f6322faf None admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:41:57.328103 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e0c7441-ae3e-472a-bdca-b3b57ee4c8e5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:41:57.328774 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e0c7441-ae3e-472a-bdca-b3b57ee4c8e5 None admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:41:57.329057 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e0c7441-ae3e-472a-bdca-b3b57ee4c8e5 None admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:41:57.329337 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e0c7441-ae3e-472a-bdca-b3b57ee4c8e5 None admin] PATH_INFO: `/v3/users` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:41:57.330204 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e0c7441-ae3e-472a-bdca-b3b57ee4c8e5 None admin] RBAC: Adding query filter params (domain_id=default, name=swift) {{(pid=92056) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} May 23 12:41:57.330521 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e0c7441-ae3e-472a-bdca-b3b57ee4c8e5 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:41:57.331957 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e0c7441-ae3e-472a-bdca-b3b57ee4c8e5 None admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:41:57.352071 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 422/840] 10.222.0.30 () {56 vars in 1135 bytes} [Mon May 23 12:41:57 2022] GET /identity/v3/users?domain_id=default&name=swift => generated 370 bytes in 190 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) May 23 12:41:57.357827 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fce2bc54-bdd4-421b-9411-60107b0b8f88 None admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:41:57.404529 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=100 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:41:57.474758 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fce2bc54-bdd4-421b-9411-60107b0b8f88 None admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:41:57.517131 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ad8da70-f287-4324-801e-8b8bf3c27e88 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:41:57.517786 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ad8da70-f287-4324-801e-8b8bf3c27e88 None admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:41:57.518104 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ad8da70-f287-4324-801e-8b8bf3c27e88 None admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:41:57.518384 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ad8da70-f287-4324-801e-8b8bf3c27e88 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:41:57.591112 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ad8da70-f287-4324-801e-8b8bf3c27e88 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:41:57.592509 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ad8da70-f287-4324-801e-8b8bf3c27e88 None admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:41:57.637218 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=100 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:41:57.714229 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 419/841] 10.222.0.30 () {58 vars in 1282 bytes} [Mon May 23 12:41:57 2022] GET /identity/v3/auth/tokens => generated 1151 bytes in 357 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) May 23 12:41:57.719711 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e0c7441-ae3e-472a-bdca-b3b57ee4c8e5 None admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:41:57.763324 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=100 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:41:57.828159 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e0c7441-ae3e-472a-bdca-b3b57ee4c8e5 None admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:41:57.870273 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e1e7b9d-8cd5-4c22-ac74-2dc6c237965d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:41:57.871136 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e1e7b9d-8cd5-4c22-ac74-2dc6c237965d None admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:41:57.871422 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e1e7b9d-8cd5-4c22-ac74-2dc6c237965d None admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:41:57.871698 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e1e7b9d-8cd5-4c22-ac74-2dc6c237965d None admin] PATH_INFO: `/v3/domains/Default` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:41:57.883203 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e1e7b9d-8cd5-4c22-ac74-2dc6c237965d None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:41:57.884731 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e1e7b9d-8cd5-4c22-ac74-2dc6c237965d None admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:41:57.903273 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application [None req-2e1e7b9d-8cd5-4c22-ac74-2dc6c237965d None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. May 23 12:41:57.903273 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application Traceback (most recent call last): May 23 12:41:57.903273 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request May 23 12:41:57.903273 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application rv = self.dispatch_request() May 23 12:41:57.903273 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request May 23 12:41:57.903273 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) May 23 12:41:57.903273 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper May 23 12:41:57.903273 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) May 23 12:41:57.903273 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view May 23 12:41:57.903273 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) May 23 12:41:57.903273 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request May 23 12:41:57.903273 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) May 23 12:41:57.903273 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get May 23 12:41:57.903273 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self._get_domain(domain_id) May 23 12:41:57.903273 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain May 23 12:41:57.904303 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) May 23 12:41:57.904303 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped May 23 12:41:57.904303 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) May 23 12:41:57.904303 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun May 23 12:41:57.904303 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) May 23 12:41:57.904303 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func May 23 12:41:57.904303 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self.get_or_create( May 23 12:41:57.904303 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create May 23 12:41:57.904303 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application with Lock( May 23 12:41:57.904303 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ May 23 12:41:57.904303 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self._enter() May 23 12:41:57.904303 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter May 23 12:41:57.904303 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) May 23 12:41:57.904303 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create May 23 12:41:57.904303 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self.creator() May 23 12:41:57.904303 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value May 23 12:41:57.904303 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application created_value = creator( May 23 12:41:57.905307 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain May 23 12:41:57.905307 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) May 23 12:41:57.905307 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. May 23 12:41:57.905307 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application May 23 12:41:57.906018 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 423/842] 10.222.0.30 () {56 vars in 1108 bytes} [Mon May 23 12:41:57 2022] GET /identity/v3/domains/Default => generated 87 bytes in 186 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) May 23 12:41:57.912199 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ad8da70-f287-4324-801e-8b8bf3c27e88 None admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:41:58.016843 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ad8da70-f287-4324-801e-8b8bf3c27e88 None admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:41:58.048023 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a413e797-cf0b-496b-bb11-44148a8e1894 None None] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:41:58.048745 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a413e797-cf0b-496b-bb11-44148a8e1894 None None] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:41:58.049097 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a413e797-cf0b-496b-bb11-44148a8e1894 None None] PATH_INFO: `/` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:41:58.050883 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 424/843] 10.222.0.30 () {54 vars in 926 bytes} [Mon May 23 12:41:58 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) May 23 12:41:58.055639 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-962e04ca-0842-49df-8ac7-5e749f160886 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:41:58.056665 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-962e04ca-0842-49df-8ac7-5e749f160886 None admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:41:58.056809 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-faef366f-0f63-4fc2-807c-31778f232f92 None None] REQUEST_METHOD: `POST` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:41:58.057175 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-faef366f-0f63-4fc2-807c-31778f232f92 None None] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:41:58.057511 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-962e04ca-0842-49df-8ac7-5e749f160886 None admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:41:58.057622 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-faef366f-0f63-4fc2-807c-31778f232f92 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:41:58.058084 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-962e04ca-0842-49df-8ac7-5e749f160886 None admin] PATH_INFO: `/v3/domains` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:41:58.058981 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-962e04ca-0842-49df-8ac7-5e749f160886 None admin] RBAC: Adding query filter params (name=Default) {{(pid=92057) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} May 23 12:41:58.059568 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-962e04ca-0842-49df-8ac7-5e749f160886 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:41:58.061291 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-962e04ca-0842-49df-8ac7-5e749f160886 None admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:41:58.076525 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 420/844] 10.222.0.30 () {56 vars in 1109 bytes} [Mon May 23 12:41:57 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 165 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) May 23 12:41:58.077161 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.auth.core [None req-faef366f-0f63-4fc2-807c-31778f232f92 None None] MFA Rules not processed for user `0f7119f8fa5c4445977671140ded91a4`. Rule list: `[]` (Enabled: `True`). {{(pid=92056) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 12:41:58.079094 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.fernet_utils [None req-faef366f-0f63-4fc2-807c-31778f232f92 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92056) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 12:41:58.083406 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-962e04ca-0842-49df-8ac7-5e749f160886 None admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:41:58.092167 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=100 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:41:58.178895 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 425/845] 10.222.0.30 () {58 vars in 1023 bytes} [Mon May 23 12:41:58 2022] POST /identity/v3/auth/tokens => generated 1151 bytes in 123 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) May 23 12:41:58.200647 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-faef366f-0f63-4fc2-807c-31778f232f92 None None] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:41:58.210744 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-962e04ca-0842-49df-8ac7-5e749f160886 None admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:41:58.226288 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=100 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:41:58.252235 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6bc78d5c-5777-45ec-bf10-8fe04e81a96c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:41:58.253466 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bc78d5c-5777-45ec-bf10-8fe04e81a96c None admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:41:58.254058 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bc78d5c-5777-45ec-bf10-8fe04e81a96c None admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:41:58.254641 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bc78d5c-5777-45ec-bf10-8fe04e81a96c None admin] PATH_INFO: `/v3/projects/service` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:41:58.264963 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6bc78d5c-5777-45ec-bf10-8fe04e81a96c None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:41:58.267019 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6bc78d5c-5777-45ec-bf10-8fe04e81a96c None admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:41:58.287352 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application [None req-6bc78d5c-5777-45ec-bf10-8fe04e81a96c None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. May 23 12:41:58.287352 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application Traceback (most recent call last): May 23 12:41:58.287352 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request May 23 12:41:58.287352 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application rv = self.dispatch_request() May 23 12:41:58.287352 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request May 23 12:41:58.287352 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) May 23 12:41:58.287352 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper May 23 12:41:58.287352 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) May 23 12:41:58.287352 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view May 23 12:41:58.287352 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) May 23 12:41:58.287352 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request May 23 12:41:58.287352 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) May 23 12:41:58.287352 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get May 23 12:41:58.287352 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self._get_project(project_id) May 23 12:41:58.287352 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project May 23 12:41:58.288373 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) May 23 12:41:58.288373 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped May 23 12:41:58.288373 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) May 23 12:41:58.288373 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun May 23 12:41:58.288373 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) May 23 12:41:58.288373 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func May 23 12:41:58.288373 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self.get_or_create( May 23 12:41:58.288373 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create May 23 12:41:58.288373 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application with Lock( May 23 12:41:58.288373 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ May 23 12:41:58.288373 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self._enter() May 23 12:41:58.288373 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter May 23 12:41:58.288373 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) May 23 12:41:58.288373 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create May 23 12:41:58.288373 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self.creator() May 23 12:41:58.288373 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value May 23 12:41:58.288373 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application created_value = creator( May 23 12:41:58.289295 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project May 23 12:41:58.289295 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) May 23 12:41:58.289295 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project May 23 12:41:58.289295 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() May 23 12:41:58.289295 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project May 23 12:41:58.289295 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) May 23 12:41:58.289295 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. May 23 12:41:58.289295 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application May 23 12:41:58.296992 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 421/846] 10.222.0.30 () {56 vars in 1111 bytes} [Mon May 23 12:41:58 2022] GET /identity/v3/projects/service => generated 88 bytes in 214 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) May 23 12:41:58.298352 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6bc78d5c-5777-45ec-bf10-8fe04e81a96c None admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:41:58.333481 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-faef366f-0f63-4fc2-807c-31778f232f92 None None] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:41:58.369525 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=100 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:41:58.383838 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9044da82-45c8-4365-96e6-65b05c5fa84a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:41:58.384639 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9044da82-45c8-4365-96e6-65b05c5fa84a None admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:41:58.388050 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9044da82-45c8-4365-96e6-65b05c5fa84a None admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:41:58.388050 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9044da82-45c8-4365-96e6-65b05c5fa84a None admin] PATH_INFO: `/v3/services/volumev3` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:41:58.388050 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9044da82-45c8-4365-96e6-65b05c5fa84a None admin] RBAC: Authorizing `identity:get_service(service_id=volumev3)` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:41:58.388050 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9044da82-45c8-4365-96e6-65b05c5fa84a None admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:41:58.405456 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application [None req-9044da82-45c8-4365-96e6-65b05c5fa84a None admin] Could not find service: volumev3.: keystone.exception.ServiceNotFound: Could not find service: volumev3. May 23 12:41:58.405456 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application Traceback (most recent call last): May 23 12:41:58.405456 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service May 23 12:41:58.405456 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) May 23 12:41:58.405456 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service May 23 12:41:58.405456 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() May 23 12:41:58.405456 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service May 23 12:41:58.405456 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) May 23 12:41:58.405456 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: volumev3. May 23 12:41:58.405456 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application May 23 12:41:58.405456 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: May 23 12:41:58.405456 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application May 23 12:41:58.405456 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application Traceback (most recent call last): May 23 12:41:58.405456 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request May 23 12:41:58.405456 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application rv = self.dispatch_request() May 23 12:41:58.405456 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request May 23 12:41:58.405456 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) May 23 12:41:58.406640 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper May 23 12:41:58.406640 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) May 23 12:41:58.406640 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view May 23 12:41:58.406640 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) May 23 12:41:58.406640 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request May 23 12:41:58.406640 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) May 23 12:41:58.406640 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get May 23 12:41:58.406640 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self._get_service(service_id) May 23 12:41:58.406640 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service May 23 12:41:58.406640 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) May 23 12:41:58.406640 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped May 23 12:41:58.406640 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) May 23 12:41:58.406640 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun May 23 12:41:58.406640 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) May 23 12:41:58.406640 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func May 23 12:41:58.406640 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self.get_or_create( May 23 12:41:58.407538 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create May 23 12:41:58.407538 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application with Lock( May 23 12:41:58.407538 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ May 23 12:41:58.407538 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self._enter() May 23 12:41:58.407538 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter May 23 12:41:58.407538 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) May 23 12:41:58.407538 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create May 23 12:41:58.407538 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self.creator() May 23 12:41:58.407538 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value May 23 12:41:58.407538 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application created_value = creator( May 23 12:41:58.407538 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service May 23 12:41:58.407538 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) May 23 12:41:58.407538 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: volumev3. May 23 12:41:58.407538 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application May 23 12:41:58.416258 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 426/847] 10.222.0.30 () {56 vars in 1114 bytes} [Mon May 23 12:41:58 2022] GET /identity/v3/services/volumev3 => generated 89 bytes in 216 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) May 23 12:41:58.417438 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9044da82-45c8-4365-96e6-65b05c5fa84a None admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:41:58.418858 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6bc78d5c-5777-45ec-bf10-8fe04e81a96c None admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:41:58.460458 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1221cfbb-92e7-4a10-8969-a272391c36f3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:41:58.461488 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1221cfbb-92e7-4a10-8969-a272391c36f3 None admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:41:58.461803 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1221cfbb-92e7-4a10-8969-a272391c36f3 None admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:41:58.462200 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1221cfbb-92e7-4a10-8969-a272391c36f3 None admin] PATH_INFO: `/v3/projects` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:41:58.463395 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1221cfbb-92e7-4a10-8969-a272391c36f3 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=92057) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} May 23 12:41:58.463773 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1221cfbb-92e7-4a10-8969-a272391c36f3 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:41:58.465753 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1221cfbb-92e7-4a10-8969-a272391c36f3 None admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:41:58.478353 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 422/848] 10.222.0.30 () {56 vars in 1148 bytes} [Mon May 23 12:41:58 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 180 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) May 23 12:41:58.479631 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa354594-c082-4509-8efa-3228e796b4e2 None None] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:41:58.479929 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa354594-c082-4509-8efa-3228e796b4e2 None None] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:41:58.480206 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa354594-c082-4509-8efa-3228e796b4e2 None None] PATH_INFO: `/` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:41:58.482013 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 423/849] 10.222.0.30 () {54 vars in 926 bytes} [Mon May 23 12:41:58 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) May 23 12:41:58.483526 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb4f4943-431e-4993-b51b-c230324bf8ae None None] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:41:58.484316 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb4f4943-431e-4993-b51b-c230324bf8ae None None] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:41:58.485051 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb4f4943-431e-4993-b51b-c230324bf8ae None None] PATH_INFO: `/` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:41:58.486512 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 424/850] 10.222.0.30 () {54 vars in 926 bytes} [Mon May 23 12:41:58 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) May 23 12:41:58.487566 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb4f4943-431e-4993-b51b-c230324bf8ae None None] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:41:58.507711 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=100 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:41:58.524340 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9044da82-45c8-4365-96e6-65b05c5fa84a None admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:41:58.562179 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46e6e3c0-a365-4c00-9813-ffa8ee01386d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:41:58.562949 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46e6e3c0-a365-4c00-9813-ffa8ee01386d None admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:41:58.563240 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46e6e3c0-a365-4c00-9813-ffa8ee01386d None admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:41:58.563518 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46e6e3c0-a365-4c00-9813-ffa8ee01386d None admin] PATH_INFO: `/v3/services` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:41:58.564217 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46e6e3c0-a365-4c00-9813-ffa8ee01386d None admin] RBAC: Adding query filter params (name=volumev3) {{(pid=92056) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} May 23 12:41:58.564524 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46e6e3c0-a365-4c00-9813-ffa8ee01386d None admin] RBAC: Authorizing `identity:list_services()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:41:58.565925 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46e6e3c0-a365-4c00-9813-ffa8ee01386d None admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:41:58.571129 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 427/851] 10.222.0.30 () {56 vars in 1114 bytes} [Mon May 23 12:41:58 2022] GET /identity/v3/services?name=volumev3 => generated 125 bytes in 154 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) May 23 12:41:58.572350 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-add80b7b-8c1f-4e1c-aa8d-b70c074efaaa None None] REQUEST_METHOD: `POST` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:41:58.572632 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-add80b7b-8c1f-4e1c-aa8d-b70c074efaaa None None] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:41:58.572909 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-add80b7b-8c1f-4e1c-aa8d-b70c074efaaa None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:41:58.588826 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.auth.core [None req-add80b7b-8c1f-4e1c-aa8d-b70c074efaaa None None] MFA Rules not processed for user `0f7119f8fa5c4445977671140ded91a4`. Rule list: `[]` (Enabled: `True`). {{(pid=92056) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 12:41:58.590035 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.fernet_utils [None req-add80b7b-8c1f-4e1c-aa8d-b70c074efaaa None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92056) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 12:41:58.610832 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb4f4943-431e-4993-b51b-c230324bf8ae None None] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:41:58.633715 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=100 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:41:58.645775 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a31237c9-36e6-492b-aa96-d3102396f62b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:41:58.656950 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a31237c9-36e6-492b-aa96-d3102396f62b None admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:41:58.656950 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a31237c9-36e6-492b-aa96-d3102396f62b None admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:41:58.656950 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a31237c9-36e6-492b-aa96-d3102396f62b None admin] PATH_INFO: `/v3/role_assignments` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:41:58.656950 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a31237c9-36e6-492b-aa96-d3102396f62b None admin] RBAC: Adding query filter params (role.id=bc3c3c9d654940cbaa5177a6be23d1bc, scope.project.id=9c389ae65aaa4542bb5a5c4fd0bacf5b, user.id=138d91edb99f450e846765f9eef00d62) {{(pid=92057) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} May 23 12:41:58.656950 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a31237c9-36e6-492b-aa96-d3102396f62b None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:41:58.656950 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a31237c9-36e6-492b-aa96-d3102396f62b None admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:41:58.657552 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 425/852] 10.222.0.30 () {56 vars in 1374 bytes} [Mon May 23 12:41:58 2022] GET /identity/v3/role_assignments?user.id=138d91edb99f450e846765f9eef00d62&scope.project.id=9c389ae65aaa4542bb5a5c4fd0bacf5b&role.id=bc3c3c9d654940cbaa5177a6be23d1bc => generated 608 bytes in 168 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) May 23 12:41:58.657552 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29f451f4-8a4c-4be8-bd32-e38f2d6996a4 None None] REQUEST_METHOD: `POST` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:41:58.657552 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29f451f4-8a4c-4be8-bd32-e38f2d6996a4 None None] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:41:58.657552 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29f451f4-8a4c-4be8-bd32-e38f2d6996a4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:41:58.663315 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 428/853] 10.222.0.30 () {58 vars in 1023 bytes} [Mon May 23 12:41:58 2022] POST /identity/v3/auth/tokens => generated 1151 bytes in 92 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) May 23 12:41:58.664577 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-add80b7b-8c1f-4e1c-aa8d-b70c074efaaa None None] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:41:58.697712 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.auth.core [None req-29f451f4-8a4c-4be8-bd32-e38f2d6996a4 None None] MFA Rules not processed for user `0f7119f8fa5c4445977671140ded91a4`. Rule list: `[]` (Enabled: `True`). {{(pid=92057) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 12:41:58.698632 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.fernet_utils [None req-29f451f4-8a4c-4be8-bd32-e38f2d6996a4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92057) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 12:41:58.767886 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=100 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:41:58.787189 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 426/854] 10.222.0.30 () {58 vars in 1023 bytes} [Mon May 23 12:41:58 2022] POST /identity/v3/auth/tokens => generated 1151 bytes in 131 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) May 23 12:41:58.789068 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29f451f4-8a4c-4be8-bd32-e38f2d6996a4 None None] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:41:58.805148 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-add80b7b-8c1f-4e1c-aa8d-b70c074efaaa None None] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:41:58.846907 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3f509a2-a1ea-4c4c-9e1b-d07764853ae2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:41:58.847576 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3f509a2-a1ea-4c4c-9e1b-d07764853ae2 None admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:41:58.847857 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3f509a2-a1ea-4c4c-9e1b-d07764853ae2 None admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:41:58.848162 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3f509a2-a1ea-4c4c-9e1b-d07764853ae2 None admin] PATH_INFO: `/v3/services` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:41:58.848866 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3f509a2-a1ea-4c4c-9e1b-d07764853ae2 None admin] RBAC: Adding query filter params (type=volumev3) {{(pid=92056) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} May 23 12:41:58.849348 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3f509a2-a1ea-4c4c-9e1b-d07764853ae2 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:41:58.851015 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3f509a2-a1ea-4c4c-9e1b-d07764853ae2 None admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:41:58.856009 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 429/855] 10.222.0.30 () {56 vars in 1114 bytes} [Mon May 23 12:41:58 2022] GET /identity/v3/services?type=volumev3 => generated 125 bytes in 192 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) May 23 12:41:58.856647 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3f509a2-a1ea-4c4c-9e1b-d07764853ae2 None admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:41:58.921246 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29f451f4-8a4c-4be8-bd32-e38f2d6996a4 None None] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:41:58.949344 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=100 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:41:58.970787 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e11066e1-44de-4683-9722-40e916a4997e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:41:58.971603 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e11066e1-44de-4683-9722-40e916a4997e None admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:41:58.971891 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e11066e1-44de-4683-9722-40e916a4997e None admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:41:58.972170 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e11066e1-44de-4683-9722-40e916a4997e None admin] PATH_INFO: `/v3/roles/service` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:41:58.986558 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e11066e1-44de-4683-9722-40e916a4997e None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} May 23 12:41:58.994742 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application [None req-e11066e1-44de-4683-9722-40e916a4997e None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. May 23 12:41:58.994742 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application Traceback (most recent call last): May 23 12:41:58.994742 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request May 23 12:41:58.994742 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application rv = self.dispatch_request() May 23 12:41:58.994742 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request May 23 12:41:58.994742 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) May 23 12:41:58.994742 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper May 23 12:41:58.994742 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) May 23 12:41:58.994742 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view May 23 12:41:58.994742 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) May 23 12:41:58.994742 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request May 23 12:41:58.994742 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) May 23 12:41:58.994742 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get May 23 12:41:58.994742 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self._get_role(role_id) May 23 12:41:58.994742 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role May 23 12:41:58.994742 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') May 23 12:41:58.995791 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call May 23 12:41:58.995791 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( May 23 12:41:58.995791 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data May 23 12:41:58.995791 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) May 23 12:41:58.995791 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped May 23 12:41:58.995791 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) May 23 12:41:58.995791 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun May 23 12:41:58.995791 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) May 23 12:41:58.995791 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func May 23 12:41:58.995791 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self.get_or_create( May 23 12:41:58.995791 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create May 23 12:41:58.995791 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application with Lock( May 23 12:41:58.995791 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ May 23 12:41:58.995791 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self._enter() May 23 12:41:58.995791 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter May 23 12:41:58.995791 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) May 23 12:41:58.996663 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create May 23 12:41:58.996663 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self.creator() May 23 12:41:58.996663 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value May 23 12:41:58.996663 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application created_value = creator( May 23 12:41:58.996663 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role May 23 12:41:58.996663 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) May 23 12:41:58.996663 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role May 23 12:41:58.996663 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() May 23 12:41:58.996663 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role May 23 12:41:58.996663 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) May 23 12:41:58.996663 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. May 23 12:41:58.996663 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application May 23 12:41:58.996663 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 427/856] 10.222.0.30 () {56 vars in 1102 bytes} [Mon May 23 12:41:58 2022] GET /identity/v3/roles/service => generated 85 bytes in 208 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) May 23 12:41:59.004038 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e11066e1-44de-4683-9722-40e916a4997e None admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:41:59.016380 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3f509a2-a1ea-4c4c-9e1b-d07764853ae2 None admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:41:59.070506 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13b01118-66dd-465d-bc3a-3c4a16466e1e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:41:59.071247 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13b01118-66dd-465d-bc3a-3c4a16466e1e None admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:41:59.071530 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13b01118-66dd-465d-bc3a-3c4a16466e1e None admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:41:59.071809 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13b01118-66dd-465d-bc3a-3c4a16466e1e None admin] PATH_INFO: `/v3/services/compute_legacy` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:41:59.072470 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13b01118-66dd-465d-bc3a-3c4a16466e1e None admin] RBAC: Authorizing `identity:get_service(service_id=compute_legacy)` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:41:59.073850 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13b01118-66dd-465d-bc3a-3c4a16466e1e None admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:41:59.091306 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application [None req-13b01118-66dd-465d-bc3a-3c4a16466e1e None admin] Could not find service: compute_legacy.: keystone.exception.ServiceNotFound: Could not find service: compute_legacy. May 23 12:41:59.091306 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application Traceback (most recent call last): May 23 12:41:59.091306 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service May 23 12:41:59.091306 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) May 23 12:41:59.091306 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service May 23 12:41:59.091306 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() May 23 12:41:59.091306 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service May 23 12:41:59.091306 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) May 23 12:41:59.091306 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute_legacy. May 23 12:41:59.091306 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application May 23 12:41:59.091306 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: May 23 12:41:59.091306 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application May 23 12:41:59.091306 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application Traceback (most recent call last): May 23 12:41:59.091306 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request May 23 12:41:59.091306 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application rv = self.dispatch_request() May 23 12:41:59.091306 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request May 23 12:41:59.092328 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) May 23 12:41:59.092328 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper May 23 12:41:59.092328 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) May 23 12:41:59.092328 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view May 23 12:41:59.092328 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) May 23 12:41:59.092328 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request May 23 12:41:59.092328 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) May 23 12:41:59.092328 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get May 23 12:41:59.092328 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self._get_service(service_id) May 23 12:41:59.092328 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service May 23 12:41:59.092328 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) May 23 12:41:59.092328 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped May 23 12:41:59.092328 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) May 23 12:41:59.092328 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun May 23 12:41:59.092328 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) May 23 12:41:59.092328 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func May 23 12:41:59.093193 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self.get_or_create( May 23 12:41:59.093193 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create May 23 12:41:59.093193 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application with Lock( May 23 12:41:59.093193 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ May 23 12:41:59.093193 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self._enter() May 23 12:41:59.093193 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter May 23 12:41:59.093193 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) May 23 12:41:59.093193 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create May 23 12:41:59.093193 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self.creator() May 23 12:41:59.093193 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value May 23 12:41:59.093193 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application created_value = creator( May 23 12:41:59.093193 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service May 23 12:41:59.093193 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) May 23 12:41:59.093193 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute_legacy. May 23 12:41:59.093193 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application May 23 12:41:59.093193 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 430/857] 10.222.0.30 () {56 vars in 1132 bytes} [Mon May 23 12:41:58 2022] GET /identity/v3/services/compute_legacy => generated 95 bytes in 236 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) May 23 12:41:59.099669 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13b01118-66dd-465d-bc3a-3c4a16466e1e None admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:41:59.109089 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=100 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:41:59.141912 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e11066e1-44de-4683-9722-40e916a4997e None admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:41:59.178960 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-449d2cac-b879-4e7c-a41b-30603e5f3608 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:41:59.179635 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-449d2cac-b879-4e7c-a41b-30603e5f3608 None admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:41:59.180308 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-449d2cac-b879-4e7c-a41b-30603e5f3608 None admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:41:59.180308 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-449d2cac-b879-4e7c-a41b-30603e5f3608 None admin] PATH_INFO: `/v3/roles` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:41:59.181046 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-449d2cac-b879-4e7c-a41b-30603e5f3608 None admin] RBAC: Adding query filter params (name=service) {{(pid=92057) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} May 23 12:41:59.181364 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-449d2cac-b879-4e7c-a41b-30603e5f3608 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:41:59.183021 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-449d2cac-b879-4e7c-a41b-30603e5f3608 None admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:41:59.192716 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 428/858] 10.222.0.30 () {56 vars in 1103 bytes} [Mon May 23 12:41:59 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 189 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) May 23 12:41:59.199678 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-449d2cac-b879-4e7c-a41b-30603e5f3608 None admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:41:59.208966 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13b01118-66dd-465d-bc3a-3c4a16466e1e None admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:41:59.245591 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1bb1d5fd-670a-45d2-9e7c-29efbe70aa35 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:41:59.246619 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bb1d5fd-670a-45d2-9e7c-29efbe70aa35 None admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:41:59.247064 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bb1d5fd-670a-45d2-9e7c-29efbe70aa35 None admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:41:59.247475 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bb1d5fd-670a-45d2-9e7c-29efbe70aa35 None admin] PATH_INFO: `/v3/services` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:41:59.248389 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1bb1d5fd-670a-45d2-9e7c-29efbe70aa35 None admin] RBAC: Adding query filter params (name=compute_legacy) {{(pid=92056) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} May 23 12:41:59.248695 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1bb1d5fd-670a-45d2-9e7c-29efbe70aa35 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:41:59.250100 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1bb1d5fd-670a-45d2-9e7c-29efbe70aa35 None admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:41:59.260805 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=100 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:41:59.261683 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 431/859] 10.222.0.30 () {56 vars in 1126 bytes} [Mon May 23 12:41:59 2022] GET /identity/v3/services?name=compute_legacy => generated 131 bytes in 158 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) May 23 12:41:59.263156 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1bb1d5fd-670a-45d2-9e7c-29efbe70aa35 None admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:41:59.335510 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-449d2cac-b879-4e7c-a41b-30603e5f3608 None admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:41:59.338645 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=100 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:41:59.372782 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f7fe292-eea1-43ba-849f-ab090d5c3071 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:41:59.373309 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1bb1d5fd-670a-45d2-9e7c-29efbe70aa35 None admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:41:59.373804 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f7fe292-eea1-43ba-849f-ab090d5c3071 None admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:41:59.374133 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f7fe292-eea1-43ba-849f-ab090d5c3071 None admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:41:59.374409 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f7fe292-eea1-43ba-849f-ab090d5c3071 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:41:59.389986 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f7fe292-eea1-43ba-849f-ab090d5c3071 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:41:59.389986 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f7fe292-eea1-43ba-849f-ab090d5c3071 None admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:41:59.409006 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application [None req-5f7fe292-eea1-43ba-849f-ab090d5c3071 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. May 23 12:41:59.409006 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application Traceback (most recent call last): May 23 12:41:59.409006 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request May 23 12:41:59.409006 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application rv = self.dispatch_request() May 23 12:41:59.409006 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request May 23 12:41:59.409006 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) May 23 12:41:59.409006 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper May 23 12:41:59.409006 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) May 23 12:41:59.409006 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view May 23 12:41:59.409006 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) May 23 12:41:59.409006 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request May 23 12:41:59.409006 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) May 23 12:41:59.409006 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get May 23 12:41:59.409006 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self._get_domain(domain_id) May 23 12:41:59.409006 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain May 23 12:41:59.410132 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) May 23 12:41:59.410132 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped May 23 12:41:59.410132 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) May 23 12:41:59.410132 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun May 23 12:41:59.410132 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) May 23 12:41:59.410132 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func May 23 12:41:59.410132 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self.get_or_create( May 23 12:41:59.410132 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create May 23 12:41:59.410132 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application with Lock( May 23 12:41:59.410132 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ May 23 12:41:59.410132 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self._enter() May 23 12:41:59.410132 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter May 23 12:41:59.410132 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) May 23 12:41:59.410132 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create May 23 12:41:59.410132 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self.creator() May 23 12:41:59.410132 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value May 23 12:41:59.410132 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application created_value = creator( May 23 12:41:59.411857 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain May 23 12:41:59.411857 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) May 23 12:41:59.411857 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. May 23 12:41:59.411857 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application May 23 12:41:59.412088 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 429/860] 10.222.0.30 () {56 vars in 1108 bytes} [Mon May 23 12:41:59 2022] GET /identity/v3/domains/Default => generated 87 bytes in 213 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) May 23 12:41:59.417775 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f7fe292-eea1-43ba-849f-ab090d5c3071 None admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:41:59.421572 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-725cee32-63f7-4c88-a3e1-fab93309beca None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:41:59.422292 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-725cee32-63f7-4c88-a3e1-fab93309beca None admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:41:59.422579 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-725cee32-63f7-4c88-a3e1-fab93309beca None admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:41:59.422858 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-725cee32-63f7-4c88-a3e1-fab93309beca None admin] PATH_INFO: `/v3/services` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:41:59.423683 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-725cee32-63f7-4c88-a3e1-fab93309beca None admin] RBAC: Adding query filter params (type=compute_legacy) {{(pid=92056) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} May 23 12:41:59.423996 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-725cee32-63f7-4c88-a3e1-fab93309beca None admin] RBAC: Authorizing `identity:list_services()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:41:59.425380 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-725cee32-63f7-4c88-a3e1-fab93309beca None admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:41:59.431624 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 432/861] 10.222.0.30 () {56 vars in 1126 bytes} [Mon May 23 12:41:59 2022] GET /identity/v3/services?type=compute_legacy => generated 131 bytes in 169 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) May 23 12:41:59.541397 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=100 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:41:59.553183 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f7fe292-eea1-43ba-849f-ab090d5c3071 None admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:41:59.595882 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84ce937c-cef9-45e5-984b-f0f8fc132703 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:41:59.596486 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84ce937c-cef9-45e5-984b-f0f8fc132703 None admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:41:59.596771 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84ce937c-cef9-45e5-984b-f0f8fc132703 None admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:41:59.597050 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84ce937c-cef9-45e5-984b-f0f8fc132703 None admin] PATH_INFO: `/v3/domains` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:41:59.597790 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-84ce937c-cef9-45e5-984b-f0f8fc132703 None admin] RBAC: Adding query filter params (name=Default) {{(pid=92057) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} May 23 12:41:59.598206 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-84ce937c-cef9-45e5-984b-f0f8fc132703 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:41:59.600157 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-84ce937c-cef9-45e5-984b-f0f8fc132703 None admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:41:59.614362 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 430/862] 10.222.0.30 () {56 vars in 1109 bytes} [Mon May 23 12:41:59 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 196 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) May 23 12:41:59.619937 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-725cee32-63f7-4c88-a3e1-fab93309beca None admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:41:59.734988 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-725cee32-63f7-4c88-a3e1-fab93309beca None admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:41:59.774955 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c7fcdfd-6c50-485f-8b91-9f30e0c207b5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:41:59.777058 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c7fcdfd-6c50-485f-8b91-9f30e0c207b5 None admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:41:59.777620 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c7fcdfd-6c50-485f-8b91-9f30e0c207b5 None admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:41:59.778125 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c7fcdfd-6c50-485f-8b91-9f30e0c207b5 None admin] PATH_INFO: `/v3/users/glance-swift` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:41:59.780669 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=100 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:41:59.787285 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c7fcdfd-6c50-485f-8b91-9f30e0c207b5 None admin] RBAC: Authorizing `identity:get_user(user_id=glance-swift)` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:41:59.788950 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c7fcdfd-6c50-485f-8b91-9f30e0c207b5 None admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:41:59.805417 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application [None req-0c7fcdfd-6c50-485f-8b91-9f30e0c207b5 None admin] Could not find user: glance-swift.: keystone.exception.UserNotFound: Could not find user: glance-swift. May 23 12:41:59.805417 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application Traceback (most recent call last): May 23 12:41:59.805417 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request May 23 12:41:59.805417 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application rv = self.dispatch_request() May 23 12:41:59.805417 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request May 23 12:41:59.805417 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) May 23 12:41:59.805417 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper May 23 12:41:59.805417 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) May 23 12:41:59.805417 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view May 23 12:41:59.805417 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) May 23 12:41:59.805417 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request May 23 12:41:59.805417 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) May 23 12:41:59.805417 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get May 23 12:41:59.805417 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self._get_user(user_id) May 23 12:41:59.805417 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user May 23 12:41:59.805417 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) May 23 12:41:59.806434 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped May 23 12:41:59.806434 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) May 23 12:41:59.806434 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper May 23 12:41:59.806434 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) May 23 12:41:59.806434 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper May 23 12:41:59.806434 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) May 23 12:41:59.806434 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun May 23 12:41:59.806434 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) May 23 12:41:59.806434 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func May 23 12:41:59.806434 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self.get_or_create( May 23 12:41:59.806434 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create May 23 12:41:59.806434 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application with Lock( May 23 12:41:59.806434 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ May 23 12:41:59.806434 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self._enter() May 23 12:41:59.806434 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter May 23 12:41:59.806434 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) May 23 12:41:59.806434 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create May 23 12:41:59.806434 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self.creator() May 23 12:41:59.807701 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value May 23 12:41:59.807701 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application created_value = creator( May 23 12:41:59.807701 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user May 23 12:41:59.807701 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) May 23 12:41:59.807701 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user May 23 12:41:59.807701 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) May 23 12:41:59.807701 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user May 23 12:41:59.807701 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) May 23 12:41:59.807701 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: glance-swift. May 23 12:41:59.807701 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application May 23 12:41:59.807701 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 433/863] 10.222.0.30 () {56 vars in 1117 bytes} [Mon May 23 12:41:59 2022] GET /identity/v3/users/glance-swift => generated 90 bytes in 187 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) May 23 12:41:59.814189 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84ce937c-cef9-45e5-984b-f0f8fc132703 None admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:41:59.919768 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84ce937c-cef9-45e5-984b-f0f8fc132703 None admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:41:59.960617 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-749ba72d-55ea-4703-8c95-4dc8e7249e1d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:41:59.961364 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-749ba72d-55ea-4703-8c95-4dc8e7249e1d None admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:41:59.961563 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-749ba72d-55ea-4703-8c95-4dc8e7249e1d None admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:41:59.961840 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-749ba72d-55ea-4703-8c95-4dc8e7249e1d None admin] PATH_INFO: `/v3/users` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:41:59.962694 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-749ba72d-55ea-4703-8c95-4dc8e7249e1d None admin] RBAC: Adding query filter params (domain_id=default, name=glance-swift) {{(pid=92057) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} May 23 12:41:59.963003 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-749ba72d-55ea-4703-8c95-4dc8e7249e1d None admin] RBAC: Authorizing `identity:list_users()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:41:59.964433 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-749ba72d-55ea-4703-8c95-4dc8e7249e1d None admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:41:59.983159 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 431/864] 10.222.0.30 () {56 vars in 1149 bytes} [Mon May 23 12:41:59 2022] GET /identity/v3/users?domain_id=default&name=glance-swift => generated 384 bytes in 169 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) May 23 12:41:59.990438 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c7fcdfd-6c50-485f-8b91-9f30e0c207b5 None admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:42:00.120654 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c7fcdfd-6c50-485f-8b91-9f30e0c207b5 None admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:42:00.157165 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e625db39-fb3f-48a4-ba32-bae8a24ac77b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:42:00.157815 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e625db39-fb3f-48a4-ba32-bae8a24ac77b None admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:42:00.158123 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e625db39-fb3f-48a4-ba32-bae8a24ac77b None admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:42:00.158394 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e625db39-fb3f-48a4-ba32-bae8a24ac77b None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:42:00.226347 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=100 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:42:00.228592 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e625db39-fb3f-48a4-ba32-bae8a24ac77b None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:42:00.230068 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e625db39-fb3f-48a4-ba32-bae8a24ac77b None admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:42:00.332176 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 434/865] 10.222.0.30 () {58 vars in 1282 bytes} [Mon May 23 12:41:59 2022] GET /identity/v3/auth/tokens => generated 1151 bytes in 343 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) May 23 12:42:00.339327 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-749ba72d-55ea-4703-8c95-4dc8e7249e1d None admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:42:00.353343 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=100 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:42:00.458406 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-749ba72d-55ea-4703-8c95-4dc8e7249e1d None admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:42:00.493546 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-704242c2-83a9-4236-8ac2-55dba1535388 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:42:00.494287 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-704242c2-83a9-4236-8ac2-55dba1535388 None admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:42:00.494569 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-704242c2-83a9-4236-8ac2-55dba1535388 None admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:42:00.494844 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-704242c2-83a9-4236-8ac2-55dba1535388 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:42:00.505907 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-704242c2-83a9-4236-8ac2-55dba1535388 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:42:00.507364 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-704242c2-83a9-4236-8ac2-55dba1535388 None admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:42:00.524884 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application [None req-704242c2-83a9-4236-8ac2-55dba1535388 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. May 23 12:42:00.524884 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application Traceback (most recent call last): May 23 12:42:00.524884 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request May 23 12:42:00.524884 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application rv = self.dispatch_request() May 23 12:42:00.524884 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request May 23 12:42:00.524884 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) May 23 12:42:00.524884 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper May 23 12:42:00.524884 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) May 23 12:42:00.524884 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view May 23 12:42:00.524884 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) May 23 12:42:00.524884 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request May 23 12:42:00.524884 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) May 23 12:42:00.524884 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get May 23 12:42:00.524884 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self._get_domain(domain_id) May 23 12:42:00.524884 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain May 23 12:42:00.525838 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) May 23 12:42:00.525838 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped May 23 12:42:00.525838 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) May 23 12:42:00.525838 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun May 23 12:42:00.525838 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) May 23 12:42:00.525838 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func May 23 12:42:00.525838 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self.get_or_create( May 23 12:42:00.525838 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create May 23 12:42:00.525838 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application with Lock( May 23 12:42:00.525838 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ May 23 12:42:00.525838 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self._enter() May 23 12:42:00.525838 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter May 23 12:42:00.525838 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) May 23 12:42:00.525838 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create May 23 12:42:00.525838 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self.creator() May 23 12:42:00.525838 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value May 23 12:42:00.525838 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application created_value = creator( May 23 12:42:00.527081 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain May 23 12:42:00.527081 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) May 23 12:42:00.527081 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. May 23 12:42:00.527081 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application May 23 12:42:00.527081 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 432/866] 10.222.0.30 () {56 vars in 1108 bytes} [Mon May 23 12:42:00 2022] GET /identity/v3/domains/Default => generated 87 bytes in 188 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) May 23 12:42:00.532853 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e625db39-fb3f-48a4-ba32-bae8a24ac77b None admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:42:00.643511 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=100 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:42:00.651086 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e625db39-fb3f-48a4-ba32-bae8a24ac77b None admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:42:00.688224 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-699e1e4f-1950-4be5-88ef-906e1547575d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:42:00.688893 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-699e1e4f-1950-4be5-88ef-906e1547575d None admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:42:00.689177 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-699e1e4f-1950-4be5-88ef-906e1547575d None admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:42:00.689447 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-699e1e4f-1950-4be5-88ef-906e1547575d None admin] PATH_INFO: `/v3/domains` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:42:00.690231 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-699e1e4f-1950-4be5-88ef-906e1547575d None admin] RBAC: Adding query filter params (name=Default) {{(pid=92056) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} May 23 12:42:00.690541 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-699e1e4f-1950-4be5-88ef-906e1547575d None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:42:00.691974 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-699e1e4f-1950-4be5-88ef-906e1547575d None admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:42:00.704906 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 435/867] 10.222.0.30 () {56 vars in 1109 bytes} [Mon May 23 12:42:00 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 173 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) May 23 12:42:00.711001 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-704242c2-83a9-4236-8ac2-55dba1535388 None admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:42:00.777835 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=100 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:42:00.822250 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-704242c2-83a9-4236-8ac2-55dba1535388 None admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:42:00.859383 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2926e91c-b943-4f05-957a-a5778e1270a1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:42:00.860188 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2926e91c-b943-4f05-957a-a5778e1270a1 None admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:42:00.860474 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2926e91c-b943-4f05-957a-a5778e1270a1 None admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:42:00.860750 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2926e91c-b943-4f05-957a-a5778e1270a1 None admin] PATH_INFO: `/v3/projects/service` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:42:00.868890 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2926e91c-b943-4f05-957a-a5778e1270a1 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:42:00.870421 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2926e91c-b943-4f05-957a-a5778e1270a1 None admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:42:00.884889 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application [None req-2926e91c-b943-4f05-957a-a5778e1270a1 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. May 23 12:42:00.884889 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application Traceback (most recent call last): May 23 12:42:00.884889 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request May 23 12:42:00.884889 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application rv = self.dispatch_request() May 23 12:42:00.884889 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request May 23 12:42:00.884889 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) May 23 12:42:00.884889 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper May 23 12:42:00.884889 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) May 23 12:42:00.884889 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view May 23 12:42:00.884889 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) May 23 12:42:00.884889 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request May 23 12:42:00.884889 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) May 23 12:42:00.884889 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get May 23 12:42:00.884889 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self._get_project(project_id) May 23 12:42:00.884889 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project May 23 12:42:00.885824 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) May 23 12:42:00.885824 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped May 23 12:42:00.885824 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) May 23 12:42:00.885824 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun May 23 12:42:00.885824 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) May 23 12:42:00.885824 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func May 23 12:42:00.885824 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self.get_or_create( May 23 12:42:00.885824 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create May 23 12:42:00.885824 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application with Lock( May 23 12:42:00.885824 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ May 23 12:42:00.885824 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self._enter() May 23 12:42:00.885824 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter May 23 12:42:00.885824 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) May 23 12:42:00.885824 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create May 23 12:42:00.885824 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self.creator() May 23 12:42:00.885824 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value May 23 12:42:00.885824 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application created_value = creator( May 23 12:42:00.886773 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project May 23 12:42:00.886773 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) May 23 12:42:00.886773 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project May 23 12:42:00.886773 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() May 23 12:42:00.886773 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project May 23 12:42:00.886773 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) May 23 12:42:00.886773 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. May 23 12:42:00.886773 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application May 23 12:42:00.886773 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 433/868] 10.222.0.30 () {56 vars in 1111 bytes} [Mon May 23 12:42:00 2022] GET /identity/v3/projects/service => generated 88 bytes in 176 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) May 23 12:42:00.892138 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-699e1e4f-1950-4be5-88ef-906e1547575d None admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:42:01.007501 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-699e1e4f-1950-4be5-88ef-906e1547575d None admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:42:01.046309 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3d5a63b-af5e-4afc-b42f-fd1f69b3fc2b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:42:01.047013 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3d5a63b-af5e-4afc-b42f-fd1f69b3fc2b None admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:42:01.047297 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3d5a63b-af5e-4afc-b42f-fd1f69b3fc2b None admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:42:01.047576 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3d5a63b-af5e-4afc-b42f-fd1f69b3fc2b None admin] PATH_INFO: `/v3/projects` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:42:01.048365 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c3d5a63b-af5e-4afc-b42f-fd1f69b3fc2b None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=92056) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} May 23 12:42:01.048677 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c3d5a63b-af5e-4afc-b42f-fd1f69b3fc2b None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:42:01.050148 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c3d5a63b-af5e-4afc-b42f-fd1f69b3fc2b None admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:42:01.063943 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 436/869] 10.222.0.30 () {56 vars in 1148 bytes} [Mon May 23 12:42:00 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 172 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) May 23 12:42:01.070979 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2926e91c-b943-4f05-957a-a5778e1270a1 None admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:42:01.159663 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-606be7d5-380d-4c2f-96d6-a73d505abddd None None] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:42:01.160021 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-606be7d5-380d-4c2f-96d6-a73d505abddd None None] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:42:01.160227 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-606be7d5-380d-4c2f-96d6-a73d505abddd None None] PATH_INFO: `/` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:42:01.162905 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 437/870] 10.222.0.30 () {54 vars in 926 bytes} [Mon May 23 12:42:01 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) May 23 12:42:01.170671 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7e6e222-d0ad-4568-b990-e48d667bbcf5 None None] REQUEST_METHOD: `POST` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:42:01.171130 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7e6e222-d0ad-4568-b990-e48d667bbcf5 None None] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:42:01.171518 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7e6e222-d0ad-4568-b990-e48d667bbcf5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:42:01.187611 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2926e91c-b943-4f05-957a-a5778e1270a1 None admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:42:01.194349 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.auth.core [None req-a7e6e222-d0ad-4568-b990-e48d667bbcf5 None None] MFA Rules not processed for user `0f7119f8fa5c4445977671140ded91a4`. Rule list: `[]` (Enabled: `True`). {{(pid=92056) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 12:42:01.195619 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.fernet_utils [None req-a7e6e222-d0ad-4568-b990-e48d667bbcf5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92056) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 12:42:01.206405 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=100 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:42:01.211723 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=100 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:42:01.227732 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26f5bf83-5d66-451d-a206-77246c4a69ab None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:42:01.228395 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26f5bf83-5d66-451d-a206-77246c4a69ab None admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:42:01.228680 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26f5bf83-5d66-451d-a206-77246c4a69ab None admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:42:01.228962 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26f5bf83-5d66-451d-a206-77246c4a69ab None admin] PATH_INFO: `/v3/role_assignments` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:42:01.229786 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26f5bf83-5d66-451d-a206-77246c4a69ab None admin] RBAC: Adding query filter params (role.id=41ab28c1ae3d4941bd09fb5d35f1dc54, scope.project.id=9c389ae65aaa4542bb5a5c4fd0bacf5b, user.id=6d21a56126fd4ccd837c33ccc08ab609) {{(pid=92057) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} May 23 12:42:01.230146 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26f5bf83-5d66-451d-a206-77246c4a69ab None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:42:01.231696 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26f5bf83-5d66-451d-a206-77246c4a69ab None admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:42:01.242246 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 434/871] 10.222.0.30 () {56 vars in 1374 bytes} [Mon May 23 12:42:01 2022] GET /identity/v3/role_assignments?user.id=6d21a56126fd4ccd837c33ccc08ab609&scope.project.id=9c389ae65aaa4542bb5a5c4fd0bacf5b&role.id=41ab28c1ae3d4941bd09fb5d35f1dc54 => generated 608 bytes in 172 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) May 23 12:42:01.244101 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ac0f8ae-8677-4115-a48c-94f393cb746c None None] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:42:01.246454 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ac0f8ae-8677-4115-a48c-94f393cb746c None None] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:42:01.247001 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ac0f8ae-8677-4115-a48c-94f393cb746c None None] PATH_INFO: `/` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:42:01.254265 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 435/872] 10.222.0.30 () {54 vars in 926 bytes} [Mon May 23 12:42:01 2022] GET /identity => generated 270 bytes in 11 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) May 23 12:42:01.256626 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96fe4b67-d06a-4946-959a-f1862f7772bc None None] REQUEST_METHOD: `POST` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:42:01.256932 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96fe4b67-d06a-4946-959a-f1862f7772bc None None] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:42:01.257202 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96fe4b67-d06a-4946-959a-f1862f7772bc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:42:01.279611 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.auth.core [None req-96fe4b67-d06a-4946-959a-f1862f7772bc None None] MFA Rules not processed for user `0f7119f8fa5c4445977671140ded91a4`. Rule list: `[]` (Enabled: `True`). {{(pid=92057) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 12:42:01.280887 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.fernet_utils [None req-96fe4b67-d06a-4946-959a-f1862f7772bc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92057) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 12:42:01.295115 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 438/873] 10.222.0.30 () {58 vars in 1023 bytes} [Mon May 23 12:42:01 2022] POST /identity/v3/auth/tokens => generated 1151 bytes in 125 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) May 23 12:42:01.318598 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7e6e222-d0ad-4568-b990-e48d667bbcf5 None None] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:42:01.386750 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 436/874] 10.222.0.30 () {58 vars in 1023 bytes} [Mon May 23 12:42:01 2022] POST /identity/v3/auth/tokens => generated 1151 bytes in 125 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) May 23 12:42:01.411392 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96fe4b67-d06a-4946-959a-f1862f7772bc None None] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:42:01.443840 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7e6e222-d0ad-4568-b990-e48d667bbcf5 None None] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:42:01.492551 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3488c4b4-bf3d-4fbc-b9a8-600b9c39dff1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:42:01.493414 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3488c4b4-bf3d-4fbc-b9a8-600b9c39dff1 None admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:42:01.493699 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3488c4b4-bf3d-4fbc-b9a8-600b9c39dff1 None admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:42:01.494204 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3488c4b4-bf3d-4fbc-b9a8-600b9c39dff1 None admin] PATH_INFO: `/v3/services/object-store` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:42:01.495070 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3488c4b4-bf3d-4fbc-b9a8-600b9c39dff1 None admin] RBAC: Authorizing `identity:get_service(service_id=object-store)` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:42:01.496493 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3488c4b4-bf3d-4fbc-b9a8-600b9c39dff1 None admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:42:01.512097 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application [None req-3488c4b4-bf3d-4fbc-b9a8-600b9c39dff1 None admin] Could not find service: object-store.: keystone.exception.ServiceNotFound: Could not find service: object-store. May 23 12:42:01.512097 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application Traceback (most recent call last): May 23 12:42:01.512097 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service May 23 12:42:01.512097 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) May 23 12:42:01.512097 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service May 23 12:42:01.512097 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() May 23 12:42:01.512097 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service May 23 12:42:01.512097 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) May 23 12:42:01.512097 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. May 23 12:42:01.512097 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application May 23 12:42:01.512097 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: May 23 12:42:01.512097 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application May 23 12:42:01.512097 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application Traceback (most recent call last): May 23 12:42:01.512097 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request May 23 12:42:01.512097 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application rv = self.dispatch_request() May 23 12:42:01.512097 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request May 23 12:42:01.512097 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) May 23 12:42:01.513223 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper May 23 12:42:01.513223 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) May 23 12:42:01.513223 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view May 23 12:42:01.513223 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) May 23 12:42:01.513223 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request May 23 12:42:01.513223 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) May 23 12:42:01.513223 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get May 23 12:42:01.513223 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self._get_service(service_id) May 23 12:42:01.513223 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service May 23 12:42:01.513223 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) May 23 12:42:01.513223 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped May 23 12:42:01.513223 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) May 23 12:42:01.513223 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun May 23 12:42:01.513223 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) May 23 12:42:01.513223 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func May 23 12:42:01.513223 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self.get_or_create( May 23 12:42:01.514176 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create May 23 12:42:01.514176 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application with Lock( May 23 12:42:01.514176 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ May 23 12:42:01.514176 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self._enter() May 23 12:42:01.514176 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter May 23 12:42:01.514176 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) May 23 12:42:01.514176 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create May 23 12:42:01.514176 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self.creator() May 23 12:42:01.514176 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value May 23 12:42:01.514176 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application created_value = creator( May 23 12:42:01.514176 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service May 23 12:42:01.514176 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) May 23 12:42:01.514176 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. May 23 12:42:01.514176 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application May 23 12:42:01.516567 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 439/875] 10.222.0.30 () {56 vars in 1126 bytes} [Mon May 23 12:42:01 2022] GET /identity/v3/services/object-store => generated 93 bytes in 199 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) May 23 12:42:01.523973 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3488c4b4-bf3d-4fbc-b9a8-600b9c39dff1 None admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:42:01.538075 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=100 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:42:01.552792 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96fe4b67-d06a-4946-959a-f1862f7772bc None None] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:42:01.571349 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=100 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:42:01.597617 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ccea3d68-6c99-4a63-a905-7c6042d5b2a9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:42:01.599209 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccea3d68-6c99-4a63-a905-7c6042d5b2a9 None admin] REQUEST_METHOD: `POST` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:42:01.599583 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccea3d68-6c99-4a63-a905-7c6042d5b2a9 None admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:42:01.599970 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccea3d68-6c99-4a63-a905-7c6042d5b2a9 None admin] PATH_INFO: `/v3/services` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:42:01.600773 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ccea3d68-6c99-4a63-a905-7c6042d5b2a9 None admin] RBAC: Authorizing `identity:create_service()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:42:01.602185 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ccea3d68-6c99-4a63-a905-7c6042d5b2a9 None admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:42:01.652009 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3488c4b4-bf3d-4fbc-b9a8-600b9c39dff1 None admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:42:01.674109 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 437/876] 10.222.0.30 () {60 vars in 1141 bytes} [Mon May 23 12:42:01 2022] POST /identity/v3/services => generated 251 bytes in 263 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) May 23 12:42:01.704140 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84eaee5a-77bd-454a-b828-84f046435fd2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:42:01.704820 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84eaee5a-77bd-454a-b828-84f046435fd2 None admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:42:01.705103 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84eaee5a-77bd-454a-b828-84f046435fd2 None admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:42:01.705438 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84eaee5a-77bd-454a-b828-84f046435fd2 None admin] PATH_INFO: `/v3/services` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:42:01.706250 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-84eaee5a-77bd-454a-b828-84f046435fd2 None admin] RBAC: Adding query filter params (name=object-store) {{(pid=92056) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} May 23 12:42:01.706558 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-84eaee5a-77bd-454a-b828-84f046435fd2 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:42:01.708135 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-84eaee5a-77bd-454a-b828-84f046435fd2 None admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:42:01.714701 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 440/877] 10.222.0.30 () {56 vars in 1122 bytes} [Mon May 23 12:42:01 2022] GET /identity/v3/services?name=object-store => generated 129 bytes in 191 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) May 23 12:42:01.720243 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ccea3d68-6c99-4a63-a905-7c6042d5b2a9 None admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:42:01.853773 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ccea3d68-6c99-4a63-a905-7c6042d5b2a9 None admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:42:01.891573 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2dfd9e3e-fd18-4986-9142-3c65d5d9420b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:42:01.892611 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2dfd9e3e-fd18-4986-9142-3c65d5d9420b None admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:42:01.893085 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2dfd9e3e-fd18-4986-9142-3c65d5d9420b None admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:42:01.893557 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2dfd9e3e-fd18-4986-9142-3c65d5d9420b None admin] PATH_INFO: `/v3/services` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:42:01.894500 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2dfd9e3e-fd18-4986-9142-3c65d5d9420b None admin] RBAC: Adding query filter params (type=object-store) {{(pid=92057) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} May 23 12:42:01.895535 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2dfd9e3e-fd18-4986-9142-3c65d5d9420b None admin] RBAC: Authorizing `identity:list_services()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:42:01.897157 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2dfd9e3e-fd18-4986-9142-3c65d5d9420b None admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:42:01.903059 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 438/878] 10.222.0.30 () {56 vars in 1122 bytes} [Mon May 23 12:42:01 2022] GET /identity/v3/services?type=object-store => generated 129 bytes in 183 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) May 23 12:42:01.975315 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02e3de7e-e2d8-4409-a592-62bad5076e46 None None] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:42:01.975833 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02e3de7e-e2d8-4409-a592-62bad5076e46 None None] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:42:01.975833 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02e3de7e-e2d8-4409-a592-62bad5076e46 None None] PATH_INFO: `/` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:42:01.977498 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 441/879] 10.222.0.30 () {54 vars in 926 bytes} [Mon May 23 12:42:01 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) May 23 12:42:01.983465 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5db17cf0-0c7e-4dd1-8bb5-735af7d21ca6 None None] REQUEST_METHOD: `POST` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:42:01.983751 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5db17cf0-0c7e-4dd1-8bb5-735af7d21ca6 None None] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:42:01.984037 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5db17cf0-0c7e-4dd1-8bb5-735af7d21ca6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:42:02.002756 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.auth.core [None req-5db17cf0-0c7e-4dd1-8bb5-735af7d21ca6 None None] MFA Rules not processed for user `0f7119f8fa5c4445977671140ded91a4`. Rule list: `[]` (Enabled: `True`). {{(pid=92057) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 12:42:02.003970 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.fernet_utils [None req-5db17cf0-0c7e-4dd1-8bb5-735af7d21ca6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92057) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 12:42:02.038838 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:42:02.082307 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 439/880] 10.222.0.30 () {58 vars in 1023 bytes} [Mon May 23 12:42:01 2022] POST /identity/v3/auth/tokens => generated 1252 bytes in 100 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) May 23 12:42:02.106750 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02e3de7e-e2d8-4409-a592-62bad5076e46 None None] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:42:02.183251 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=100 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:42:02.218773 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02e3de7e-e2d8-4409-a592-62bad5076e46 None None] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:42:02.259358 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c7a4942-08c7-4985-8cc2-b8874ea56b17 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:42:02.260031 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c7a4942-08c7-4985-8cc2-b8874ea56b17 None admin] REQUEST_METHOD: `POST` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:42:02.260317 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c7a4942-08c7-4985-8cc2-b8874ea56b17 None admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:42:02.260596 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c7a4942-08c7-4985-8cc2-b8874ea56b17 None admin] PATH_INFO: `/v3/services` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:42:02.261328 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c7a4942-08c7-4985-8cc2-b8874ea56b17 None admin] RBAC: Authorizing `identity:create_service()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:42:02.262732 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c7a4942-08c7-4985-8cc2-b8874ea56b17 None admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:42:02.382143 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 442/881] 10.222.0.30 () {60 vars in 1141 bytes} [Mon May 23 12:42:02 2022] POST /identity/v3/services => generated 269 bytes in 276 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) May 23 12:42:03.686641 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-089d10b8-edeb-437f-8afa-ea18418b298e None None] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:42:03.687160 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-089d10b8-edeb-437f-8afa-ea18418b298e None None] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:42:03.687241 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-089d10b8-edeb-437f-8afa-ea18418b298e None None] PATH_INFO: `/` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:42:03.688826 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 440/882] 10.222.0.30 () {54 vars in 926 bytes} [Mon May 23 12:42:03 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) May 23 12:42:03.699945 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c3e4b3e-3f02-43e7-9657-4744487652be None None] REQUEST_METHOD: `POST` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:42:03.700721 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c3e4b3e-3f02-43e7-9657-4744487652be None None] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:42:03.701170 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c3e4b3e-3f02-43e7-9657-4744487652be None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:42:03.719266 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.auth.core [None req-9c3e4b3e-3f02-43e7-9657-4744487652be None None] MFA Rules not processed for user `0f7119f8fa5c4445977671140ded91a4`. Rule list: `[]` (Enabled: `True`). {{(pid=92056) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 12:42:03.720714 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.fernet_utils [None req-9c3e4b3e-3f02-43e7-9657-4744487652be None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92056) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 12:42:03.795776 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 443/883] 10.222.0.30 () {58 vars in 1023 bytes} [Mon May 23 12:42:03 2022] POST /identity/v3/auth/tokens => generated 1362 bytes in 97 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) May 23 12:42:03.819918 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-089d10b8-edeb-437f-8afa-ea18418b298e None None] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:42:03.933779 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-089d10b8-edeb-437f-8afa-ea18418b298e None None] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:42:03.970065 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d5456bc-70a9-4d71-9c05-9d9a0c51acd4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:42:03.971222 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d5456bc-70a9-4d71-9c05-9d9a0c51acd4 None admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:42:03.971712 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d5456bc-70a9-4d71-9c05-9d9a0c51acd4 None admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:42:03.972187 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d5456bc-70a9-4d71-9c05-9d9a0c51acd4 None admin] PATH_INFO: `/v3/roles/ResellerAdmin` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:42:03.983627 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d5456bc-70a9-4d71-9c05-9d9a0c51acd4 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} May 23 12:42:03.992677 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application [None req-0d5456bc-70a9-4d71-9c05-9d9a0c51acd4 None admin] Could not find role: ResellerAdmin.: keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. May 23 12:42:03.992677 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application Traceback (most recent call last): May 23 12:42:03.992677 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request May 23 12:42:03.992677 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application rv = self.dispatch_request() May 23 12:42:03.992677 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request May 23 12:42:03.992677 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) May 23 12:42:03.992677 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper May 23 12:42:03.992677 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) May 23 12:42:03.992677 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view May 23 12:42:03.992677 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) May 23 12:42:03.992677 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request May 23 12:42:03.992677 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) May 23 12:42:03.992677 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get May 23 12:42:03.992677 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self._get_role(role_id) May 23 12:42:03.992677 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role May 23 12:42:03.992677 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') May 23 12:42:03.993647 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call May 23 12:42:03.993647 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( May 23 12:42:03.993647 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data May 23 12:42:03.993647 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) May 23 12:42:03.993647 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped May 23 12:42:03.993647 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) May 23 12:42:03.993647 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun May 23 12:42:03.993647 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) May 23 12:42:03.993647 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func May 23 12:42:03.993647 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self.get_or_create( May 23 12:42:03.993647 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create May 23 12:42:03.993647 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application with Lock( May 23 12:42:03.993647 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ May 23 12:42:03.993647 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self._enter() May 23 12:42:03.993647 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter May 23 12:42:03.993647 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) May 23 12:42:03.994570 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create May 23 12:42:03.994570 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self.creator() May 23 12:42:03.994570 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value May 23 12:42:03.994570 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application created_value = creator( May 23 12:42:03.994570 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role May 23 12:42:03.994570 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) May 23 12:42:03.994570 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role May 23 12:42:03.994570 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() May 23 12:42:03.994570 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role May 23 12:42:03.994570 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) May 23 12:42:03.994570 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. May 23 12:42:03.994570 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application May 23 12:42:03.997285 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 441/884] 10.222.0.30 () {56 vars in 1120 bytes} [Mon May 23 12:42:03 2022] GET /identity/v3/roles/ResellerAdmin => generated 91 bytes in 178 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) May 23 12:42:04.003371 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c3e4b3e-3f02-43e7-9657-4744487652be None None] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:42:04.062538 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:42:04.100768 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b84bf74-f1f8-4f59-9318-941260afac38 None None] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:42:04.101445 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b84bf74-f1f8-4f59-9318-941260afac38 None None] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:42:04.101978 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b84bf74-f1f8-4f59-9318-941260afac38 None None] PATH_INFO: `/` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:42:04.103920 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 442/885] 10.222.0.30 () {54 vars in 926 bytes} [Mon May 23 12:42:04 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) May 23 12:42:04.110609 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e8c5c19-8717-4004-a9f6-0308e9564d04 None None] REQUEST_METHOD: `POST` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:42:04.111224 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e8c5c19-8717-4004-a9f6-0308e9564d04 None None] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:42:04.111733 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e8c5c19-8717-4004-a9f6-0308e9564d04 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:42:04.113995 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c3e4b3e-3f02-43e7-9657-4744487652be None None] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:42:04.130761 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.auth.core [None req-9e8c5c19-8717-4004-a9f6-0308e9564d04 None None] MFA Rules not processed for user `0f7119f8fa5c4445977671140ded91a4`. Rule list: `[]` (Enabled: `True`). {{(pid=92057) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 12:42:04.132320 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.fernet_utils [None req-9e8c5c19-8717-4004-a9f6-0308e9564d04 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92057) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 12:42:04.155914 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a36fa09-9c90-43a4-a582-7c6ced4c8d74 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:42:04.156609 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a36fa09-9c90-43a4-a582-7c6ced4c8d74 None admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:42:04.156888 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a36fa09-9c90-43a4-a582-7c6ced4c8d74 None admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:42:04.157157 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a36fa09-9c90-43a4-a582-7c6ced4c8d74 None admin] PATH_INFO: `/v3/roles` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:42:04.158101 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a36fa09-9c90-43a4-a582-7c6ced4c8d74 None admin] RBAC: Adding query filter params (name=ResellerAdmin) {{(pid=92056) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} May 23 12:42:04.158453 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a36fa09-9c90-43a4-a582-7c6ced4c8d74 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:42:04.159974 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a36fa09-9c90-43a4-a582-7c6ced4c8d74 None admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:42:04.161668 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=100 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:42:04.175114 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 444/886] 10.222.0.30 () {56 vars in 1115 bytes} [Mon May 23 12:42:04 2022] GET /identity/v3/roles?name=ResellerAdmin => generated 338 bytes in 168 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) May 23 12:42:04.177417 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a36fa09-9c90-43a4-a582-7c6ced4c8d74 None admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:42:04.228843 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 443/887] 10.222.0.30 () {58 vars in 1023 bytes} [Mon May 23 12:42:04 2022] POST /identity/v3/auth/tokens => generated 1362 bytes in 112 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) May 23 12:42:04.246232 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e8c5c19-8717-4004-a9f6-0308e9564d04 None None] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:42:04.299511 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a36fa09-9c90-43a4-a582-7c6ced4c8d74 None admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:42:04.321554 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=100 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:42:04.341364 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd777405-d3ce-42b7-af9f-a31183659b24 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:42:04.342190 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd777405-d3ce-42b7-af9f-a31183659b24 None admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:42:04.342467 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd777405-d3ce-42b7-af9f-a31183659b24 None admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:42:04.342741 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd777405-d3ce-42b7-af9f-a31183659b24 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:42:04.354978 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd777405-d3ce-42b7-af9f-a31183659b24 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:42:04.355290 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e8c5c19-8717-4004-a9f6-0308e9564d04 None None] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:42:04.356592 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd777405-d3ce-42b7-af9f-a31183659b24 None admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:42:04.377428 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application [None req-bd777405-d3ce-42b7-af9f-a31183659b24 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. May 23 12:42:04.377428 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application Traceback (most recent call last): May 23 12:42:04.377428 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request May 23 12:42:04.377428 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application rv = self.dispatch_request() May 23 12:42:04.377428 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request May 23 12:42:04.377428 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) May 23 12:42:04.377428 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper May 23 12:42:04.377428 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) May 23 12:42:04.377428 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view May 23 12:42:04.377428 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) May 23 12:42:04.377428 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request May 23 12:42:04.377428 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) May 23 12:42:04.377428 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get May 23 12:42:04.377428 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self._get_domain(domain_id) May 23 12:42:04.377428 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain May 23 12:42:04.383607 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) May 23 12:42:04.383607 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped May 23 12:42:04.383607 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) May 23 12:42:04.383607 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun May 23 12:42:04.383607 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) May 23 12:42:04.383607 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func May 23 12:42:04.383607 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self.get_or_create( May 23 12:42:04.383607 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create May 23 12:42:04.383607 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application with Lock( May 23 12:42:04.383607 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ May 23 12:42:04.383607 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self._enter() May 23 12:42:04.383607 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter May 23 12:42:04.383607 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) May 23 12:42:04.383607 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create May 23 12:42:04.383607 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self.creator() May 23 12:42:04.383607 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value May 23 12:42:04.383607 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application created_value = creator( May 23 12:42:04.384606 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain May 23 12:42:04.384606 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) May 23 12:42:04.384606 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. May 23 12:42:04.384606 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application May 23 12:42:04.384606 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 445/888] 10.222.0.30 () {56 vars in 1108 bytes} [Mon May 23 12:42:04 2022] GET /identity/v3/domains/Default => generated 87 bytes in 203 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) May 23 12:42:04.384606 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09832fe1-407d-4340-b758-2074e0ee0131 None None] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:42:04.384606 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09832fe1-407d-4340-b758-2074e0ee0131 None None] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:42:04.384606 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09832fe1-407d-4340-b758-2074e0ee0131 None None] PATH_INFO: `/` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:42:04.384606 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 446/889] 10.222.0.30 () {54 vars in 926 bytes} [Mon May 23 12:42:04 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) May 23 12:42:04.387184 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09832fe1-407d-4340-b758-2074e0ee0131 None None] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:42:04.398019 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee876f8e-351e-492e-a4b6-de3cd59333b0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:42:04.398817 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee876f8e-351e-492e-a4b6-de3cd59333b0 None admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:42:04.399103 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee876f8e-351e-492e-a4b6-de3cd59333b0 None admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:42:04.399378 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee876f8e-351e-492e-a4b6-de3cd59333b0 None admin] PATH_INFO: `/v3/services/volumev3` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:42:04.400030 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee876f8e-351e-492e-a4b6-de3cd59333b0 None admin] RBAC: Authorizing `identity:get_service(service_id=volumev3)` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:42:04.401670 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee876f8e-351e-492e-a4b6-de3cd59333b0 None admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:42:04.414776 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application [None req-ee876f8e-351e-492e-a4b6-de3cd59333b0 None admin] Could not find service: volumev3.: keystone.exception.ServiceNotFound: Could not find service: volumev3. May 23 12:42:04.414776 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application Traceback (most recent call last): May 23 12:42:04.414776 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service May 23 12:42:04.414776 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) May 23 12:42:04.414776 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service May 23 12:42:04.414776 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() May 23 12:42:04.414776 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service May 23 12:42:04.414776 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) May 23 12:42:04.414776 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: volumev3. May 23 12:42:04.414776 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application May 23 12:42:04.414776 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: May 23 12:42:04.414776 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application May 23 12:42:04.414776 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application Traceback (most recent call last): May 23 12:42:04.414776 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request May 23 12:42:04.414776 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application rv = self.dispatch_request() May 23 12:42:04.414776 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request May 23 12:42:04.414776 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) May 23 12:42:04.415899 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper May 23 12:42:04.415899 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) May 23 12:42:04.415899 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view May 23 12:42:04.415899 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) May 23 12:42:04.415899 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request May 23 12:42:04.415899 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) May 23 12:42:04.415899 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get May 23 12:42:04.415899 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self._get_service(service_id) May 23 12:42:04.415899 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service May 23 12:42:04.415899 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) May 23 12:42:04.415899 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped May 23 12:42:04.415899 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) May 23 12:42:04.415899 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun May 23 12:42:04.415899 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) May 23 12:42:04.415899 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func May 23 12:42:04.415899 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self.get_or_create( May 23 12:42:04.416770 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create May 23 12:42:04.416770 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application with Lock( May 23 12:42:04.416770 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ May 23 12:42:04.416770 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self._enter() May 23 12:42:04.416770 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter May 23 12:42:04.416770 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) May 23 12:42:04.416770 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create May 23 12:42:04.416770 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self.creator() May 23 12:42:04.416770 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value May 23 12:42:04.416770 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application created_value = creator( May 23 12:42:04.416770 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service May 23 12:42:04.416770 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) May 23 12:42:04.416770 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: volumev3. May 23 12:42:04.416770 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application May 23 12:42:04.416770 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 444/890] 10.222.0.30 () {56 vars in 1114 bytes} [Mon May 23 12:42:04 2022] GET /identity/v3/services/volumev3 => generated 89 bytes in 170 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) May 23 12:42:04.417574 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c935371-02de-419b-806f-b200bae8188c None None] REQUEST_METHOD: `POST` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:42:04.417574 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c935371-02de-419b-806f-b200bae8188c None None] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:42:04.417709 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c935371-02de-419b-806f-b200bae8188c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:42:04.435338 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.auth.core [None req-1c935371-02de-419b-806f-b200bae8188c None None] MFA Rules not processed for user `0f7119f8fa5c4445977671140ded91a4`. Rule list: `[]` (Enabled: `True`). {{(pid=92057) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 12:42:04.436576 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.fernet_utils [None req-1c935371-02de-419b-806f-b200bae8188c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92057) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 12:42:04.457316 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=100 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:42:04.513942 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09832fe1-407d-4340-b758-2074e0ee0131 None None] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:42:04.518723 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 445/891] 10.222.0.30 () {58 vars in 1023 bytes} [Mon May 23 12:42:04 2022] POST /identity/v3/auth/tokens => generated 1362 bytes in 102 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) May 23 12:42:04.519831 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c935371-02de-419b-806f-b200bae8188c None None] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:42:04.563253 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6f00a10-9354-4e08-b880-d451a2158ff8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:42:04.564055 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6f00a10-9354-4e08-b880-d451a2158ff8 None admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:42:04.564485 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6f00a10-9354-4e08-b880-d451a2158ff8 None admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:42:04.564860 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6f00a10-9354-4e08-b880-d451a2158ff8 None admin] PATH_INFO: `/v3/domains` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:42:04.565621 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6f00a10-9354-4e08-b880-d451a2158ff8 None admin] RBAC: Adding query filter params (name=Default) {{(pid=92056) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} May 23 12:42:04.565956 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6f00a10-9354-4e08-b880-d451a2158ff8 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:42:04.567611 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6f00a10-9354-4e08-b880-d451a2158ff8 None admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:42:04.580240 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 447/892] 10.222.0.30 () {56 vars in 1109 bytes} [Mon May 23 12:42:04 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 194 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) May 23 12:42:04.581040 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6f00a10-9354-4e08-b880-d451a2158ff8 None admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:42:04.620945 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=100 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:42:04.647647 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c935371-02de-419b-806f-b200bae8188c None None] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:42:04.683222 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3500e7a9-823d-4e2c-ba18-6e03dddedc17 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:42:04.683990 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3500e7a9-823d-4e2c-ba18-6e03dddedc17 None admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:42:04.684386 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3500e7a9-823d-4e2c-ba18-6e03dddedc17 None admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:42:04.685144 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3500e7a9-823d-4e2c-ba18-6e03dddedc17 None admin] PATH_INFO: `/v3/services` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:42:04.685518 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3500e7a9-823d-4e2c-ba18-6e03dddedc17 None admin] RBAC: Adding query filter params (name=volumev3) {{(pid=92057) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} May 23 12:42:04.685834 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3500e7a9-823d-4e2c-ba18-6e03dddedc17 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:42:04.687253 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3500e7a9-823d-4e2c-ba18-6e03dddedc17 None admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:42:04.692775 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 446/893] 10.222.0.30 () {56 vars in 1114 bytes} [Mon May 23 12:42:04 2022] GET /identity/v3/services?name=volumev3 => generated 125 bytes in 173 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) May 23 12:42:04.693648 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3500e7a9-823d-4e2c-ba18-6e03dddedc17 None admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:42:04.699325 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6f00a10-9354-4e08-b880-d451a2158ff8 None admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:42:04.715491 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=100 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:42:04.733434 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ddb5e888-4183-47d2-8d45-88229937d358 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:42:04.734119 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddb5e888-4183-47d2-8d45-88229937d358 None admin] REQUEST_METHOD: `POST` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:42:04.734400 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddb5e888-4183-47d2-8d45-88229937d358 None admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:42:04.734674 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddb5e888-4183-47d2-8d45-88229937d358 None admin] PATH_INFO: `/v3/services` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:42:04.735389 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ddb5e888-4183-47d2-8d45-88229937d358 None admin] RBAC: Authorizing `identity:create_service()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:42:04.736782 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ddb5e888-4183-47d2-8d45-88229937d358 None admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:42:04.794864 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3500e7a9-823d-4e2c-ba18-6e03dddedc17 None admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:42:04.815035 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 448/894] 10.222.0.30 () {60 vars in 1141 bytes} [Mon May 23 12:42:04 2022] POST /identity/v3/services => generated 241 bytes in 234 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) May 23 12:42:04.816324 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f99cc647-ef97-480c-98d7-6a055d07caab None None] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:42:04.817282 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f99cc647-ef97-480c-98d7-6a055d07caab None None] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:42:04.817562 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f99cc647-ef97-480c-98d7-6a055d07caab None None] PATH_INFO: `/` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:42:04.818843 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 449/895] 10.222.0.30 () {54 vars in 926 bytes} [Mon May 23 12:42:04 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) May 23 12:42:04.819559 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f99cc647-ef97-480c-98d7-6a055d07caab None None] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:42:04.848862 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43cb39ae-afd2-41c1-9ac9-7815c73ff832 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:42:04.849705 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43cb39ae-afd2-41c1-9ac9-7815c73ff832 None admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:42:04.850044 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43cb39ae-afd2-41c1-9ac9-7815c73ff832 None admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:42:04.850323 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43cb39ae-afd2-41c1-9ac9-7815c73ff832 None admin] PATH_INFO: `/v3/users/glance-swift` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:42:04.858975 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43cb39ae-afd2-41c1-9ac9-7815c73ff832 None admin] RBAC: Authorizing `identity:get_user(user_id=glance-swift)` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:42:04.860400 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43cb39ae-afd2-41c1-9ac9-7815c73ff832 None admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:42:04.877683 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application [None req-43cb39ae-afd2-41c1-9ac9-7815c73ff832 None admin] Could not find user: glance-swift.: keystone.exception.UserNotFound: Could not find user: glance-swift. May 23 12:42:04.877683 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application Traceback (most recent call last): May 23 12:42:04.877683 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request May 23 12:42:04.877683 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application rv = self.dispatch_request() May 23 12:42:04.877683 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request May 23 12:42:04.877683 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) May 23 12:42:04.877683 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper May 23 12:42:04.877683 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) May 23 12:42:04.877683 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view May 23 12:42:04.877683 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) May 23 12:42:04.877683 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request May 23 12:42:04.877683 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) May 23 12:42:04.877683 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get May 23 12:42:04.877683 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self._get_user(user_id) May 23 12:42:04.877683 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user May 23 12:42:04.877683 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) May 23 12:42:04.878780 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped May 23 12:42:04.878780 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) May 23 12:42:04.878780 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper May 23 12:42:04.878780 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) May 23 12:42:04.878780 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper May 23 12:42:04.878780 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) May 23 12:42:04.878780 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun May 23 12:42:04.878780 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) May 23 12:42:04.878780 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func May 23 12:42:04.878780 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self.get_or_create( May 23 12:42:04.878780 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create May 23 12:42:04.878780 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application with Lock( May 23 12:42:04.878780 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ May 23 12:42:04.878780 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self._enter() May 23 12:42:04.878780 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter May 23 12:42:04.878780 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) May 23 12:42:04.878780 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create May 23 12:42:04.878780 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self.creator() May 23 12:42:04.880387 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value May 23 12:42:04.880387 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application created_value = creator( May 23 12:42:04.880387 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user May 23 12:42:04.880387 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) May 23 12:42:04.880387 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user May 23 12:42:04.880387 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) May 23 12:42:04.880387 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user May 23 12:42:04.880387 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) May 23 12:42:04.880387 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: glance-swift. May 23 12:42:04.880387 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application May 23 12:42:04.880387 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 447/896] 10.222.0.30 () {56 vars in 1117 bytes} [Mon May 23 12:42:04 2022] GET /identity/v3/users/glance-swift => generated 90 bytes in 186 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) May 23 12:42:04.881914 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e043e6e2-ca8b-4aef-98e7-77d4143ce7da None None] REQUEST_METHOD: `POST` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:42:04.881914 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e043e6e2-ca8b-4aef-98e7-77d4143ce7da None None] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:42:04.881914 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e043e6e2-ca8b-4aef-98e7-77d4143ce7da None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:42:04.899108 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.auth.core [None req-e043e6e2-ca8b-4aef-98e7-77d4143ce7da None None] MFA Rules not processed for user `0f7119f8fa5c4445977671140ded91a4`. Rule list: `[]` (Enabled: `True`). {{(pid=92057) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 12:42:04.900323 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.fernet_utils [None req-e043e6e2-ca8b-4aef-98e7-77d4143ce7da None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92057) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 12:42:04.932371 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:42:04.947900 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f99cc647-ef97-480c-98d7-6a055d07caab None None] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:42:04.974159 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 448/897] 10.222.0.30 () {58 vars in 1023 bytes} [Mon May 23 12:42:04 2022] POST /identity/v3/auth/tokens => generated 1464 bytes in 94 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) May 23 12:42:04.975704 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e043e6e2-ca8b-4aef-98e7-77d4143ce7da None None] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:42:04.978458 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=100 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:42:05.000219 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-996c49b2-adb2-424b-91f4-23cee5b72ca5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:42:05.001177 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-996c49b2-adb2-424b-91f4-23cee5b72ca5 None admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:42:05.001465 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-996c49b2-adb2-424b-91f4-23cee5b72ca5 None admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:42:05.001740 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-996c49b2-adb2-424b-91f4-23cee5b72ca5 None admin] PATH_INFO: `/v3/services` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:42:05.002486 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-996c49b2-adb2-424b-91f4-23cee5b72ca5 None admin] RBAC: Adding query filter params (type=volumev3) {{(pid=92056) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} May 23 12:42:05.002793 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-996c49b2-adb2-424b-91f4-23cee5b72ca5 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:42:05.004835 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-996c49b2-adb2-424b-91f4-23cee5b72ca5 None admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:42:05.010307 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 450/898] 10.222.0.30 () {56 vars in 1114 bytes} [Mon May 23 12:42:04 2022] GET /identity/v3/services?type=volumev3 => generated 362 bytes in 191 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) May 23 12:42:05.011115 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-996c49b2-adb2-424b-91f4-23cee5b72ca5 None admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:42:05.102178 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e043e6e2-ca8b-4aef-98e7-77d4143ce7da None None] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:42:05.118363 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-996c49b2-adb2-424b-91f4-23cee5b72ca5 None admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:42:05.140506 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b6e5cb4-e7e5-409f-9be1-6209521d6845 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:42:05.141614 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b6e5cb4-e7e5-409f-9be1-6209521d6845 None admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:42:05.142217 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b6e5cb4-e7e5-409f-9be1-6209521d6845 None admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:42:05.142707 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b6e5cb4-e7e5-409f-9be1-6209521d6845 None admin] PATH_INFO: `/v3/users` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:42:05.143744 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b6e5cb4-e7e5-409f-9be1-6209521d6845 None admin] RBAC: Adding query filter params (domain_id=default, name=glance-swift) {{(pid=92057) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} May 23 12:42:05.144310 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b6e5cb4-e7e5-409f-9be1-6209521d6845 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:42:05.146792 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b6e5cb4-e7e5-409f-9be1-6209521d6845 None admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:42:05.153113 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60286a5f-8e8f-48c9-b6f6-1f2ebf977ef8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:42:05.153953 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60286a5f-8e8f-48c9-b6f6-1f2ebf977ef8 None admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:42:05.154243 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60286a5f-8e8f-48c9-b6f6-1f2ebf977ef8 None admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:42:05.154520 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60286a5f-8e8f-48c9-b6f6-1f2ebf977ef8 None admin] PATH_INFO: `/v3/services/compute_legacy` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:42:05.155178 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60286a5f-8e8f-48c9-b6f6-1f2ebf977ef8 None admin] RBAC: Authorizing `identity:get_service(service_id=compute_legacy)` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:42:05.156548 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60286a5f-8e8f-48c9-b6f6-1f2ebf977ef8 None admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:42:05.166667 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 449/899] 10.222.0.30 () {56 vars in 1149 bytes} [Mon May 23 12:42:04 2022] GET /identity/v3/users?domain_id=default&name=glance-swift => generated 384 bytes in 191 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) May 23 12:42:05.167739 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b6e5cb4-e7e5-409f-9be1-6209521d6845 None admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:42:05.169071 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application [None req-60286a5f-8e8f-48c9-b6f6-1f2ebf977ef8 None admin] Could not find service: compute_legacy.: keystone.exception.ServiceNotFound: Could not find service: compute_legacy. May 23 12:42:05.169071 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application Traceback (most recent call last): May 23 12:42:05.169071 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service May 23 12:42:05.169071 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) May 23 12:42:05.169071 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service May 23 12:42:05.169071 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() May 23 12:42:05.169071 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service May 23 12:42:05.169071 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) May 23 12:42:05.169071 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute_legacy. May 23 12:42:05.169071 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application May 23 12:42:05.169071 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: May 23 12:42:05.169071 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application May 23 12:42:05.169071 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application Traceback (most recent call last): May 23 12:42:05.169071 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request May 23 12:42:05.169071 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application rv = self.dispatch_request() May 23 12:42:05.169071 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request May 23 12:42:05.170104 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) May 23 12:42:05.170104 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper May 23 12:42:05.170104 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) May 23 12:42:05.170104 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view May 23 12:42:05.170104 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) May 23 12:42:05.170104 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request May 23 12:42:05.170104 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) May 23 12:42:05.170104 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get May 23 12:42:05.170104 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self._get_service(service_id) May 23 12:42:05.170104 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service May 23 12:42:05.170104 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) May 23 12:42:05.170104 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped May 23 12:42:05.170104 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) May 23 12:42:05.170104 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun May 23 12:42:05.170104 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) May 23 12:42:05.170104 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func May 23 12:42:05.170971 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self.get_or_create( May 23 12:42:05.170971 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create May 23 12:42:05.170971 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application with Lock( May 23 12:42:05.170971 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ May 23 12:42:05.170971 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self._enter() May 23 12:42:05.170971 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter May 23 12:42:05.170971 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) May 23 12:42:05.170971 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create May 23 12:42:05.170971 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self.creator() May 23 12:42:05.170971 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value May 23 12:42:05.170971 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application created_value = creator( May 23 12:42:05.170971 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service May 23 12:42:05.170971 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) May 23 12:42:05.170971 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute_legacy. May 23 12:42:05.170971 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application May 23 12:42:05.174274 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 451/900] 10.222.0.30 () {56 vars in 1132 bytes} [Mon May 23 12:42:05 2022] GET /identity/v3/services/compute_legacy => generated 95 bytes in 163 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) May 23 12:42:05.175281 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60286a5f-8e8f-48c9-b6f6-1f2ebf977ef8 None admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:42:05.194869 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=100 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:42:05.233073 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=100 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:42:05.281486 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b6e5cb4-e7e5-409f-9be1-6209521d6845 None admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:42:05.287845 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60286a5f-8e8f-48c9-b6f6-1f2ebf977ef8 None admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:42:05.319890 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3437c02-0af4-4fd6-8ac0-d97394ad2699 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:42:05.320554 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3437c02-0af4-4fd6-8ac0-d97394ad2699 None admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:42:05.320841 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3437c02-0af4-4fd6-8ac0-d97394ad2699 None admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:42:05.321122 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3437c02-0af4-4fd6-8ac0-d97394ad2699 None admin] PATH_INFO: `/v3/endpoints` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:42:05.321925 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3437c02-0af4-4fd6-8ac0-d97394ad2699 None admin] RBAC: Adding query filter params (interface=public, service_id=41105bf3b54447c5aad3898438571a35, region_id=RegionOne) {{(pid=92057) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} May 23 12:42:05.322247 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3437c02-0af4-4fd6-8ac0-d97394ad2699 None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:42:05.323650 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3437c02-0af4-4fd6-8ac0-d97394ad2699 None admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:42:05.328889 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23261cdb-6281-4451-98cf-930afb37975b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:42:05.329054 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 450/901] 10.222.0.30 () {56 vars in 1251 bytes} [Mon May 23 12:42:05 2022] GET /identity/v3/endpoints?service_id=41105bf3b54447c5aad3898438571a35&interface=public®ion_id=RegionOne => generated 194 bytes in 162 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) May 23 12:42:05.329804 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3437c02-0af4-4fd6-8ac0-d97394ad2699 None admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:42:05.330169 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23261cdb-6281-4451-98cf-930afb37975b None admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:42:05.330463 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23261cdb-6281-4451-98cf-930afb37975b None admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:42:05.330745 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23261cdb-6281-4451-98cf-930afb37975b None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:42:05.406458 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-23261cdb-6281-4451-98cf-930afb37975b None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:42:05.408094 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-23261cdb-6281-4451-98cf-930afb37975b None admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:42:05.428718 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=100 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:42:05.451585 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3437c02-0af4-4fd6-8ac0-d97394ad2699 None admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:42:05.476675 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=100 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:42:05.487313 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3476a493-0d02-4e66-9345-0756d3bdd160 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:42:05.487977 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3476a493-0d02-4e66-9345-0756d3bdd160 None admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:42:05.488803 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3476a493-0d02-4e66-9345-0756d3bdd160 None admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:42:05.489088 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3476a493-0d02-4e66-9345-0756d3bdd160 None admin] PATH_INFO: `/v3/services` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:42:05.489802 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3476a493-0d02-4e66-9345-0756d3bdd160 None admin] RBAC: Adding query filter params (name=compute_legacy) {{(pid=92057) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} May 23 12:42:05.490141 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3476a493-0d02-4e66-9345-0756d3bdd160 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:42:05.491863 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3476a493-0d02-4e66-9345-0756d3bdd160 None admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:42:05.496298 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 451/902] 10.222.0.30 () {56 vars in 1126 bytes} [Mon May 23 12:42:05 2022] GET /identity/v3/services?name=compute_legacy => generated 131 bytes in 167 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) May 23 12:42:05.497022 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3476a493-0d02-4e66-9345-0756d3bdd160 None admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:42:05.538335 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 452/903] 10.222.0.30 () {58 vars in 1282 bytes} [Mon May 23 12:42:05 2022] GET /identity/v3/auth/tokens => generated 1464 bytes in 359 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) May 23 12:42:05.538335 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23261cdb-6281-4451-98cf-930afb37975b None admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:42:05.614917 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3476a493-0d02-4e66-9345-0756d3bdd160 None admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:42:05.645427 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23261cdb-6281-4451-98cf-930afb37975b None admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:42:05.662554 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-072b2740-c19a-4a0b-a27b-280afe48542d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:42:05.663089 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=100 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:42:05.664329 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-072b2740-c19a-4a0b-a27b-280afe48542d None admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:42:05.664532 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-072b2740-c19a-4a0b-a27b-280afe48542d None admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:42:05.664811 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-072b2740-c19a-4a0b-a27b-280afe48542d None admin] PATH_INFO: `/v3/services` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:42:05.665499 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-072b2740-c19a-4a0b-a27b-280afe48542d None admin] RBAC: Authorizing `identity:list_services()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:42:05.666924 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-072b2740-c19a-4a0b-a27b-280afe48542d None admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:42:05.673183 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 452/904] 10.222.0.30 () {56 vars in 1087 bytes} [Mon May 23 12:42:05 2022] GET /identity/v3/services => generated 1497 bytes in 176 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) May 23 12:42:05.674173 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-072b2740-c19a-4a0b-a27b-280afe48542d None admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:42:05.700261 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8312d9d-10ef-4825-b2a9-b70b602fc99a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:42:05.701108 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8312d9d-10ef-4825-b2a9-b70b602fc99a None admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:42:05.701542 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8312d9d-10ef-4825-b2a9-b70b602fc99a None admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:42:05.702045 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8312d9d-10ef-4825-b2a9-b70b602fc99a None admin] PATH_INFO: `/v3/services` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:42:05.703067 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8312d9d-10ef-4825-b2a9-b70b602fc99a None admin] RBAC: Adding query filter params (type=compute_legacy) {{(pid=92056) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} May 23 12:42:05.703571 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8312d9d-10ef-4825-b2a9-b70b602fc99a None admin] RBAC: Authorizing `identity:list_services()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:42:05.705285 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8312d9d-10ef-4825-b2a9-b70b602fc99a None admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:42:05.717378 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 453/905] 10.222.0.30 () {56 vars in 1126 bytes} [Mon May 23 12:42:05 2022] GET /identity/v3/services?type=compute_legacy => generated 386 bytes in 183 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) May 23 12:42:05.718522 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8312d9d-10ef-4825-b2a9-b70b602fc99a None admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:42:05.759569 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=100 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:42:05.818488 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-072b2740-c19a-4a0b-a27b-280afe48542d None admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:42:05.839163 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8312d9d-10ef-4825-b2a9-b70b602fc99a None admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:42:05.861377 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1030fbeb-960e-4ea8-95f2-7098d05b4da5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:42:05.862987 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1030fbeb-960e-4ea8-95f2-7098d05b4da5 None admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:42:05.863952 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1030fbeb-960e-4ea8-95f2-7098d05b4da5 None admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:42:05.864372 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1030fbeb-960e-4ea8-95f2-7098d05b4da5 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:42:05.875983 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1030fbeb-960e-4ea8-95f2-7098d05b4da5 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:42:05.877486 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1030fbeb-960e-4ea8-95f2-7098d05b4da5 None admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:42:05.880464 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5d61717-ef14-4b57-95fc-2bd27a13b450 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:42:05.881215 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5d61717-ef14-4b57-95fc-2bd27a13b450 None admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:42:05.881503 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5d61717-ef14-4b57-95fc-2bd27a13b450 None admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:42:05.881786 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5d61717-ef14-4b57-95fc-2bd27a13b450 None admin] PATH_INFO: `/v3/endpoints` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:42:05.883490 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5d61717-ef14-4b57-95fc-2bd27a13b450 None admin] RBAC: Adding query filter params (interface=public, service_id=5d07d6183d56476994bb9ec72940652b, region_id=RegionOne) {{(pid=92056) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} May 23 12:42:05.883811 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5d61717-ef14-4b57-95fc-2bd27a13b450 None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:42:05.885199 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5d61717-ef14-4b57-95fc-2bd27a13b450 None admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:42:05.893494 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 454/906] 10.222.0.30 () {56 vars in 1251 bytes} [Mon May 23 12:42:05 2022] GET /identity/v3/endpoints?service_id=5d07d6183d56476994bb9ec72940652b&interface=public®ion_id=RegionOne => generated 194 bytes in 175 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) May 23 12:42:05.899365 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5d61717-ef14-4b57-95fc-2bd27a13b450 None admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:42:05.908310 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application [None req-1030fbeb-960e-4ea8-95f2-7098d05b4da5 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. May 23 12:42:05.908310 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application Traceback (most recent call last): May 23 12:42:05.908310 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request May 23 12:42:05.908310 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application rv = self.dispatch_request() May 23 12:42:05.908310 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request May 23 12:42:05.908310 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) May 23 12:42:05.908310 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper May 23 12:42:05.908310 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) May 23 12:42:05.908310 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view May 23 12:42:05.908310 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) May 23 12:42:05.908310 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request May 23 12:42:05.908310 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) May 23 12:42:05.908310 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get May 23 12:42:05.908310 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self._get_domain(domain_id) May 23 12:42:05.908310 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain May 23 12:42:05.909938 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) May 23 12:42:05.909938 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped May 23 12:42:05.909938 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) May 23 12:42:05.909938 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun May 23 12:42:05.909938 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) May 23 12:42:05.909938 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func May 23 12:42:05.909938 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self.get_or_create( May 23 12:42:05.909938 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create May 23 12:42:05.909938 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application with Lock( May 23 12:42:05.909938 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ May 23 12:42:05.909938 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self._enter() May 23 12:42:05.909938 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter May 23 12:42:05.909938 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) May 23 12:42:05.909938 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create May 23 12:42:05.909938 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self.creator() May 23 12:42:05.909938 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value May 23 12:42:05.909938 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application created_value = creator( May 23 12:42:05.911995 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain May 23 12:42:05.911995 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) May 23 12:42:05.911995 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. May 23 12:42:05.911995 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application May 23 12:42:05.914408 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 453/907] 10.222.0.30 () {56 vars in 1108 bytes} [Mon May 23 12:42:05 2022] GET /identity/v3/domains/Default => generated 87 bytes in 240 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) May 23 12:42:05.921487 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1030fbeb-960e-4ea8-95f2-7098d05b4da5 None admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:42:05.956564 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=100 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:42:06.018285 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5d61717-ef14-4b57-95fc-2bd27a13b450 None admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:42:06.057440 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e891c13-53b9-4bde-bfc3-6c22a83e3dc4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:42:06.058239 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e891c13-53b9-4bde-bfc3-6c22a83e3dc4 None admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:42:06.058532 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e891c13-53b9-4bde-bfc3-6c22a83e3dc4 None admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:42:06.058811 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e891c13-53b9-4bde-bfc3-6c22a83e3dc4 None admin] PATH_INFO: `/v3/services` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:42:06.059503 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e891c13-53b9-4bde-bfc3-6c22a83e3dc4 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:42:06.060890 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e891c13-53b9-4bde-bfc3-6c22a83e3dc4 None admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:42:06.061374 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1030fbeb-960e-4ea8-95f2-7098d05b4da5 None admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:42:06.064467 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=100 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:42:06.069676 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 455/908] 10.222.0.30 () {56 vars in 1087 bytes} [Mon May 23 12:42:05 2022] GET /identity/v3/services => generated 1497 bytes in 170 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) May 23 12:42:06.109318 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f28a3765-20b0-4ca9-bd89-c1d1851e5219 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:42:06.110721 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f28a3765-20b0-4ca9-bd89-c1d1851e5219 None admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:42:06.111016 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f28a3765-20b0-4ca9-bd89-c1d1851e5219 None admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:42:06.111280 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f28a3765-20b0-4ca9-bd89-c1d1851e5219 None admin] PATH_INFO: `/v3/domains` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:42:06.112041 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f28a3765-20b0-4ca9-bd89-c1d1851e5219 None admin] RBAC: Adding query filter params (name=Default) {{(pid=92057) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} May 23 12:42:06.112344 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f28a3765-20b0-4ca9-bd89-c1d1851e5219 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:42:06.113822 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f28a3765-20b0-4ca9-bd89-c1d1851e5219 None admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:42:06.130650 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 454/909] 10.222.0.30 () {56 vars in 1109 bytes} [Mon May 23 12:42:05 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 210 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) May 23 12:42:06.135956 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e891c13-53b9-4bde-bfc3-6c22a83e3dc4 None admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:42:06.258484 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e891c13-53b9-4bde-bfc3-6c22a83e3dc4 None admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:42:06.268719 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=100 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:42:06.296344 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6087563d-296e-4055-a01a-3378ad60780a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:42:06.297138 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6087563d-296e-4055-a01a-3378ad60780a None admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:42:06.297418 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6087563d-296e-4055-a01a-3378ad60780a None admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:42:06.297686 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6087563d-296e-4055-a01a-3378ad60780a None admin] PATH_INFO: `/v3/projects/service` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:42:06.307045 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6087563d-296e-4055-a01a-3378ad60780a None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:42:06.308995 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6087563d-296e-4055-a01a-3378ad60780a None admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:42:06.324434 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application [None req-6087563d-296e-4055-a01a-3378ad60780a None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. May 23 12:42:06.324434 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application Traceback (most recent call last): May 23 12:42:06.324434 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request May 23 12:42:06.324434 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application rv = self.dispatch_request() May 23 12:42:06.324434 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request May 23 12:42:06.324434 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) May 23 12:42:06.324434 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper May 23 12:42:06.324434 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) May 23 12:42:06.324434 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view May 23 12:42:06.324434 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) May 23 12:42:06.324434 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request May 23 12:42:06.324434 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) May 23 12:42:06.324434 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get May 23 12:42:06.324434 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self._get_project(project_id) May 23 12:42:06.324434 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project May 23 12:42:06.325408 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) May 23 12:42:06.325408 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped May 23 12:42:06.325408 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) May 23 12:42:06.325408 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun May 23 12:42:06.325408 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) May 23 12:42:06.325408 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func May 23 12:42:06.325408 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self.get_or_create( May 23 12:42:06.325408 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create May 23 12:42:06.325408 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application with Lock( May 23 12:42:06.325408 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ May 23 12:42:06.325408 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self._enter() May 23 12:42:06.325408 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter May 23 12:42:06.325408 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) May 23 12:42:06.325408 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create May 23 12:42:06.325408 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self.creator() May 23 12:42:06.325408 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value May 23 12:42:06.325408 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application created_value = creator( May 23 12:42:06.326351 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project May 23 12:42:06.326351 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) May 23 12:42:06.326351 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project May 23 12:42:06.326351 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() May 23 12:42:06.326351 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project May 23 12:42:06.326351 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) May 23 12:42:06.326351 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. May 23 12:42:06.326351 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application May 23 12:42:06.326351 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 456/910] 10.222.0.30 () {56 vars in 1111 bytes} [Mon May 23 12:42:06 2022] GET /identity/v3/projects/service => generated 88 bytes in 190 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) May 23 12:42:06.333086 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f28a3765-20b0-4ca9-bd89-c1d1851e5219 None admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:42:06.444338 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f28a3765-20b0-4ca9-bd89-c1d1851e5219 None admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:42:06.482007 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1077f460-9a38-4d3c-b2e6-81c47d5919e9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:42:06.482745 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1077f460-9a38-4d3c-b2e6-81c47d5919e9 None admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:42:06.483034 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1077f460-9a38-4d3c-b2e6-81c47d5919e9 None admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:42:06.483316 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1077f460-9a38-4d3c-b2e6-81c47d5919e9 None admin] PATH_INFO: `/v3/projects` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:42:06.484165 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1077f460-9a38-4d3c-b2e6-81c47d5919e9 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=92057) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} May 23 12:42:06.484486 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1077f460-9a38-4d3c-b2e6-81c47d5919e9 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:42:06.485966 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1077f460-9a38-4d3c-b2e6-81c47d5919e9 None admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:42:06.500478 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 455/911] 10.222.0.30 () {56 vars in 1148 bytes} [Mon May 23 12:42:06 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 168 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) May 23 12:42:06.506690 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6087563d-296e-4055-a01a-3378ad60780a None admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:42:06.616237 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6087563d-296e-4055-a01a-3378ad60780a None admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:42:06.652522 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8d14229-c88f-4b13-ace3-945602557c52 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:42:06.653180 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8d14229-c88f-4b13-ace3-945602557c52 None admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:42:06.653463 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8d14229-c88f-4b13-ace3-945602557c52 None admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:42:06.653739 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8d14229-c88f-4b13-ace3-945602557c52 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:42:06.654592 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8d14229-c88f-4b13-ace3-945602557c52 None admin] RBAC: Adding query filter params (role.id=940396c67e6744c784e95e01fe84ad45, scope.project.id=9c389ae65aaa4542bb5a5c4fd0bacf5b, user.id=6d21a56126fd4ccd837c33ccc08ab609) {{(pid=92056) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} May 23 12:42:06.654901 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8d14229-c88f-4b13-ace3-945602557c52 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:42:06.656343 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8d14229-c88f-4b13-ace3-945602557c52 None admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:42:06.662552 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 457/912] 10.222.0.30 () {56 vars in 1374 bytes} [Mon May 23 12:42:06 2022] GET /identity/v3/role_assignments?user.id=6d21a56126fd4ccd837c33ccc08ab609&scope.project.id=9c389ae65aaa4542bb5a5c4fd0bacf5b&role.id=940396c67e6744c784e95e01fe84ad45 => generated 259 bytes in 156 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) May 23 12:42:07.096176 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f3e7592-70f3-4c5d-aa28-e682edc539c8 None None] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:42:07.096672 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f3e7592-70f3-4c5d-aa28-e682edc539c8 None None] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:42:07.096672 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f3e7592-70f3-4c5d-aa28-e682edc539c8 None None] PATH_INFO: `/` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:42:07.098283 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 456/913] 10.222.0.30 () {54 vars in 926 bytes} [Mon May 23 12:42:07 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) May 23 12:42:07.103644 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87600fa0-5f6c-4316-8d30-d2e257c753bc None None] REQUEST_METHOD: `POST` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:42:07.104044 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87600fa0-5f6c-4316-8d30-d2e257c753bc None None] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:42:07.104237 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87600fa0-5f6c-4316-8d30-d2e257c753bc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:42:07.121590 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.auth.core [None req-87600fa0-5f6c-4316-8d30-d2e257c753bc None None] MFA Rules not processed for user `0f7119f8fa5c4445977671140ded91a4`. Rule list: `[]` (Enabled: `True`). {{(pid=92056) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 12:42:07.122875 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.fernet_utils [None req-87600fa0-5f6c-4316-8d30-d2e257c753bc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92056) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 12:42:07.149070 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=100 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:42:07.192279 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 458/914] 10.222.0.30 () {58 vars in 1023 bytes} [Mon May 23 12:42:07 2022] POST /identity/v3/auth/tokens => generated 1464 bytes in 89 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) May 23 12:42:07.214542 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f3e7592-70f3-4c5d-aa28-e682edc539c8 None None] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:42:07.241548 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=100 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:42:07.324055 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f3e7592-70f3-4c5d-aa28-e682edc539c8 None None] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:42:07.364850 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4dff1df6-1f5d-4fb7-99e7-7101ddc8fad1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:42:07.365666 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4dff1df6-1f5d-4fb7-99e7-7101ddc8fad1 None admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:42:07.367109 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4dff1df6-1f5d-4fb7-99e7-7101ddc8fad1 None admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:42:07.367109 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4dff1df6-1f5d-4fb7-99e7-7101ddc8fad1 None admin] PATH_INFO: `/v3/services/object-store` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:42:07.367109 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4dff1df6-1f5d-4fb7-99e7-7101ddc8fad1 None admin] RBAC: Authorizing `identity:get_service(service_id=object-store)` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:42:07.368373 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4dff1df6-1f5d-4fb7-99e7-7101ddc8fad1 None admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:42:07.381082 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application [None req-4dff1df6-1f5d-4fb7-99e7-7101ddc8fad1 None admin] Could not find service: object-store.: keystone.exception.ServiceNotFound: Could not find service: object-store. May 23 12:42:07.381082 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application Traceback (most recent call last): May 23 12:42:07.381082 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service May 23 12:42:07.381082 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) May 23 12:42:07.381082 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service May 23 12:42:07.381082 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() May 23 12:42:07.381082 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service May 23 12:42:07.381082 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) May 23 12:42:07.381082 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. May 23 12:42:07.381082 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application May 23 12:42:07.381082 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: May 23 12:42:07.381082 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application May 23 12:42:07.381082 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application Traceback (most recent call last): May 23 12:42:07.381082 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request May 23 12:42:07.381082 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application rv = self.dispatch_request() May 23 12:42:07.381082 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request May 23 12:42:07.381082 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) May 23 12:42:07.382175 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper May 23 12:42:07.382175 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) May 23 12:42:07.382175 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view May 23 12:42:07.382175 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) May 23 12:42:07.382175 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request May 23 12:42:07.382175 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) May 23 12:42:07.382175 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get May 23 12:42:07.382175 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self._get_service(service_id) May 23 12:42:07.382175 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service May 23 12:42:07.382175 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) May 23 12:42:07.382175 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped May 23 12:42:07.382175 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) May 23 12:42:07.382175 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun May 23 12:42:07.382175 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) May 23 12:42:07.382175 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func May 23 12:42:07.382175 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self.get_or_create( May 23 12:42:07.383644 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create May 23 12:42:07.383644 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application with Lock( May 23 12:42:07.383644 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ May 23 12:42:07.383644 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self._enter() May 23 12:42:07.383644 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter May 23 12:42:07.383644 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) May 23 12:42:07.383644 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create May 23 12:42:07.383644 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self.creator() May 23 12:42:07.383644 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value May 23 12:42:07.383644 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application created_value = creator( May 23 12:42:07.383644 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service May 23 12:42:07.383644 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) May 23 12:42:07.383644 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. May 23 12:42:07.383644 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application May 23 12:42:07.383644 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 457/915] 10.222.0.30 () {56 vars in 1126 bytes} [Mon May 23 12:42:07 2022] GET /identity/v3/services/object-store => generated 93 bytes in 169 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) May 23 12:42:07.388481 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87600fa0-5f6c-4316-8d30-d2e257c753bc None None] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:42:07.490299 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87600fa0-5f6c-4316-8d30-d2e257c753bc None None] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:42:07.526702 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b545c910-09c3-490e-8ecf-8090651f2d3a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:42:07.527363 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b545c910-09c3-490e-8ecf-8090651f2d3a None admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:42:07.527640 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b545c910-09c3-490e-8ecf-8090651f2d3a None admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:42:07.527909 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b545c910-09c3-490e-8ecf-8090651f2d3a None admin] PATH_INFO: `/v3/services` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:42:07.528609 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b545c910-09c3-490e-8ecf-8090651f2d3a None admin] RBAC: Adding query filter params (name=object-store) {{(pid=92056) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} May 23 12:42:07.528911 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b545c910-09c3-490e-8ecf-8090651f2d3a None admin] RBAC: Authorizing `identity:list_services()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:42:07.530305 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b545c910-09c3-490e-8ecf-8090651f2d3a None admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:42:07.535585 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 459/916] 10.222.0.30 () {56 vars in 1122 bytes} [Mon May 23 12:42:07 2022] GET /identity/v3/services?name=object-store => generated 129 bytes in 147 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) May 23 12:42:07.541154 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4dff1df6-1f5d-4fb7-99e7-7101ddc8fad1 None admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:42:07.633303 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=100 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:42:07.646729 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4dff1df6-1f5d-4fb7-99e7-7101ddc8fad1 None admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:42:07.680942 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02becf74-b681-42a1-89f3-e0c706463161 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:42:07.681610 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02becf74-b681-42a1-89f3-e0c706463161 None admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:42:07.681910 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02becf74-b681-42a1-89f3-e0c706463161 None admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:42:07.682198 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02becf74-b681-42a1-89f3-e0c706463161 None admin] PATH_INFO: `/v3/services` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:42:07.682962 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-02becf74-b681-42a1-89f3-e0c706463161 None admin] RBAC: Adding query filter params (type=object-store) {{(pid=92057) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} May 23 12:42:07.683340 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-02becf74-b681-42a1-89f3-e0c706463161 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:42:07.684701 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-02becf74-b681-42a1-89f3-e0c706463161 None admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:42:07.690260 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 458/917] 10.222.0.30 () {56 vars in 1122 bytes} [Mon May 23 12:42:07 2022] GET /identity/v3/services?type=object-store => generated 356 bytes in 149 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) May 23 12:42:07.696762 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b545c910-09c3-490e-8ecf-8090651f2d3a None admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:42:07.726242 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=100 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:42:07.798161 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b545c910-09c3-490e-8ecf-8090651f2d3a None admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:42:07.835212 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2b0f4a6-c33a-4930-8cbb-25f896d8f55f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:42:07.835894 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2b0f4a6-c33a-4930-8cbb-25f896d8f55f None admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:42:07.836177 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2b0f4a6-c33a-4930-8cbb-25f896d8f55f None admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:42:07.836453 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2b0f4a6-c33a-4930-8cbb-25f896d8f55f None admin] PATH_INFO: `/v3/endpoints` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:42:07.837271 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2b0f4a6-c33a-4930-8cbb-25f896d8f55f None admin] RBAC: Adding query filter params (interface=public, service_id=24b330c81aa74757ba04d66aa30e333c, region_id=RegionOne) {{(pid=92056) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} May 23 12:42:07.837583 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2b0f4a6-c33a-4930-8cbb-25f896d8f55f None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:42:07.839057 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2b0f4a6-c33a-4930-8cbb-25f896d8f55f None admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:42:07.845618 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 460/918] 10.222.0.30 () {56 vars in 1251 bytes} [Mon May 23 12:42:07 2022] GET /identity/v3/endpoints?service_id=24b330c81aa74757ba04d66aa30e333c&interface=public®ion_id=RegionOne => generated 194 bytes in 149 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) May 23 12:42:07.851307 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02becf74-b681-42a1-89f3-e0c706463161 None admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:42:07.966395 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02becf74-b681-42a1-89f3-e0c706463161 None admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:42:08.001670 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d819ed3f-43f0-42a7-91c2-f401019fb8dc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:42:08.002371 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d819ed3f-43f0-42a7-91c2-f401019fb8dc None admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:42:08.002671 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d819ed3f-43f0-42a7-91c2-f401019fb8dc None admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:42:08.002948 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d819ed3f-43f0-42a7-91c2-f401019fb8dc None admin] PATH_INFO: `/v3/services` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:42:08.003636 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d819ed3f-43f0-42a7-91c2-f401019fb8dc None admin] RBAC: Authorizing `identity:list_services()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:42:08.005147 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d819ed3f-43f0-42a7-91c2-f401019fb8dc None admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:42:08.011802 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 459/919] 10.222.0.30 () {56 vars in 1087 bytes} [Mon May 23 12:42:07 2022] GET /identity/v3/services => generated 1497 bytes in 161 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) May 23 12:42:08.110925 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e88a849-1378-4ada-ab31-a312628d4ff5 None None] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:42:08.111495 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e88a849-1378-4ada-ab31-a312628d4ff5 None None] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:42:08.111733 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e88a849-1378-4ada-ab31-a312628d4ff5 None None] PATH_INFO: `/` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:42:08.114192 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 461/920] 10.222.0.30 () {54 vars in 926 bytes} [Mon May 23 12:42:08 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) May 23 12:42:08.120035 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7fe7827-e4f0-419c-8354-9489311845e9 None None] REQUEST_METHOD: `POST` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:42:08.120561 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7fe7827-e4f0-419c-8354-9489311845e9 None None] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:42:08.120821 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7fe7827-e4f0-419c-8354-9489311845e9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:42:08.138713 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.auth.core [None req-a7fe7827-e4f0-419c-8354-9489311845e9 None None] MFA Rules not processed for user `0f7119f8fa5c4445977671140ded91a4`. Rule list: `[]` (Enabled: `True`). {{(pid=92057) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 12:42:08.139928 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.fernet_utils [None req-a7fe7827-e4f0-419c-8354-9489311845e9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92057) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 12:42:08.152757 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=100 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:42:08.216631 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 460/921] 10.222.0.30 () {58 vars in 1023 bytes} [Mon May 23 12:42:08 2022] POST /identity/v3/auth/tokens => generated 1464 bytes in 98 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) May 23 12:42:08.240406 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e88a849-1378-4ada-ab31-a312628d4ff5 None None] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:42:08.346134 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=100 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:42:08.356687 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e88a849-1378-4ada-ab31-a312628d4ff5 None None] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:42:08.392538 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39a76460-805a-4cac-97a5-30a96e3b7fce None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:42:08.393339 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39a76460-805a-4cac-97a5-30a96e3b7fce None admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:42:08.393624 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39a76460-805a-4cac-97a5-30a96e3b7fce None admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:42:08.393914 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39a76460-805a-4cac-97a5-30a96e3b7fce None admin] PATH_INFO: `/v3/services/volumev3` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:42:08.395170 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39a76460-805a-4cac-97a5-30a96e3b7fce None admin] RBAC: Authorizing `identity:get_service(service_id=volumev3)` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:42:08.396683 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39a76460-805a-4cac-97a5-30a96e3b7fce None admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:42:08.414271 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application [None req-39a76460-805a-4cac-97a5-30a96e3b7fce None admin] Could not find service: volumev3.: keystone.exception.ServiceNotFound: Could not find service: volumev3. May 23 12:42:08.414271 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application Traceback (most recent call last): May 23 12:42:08.414271 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service May 23 12:42:08.414271 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) May 23 12:42:08.414271 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service May 23 12:42:08.414271 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() May 23 12:42:08.414271 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service May 23 12:42:08.414271 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) May 23 12:42:08.414271 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: volumev3. May 23 12:42:08.414271 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application May 23 12:42:08.414271 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: May 23 12:42:08.414271 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application May 23 12:42:08.414271 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application Traceback (most recent call last): May 23 12:42:08.414271 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request May 23 12:42:08.414271 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application rv = self.dispatch_request() May 23 12:42:08.414271 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request May 23 12:42:08.414271 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) May 23 12:42:08.416633 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper May 23 12:42:08.416633 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) May 23 12:42:08.416633 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view May 23 12:42:08.416633 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) May 23 12:42:08.416633 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request May 23 12:42:08.416633 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) May 23 12:42:08.416633 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get May 23 12:42:08.416633 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self._get_service(service_id) May 23 12:42:08.416633 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service May 23 12:42:08.416633 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) May 23 12:42:08.416633 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped May 23 12:42:08.416633 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) May 23 12:42:08.416633 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun May 23 12:42:08.416633 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) May 23 12:42:08.416633 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func May 23 12:42:08.416633 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self.get_or_create( May 23 12:42:08.417696 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create May 23 12:42:08.417696 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application with Lock( May 23 12:42:08.417696 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ May 23 12:42:08.417696 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self._enter() May 23 12:42:08.417696 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter May 23 12:42:08.417696 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) May 23 12:42:08.417696 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create May 23 12:42:08.417696 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self.creator() May 23 12:42:08.417696 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value May 23 12:42:08.417696 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application created_value = creator( May 23 12:42:08.417696 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service May 23 12:42:08.417696 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) May 23 12:42:08.417696 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: volumev3. May 23 12:42:08.417696 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application May 23 12:42:08.417696 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 462/922] 10.222.0.30 () {56 vars in 1114 bytes} [Mon May 23 12:42:08 2022] GET /identity/v3/services/volumev3 => generated 89 bytes in 176 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) May 23 12:42:08.421463 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7fe7827-e4f0-419c-8354-9489311845e9 None None] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:42:08.480516 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d21da93-88a1-43bd-9c38-6574a049cf9f None None] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:42:08.480803 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d21da93-88a1-43bd-9c38-6574a049cf9f None None] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:42:08.480983 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d21da93-88a1-43bd-9c38-6574a049cf9f None None] PATH_INFO: `/` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:42:08.482625 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 463/923] 10.222.0.30 () {54 vars in 926 bytes} [Mon May 23 12:42:08 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) May 23 12:42:08.488906 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11fa1056-0e29-40ae-8282-460982b457a5 None None] REQUEST_METHOD: `POST` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:42:08.488906 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11fa1056-0e29-40ae-8282-460982b457a5 None None] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:42:08.489121 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11fa1056-0e29-40ae-8282-460982b457a5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:42:08.508819 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.auth.core [None req-11fa1056-0e29-40ae-8282-460982b457a5 None None] MFA Rules not processed for user `0f7119f8fa5c4445977671140ded91a4`. Rule list: `[]` (Enabled: `True`). {{(pid=92056) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 12:42:08.510441 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.fernet_utils [None req-11fa1056-0e29-40ae-8282-460982b457a5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92056) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 12:42:08.534938 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7fe7827-e4f0-419c-8354-9489311845e9 None None] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:42:08.577159 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3aad28d4-5a4c-4576-87c1-a1c9fc2598bb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:42:08.577968 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3aad28d4-5a4c-4576-87c1-a1c9fc2598bb None admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:42:08.578253 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3aad28d4-5a4c-4576-87c1-a1c9fc2598bb None admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:42:08.578527 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3aad28d4-5a4c-4576-87c1-a1c9fc2598bb None admin] PATH_INFO: `/v3/services` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:42:08.579243 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3aad28d4-5a4c-4576-87c1-a1c9fc2598bb None admin] RBAC: Adding query filter params (name=volumev3) {{(pid=92057) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} May 23 12:42:08.579550 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3aad28d4-5a4c-4576-87c1-a1c9fc2598bb None admin] RBAC: Authorizing `identity:list_services()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:42:08.581053 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3aad28d4-5a4c-4576-87c1-a1c9fc2598bb None admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:42:08.587365 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 461/924] 10.222.0.30 () {56 vars in 1114 bytes} [Mon May 23 12:42:08 2022] GET /identity/v3/services?name=volumev3 => generated 125 bytes in 166 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) May 23 12:42:08.592567 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3aad28d4-5a4c-4576-87c1-a1c9fc2598bb None admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:42:08.596264 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 464/925] 10.222.0.30 () {58 vars in 1023 bytes} [Mon May 23 12:42:08 2022] POST /identity/v3/auth/tokens => generated 1464 bytes in 109 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) May 23 12:42:08.621451 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=100 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:42:08.621724 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11fa1056-0e29-40ae-8282-460982b457a5 None None] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:42:08.729809 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=100 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:42:08.741902 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3aad28d4-5a4c-4576-87c1-a1c9fc2598bb None admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:42:08.748753 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11fa1056-0e29-40ae-8282-460982b457a5 None None] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:42:08.790164 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60b5b2bc-cf41-4c2b-9227-43bfff698303 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:42:08.791081 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60b5b2bc-cf41-4c2b-9227-43bfff698303 None admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:42:08.791545 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60b5b2bc-cf41-4c2b-9227-43bfff698303 None admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:42:08.791969 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60b5b2bc-cf41-4c2b-9227-43bfff698303 None admin] PATH_INFO: `/v3/services` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:42:08.792837 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60b5b2bc-cf41-4c2b-9227-43bfff698303 None admin] RBAC: Adding query filter params (type=volumev3) {{(pid=92057) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} May 23 12:42:08.793294 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60b5b2bc-cf41-4c2b-9227-43bfff698303 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:42:08.793694 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7cef971a-6047-474e-b7b4-0da8d569ef47 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:42:08.794634 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7cef971a-6047-474e-b7b4-0da8d569ef47 None admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:42:08.795006 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7cef971a-6047-474e-b7b4-0da8d569ef47 None admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:42:08.795370 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7cef971a-6047-474e-b7b4-0da8d569ef47 None admin] PATH_INFO: `/v3/services/compute_legacy` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:42:08.795553 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60b5b2bc-cf41-4c2b-9227-43bfff698303 None admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:42:08.796214 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7cef971a-6047-474e-b7b4-0da8d569ef47 None admin] RBAC: Authorizing `identity:get_service(service_id=compute_legacy)` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:42:08.797868 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7cef971a-6047-474e-b7b4-0da8d569ef47 None admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:42:08.807483 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 462/926] 10.222.0.30 () {56 vars in 1114 bytes} [Mon May 23 12:42:08 2022] GET /identity/v3/services?type=volumev3 => generated 362 bytes in 214 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) May 23 12:42:08.808767 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60b5b2bc-cf41-4c2b-9227-43bfff698303 None admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:42:08.821153 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application [None req-7cef971a-6047-474e-b7b4-0da8d569ef47 None admin] Could not find service: compute_legacy.: keystone.exception.ServiceNotFound: Could not find service: compute_legacy. May 23 12:42:08.821153 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application Traceback (most recent call last): May 23 12:42:08.821153 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service May 23 12:42:08.821153 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) May 23 12:42:08.821153 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service May 23 12:42:08.821153 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() May 23 12:42:08.821153 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service May 23 12:42:08.821153 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) May 23 12:42:08.821153 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute_legacy. May 23 12:42:08.821153 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application May 23 12:42:08.821153 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: May 23 12:42:08.821153 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application May 23 12:42:08.821153 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application Traceback (most recent call last): May 23 12:42:08.821153 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request May 23 12:42:08.821153 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application rv = self.dispatch_request() May 23 12:42:08.821153 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request May 23 12:42:08.822275 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) May 23 12:42:08.822275 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper May 23 12:42:08.822275 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) May 23 12:42:08.822275 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view May 23 12:42:08.822275 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) May 23 12:42:08.822275 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request May 23 12:42:08.822275 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) May 23 12:42:08.822275 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get May 23 12:42:08.822275 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self._get_service(service_id) May 23 12:42:08.822275 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service May 23 12:42:08.822275 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) May 23 12:42:08.822275 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped May 23 12:42:08.822275 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) May 23 12:42:08.822275 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun May 23 12:42:08.822275 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) May 23 12:42:08.822275 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func May 23 12:42:08.823165 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self.get_or_create( May 23 12:42:08.823165 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create May 23 12:42:08.823165 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application with Lock( May 23 12:42:08.823165 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ May 23 12:42:08.823165 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self._enter() May 23 12:42:08.823165 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter May 23 12:42:08.823165 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) May 23 12:42:08.823165 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create May 23 12:42:08.823165 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self.creator() May 23 12:42:08.823165 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value May 23 12:42:08.823165 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application created_value = creator( May 23 12:42:08.823165 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service May 23 12:42:08.823165 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) May 23 12:42:08.823165 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute_legacy. May 23 12:42:08.823165 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application May 23 12:42:08.825988 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 465/927] 10.222.0.30 () {56 vars in 1132 bytes} [Mon May 23 12:42:08 2022] GET /identity/v3/services/compute_legacy => generated 95 bytes in 202 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) May 23 12:42:08.830656 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7cef971a-6047-474e-b7b4-0da8d569ef47 None admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:42:08.921944 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=100 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:42:08.954370 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60b5b2bc-cf41-4c2b-9227-43bfff698303 None admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:42:08.969530 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7cef971a-6047-474e-b7b4-0da8d569ef47 None admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:42:09.007426 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-474565f6-cbc0-487a-99d2-f6c3219219b0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:42:09.008121 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-474565f6-cbc0-487a-99d2-f6c3219219b0 None admin] REQUEST_METHOD: `POST` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:42:09.008876 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-474565f6-cbc0-487a-99d2-f6c3219219b0 None admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:42:09.009236 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-474565f6-cbc0-487a-99d2-f6c3219219b0 None admin] PATH_INFO: `/v3/endpoints` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:42:09.010773 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-474565f6-cbc0-487a-99d2-f6c3219219b0 None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:42:09.012270 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-474565f6-cbc0-487a-99d2-f6c3219219b0 None admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:42:09.014442 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f633ed82-a908-4a28-a11d-e6af5fa7d80a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:42:09.015111 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f633ed82-a908-4a28-a11d-e6af5fa7d80a None admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:42:09.015393 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f633ed82-a908-4a28-a11d-e6af5fa7d80a None admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:42:09.015850 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f633ed82-a908-4a28-a11d-e6af5fa7d80a None admin] PATH_INFO: `/v3/services` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:42:09.016873 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f633ed82-a908-4a28-a11d-e6af5fa7d80a None admin] RBAC: Adding query filter params (name=compute_legacy) {{(pid=92056) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} May 23 12:42:09.017187 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f633ed82-a908-4a28-a11d-e6af5fa7d80a None admin] RBAC: Authorizing `identity:list_services()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:42:09.018864 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f633ed82-a908-4a28-a11d-e6af5fa7d80a None admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:42:09.026366 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 466/928] 10.222.0.30 () {56 vars in 1126 bytes} [Mon May 23 12:42:08 2022] GET /identity/v3/services?name=compute_legacy => generated 131 bytes in 196 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) May 23 12:42:09.027669 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a9f52c1-52f0-49ea-9121-4a4423ddc3b3 None None] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:42:09.027963 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a9f52c1-52f0-49ea-9121-4a4423ddc3b3 None None] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:42:09.028241 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a9f52c1-52f0-49ea-9121-4a4423ddc3b3 None None] PATH_INFO: `/` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:42:09.029923 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 467/929] 10.222.0.30 () {54 vars in 926 bytes} [Mon May 23 12:42:09 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) May 23 12:42:09.031970 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a9f52c1-52f0-49ea-9121-4a4423ddc3b3 None None] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:42:09.047786 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=100 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:42:09.101223 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 463/930] 10.222.0.30 () {60 vars in 1144 bytes} [Mon May 23 12:42:08 2022] POST /identity/v3/endpoints => generated 346 bytes in 293 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) May 23 12:42:09.102836 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f969c36-79bd-4dbd-a65a-13e9a6ed0023 None None] REQUEST_METHOD: `POST` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:42:09.103190 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f969c36-79bd-4dbd-a65a-13e9a6ed0023 None None] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:42:09.103540 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f969c36-79bd-4dbd-a65a-13e9a6ed0023 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:42:09.123174 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.auth.core [None req-2f969c36-79bd-4dbd-a65a-13e9a6ed0023 None None] MFA Rules not processed for user `0f7119f8fa5c4445977671140ded91a4`. Rule list: `[]` (Enabled: `True`). {{(pid=92057) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 12:42:09.125244 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.fernet_utils [None req-2f969c36-79bd-4dbd-a65a-13e9a6ed0023 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92057) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 12:42:09.162521 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a9f52c1-52f0-49ea-9121-4a4423ddc3b3 None None] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:42:09.211235 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9c73176-f957-4270-9751-0ff08e6f84de None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:42:09.212539 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9c73176-f957-4270-9751-0ff08e6f84de None admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:42:09.213133 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9c73176-f957-4270-9751-0ff08e6f84de None admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:42:09.213615 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9c73176-f957-4270-9751-0ff08e6f84de None admin] PATH_INFO: `/v3/services` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:42:09.214666 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9c73176-f957-4270-9751-0ff08e6f84de None admin] RBAC: Adding query filter params (type=compute_legacy) {{(pid=92056) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} May 23 12:42:09.215192 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9c73176-f957-4270-9751-0ff08e6f84de None admin] RBAC: Authorizing `identity:list_services()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:42:09.216974 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9c73176-f957-4270-9751-0ff08e6f84de None admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:42:09.224194 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 468/931] 10.222.0.30 () {56 vars in 1126 bytes} [Mon May 23 12:42:09 2022] GET /identity/v3/services?type=compute_legacy => generated 386 bytes in 192 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) May 23 12:42:09.229440 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9c73176-f957-4270-9751-0ff08e6f84de None admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:42:09.234030 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 464/932] 10.222.0.30 () {58 vars in 1023 bytes} [Mon May 23 12:42:09 2022] POST /identity/v3/auth/tokens => generated 1464 bytes in 132 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) May 23 12:42:09.259447 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f969c36-79bd-4dbd-a65a-13e9a6ed0023 None None] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:42:09.330275 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:42:09.335892 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=100 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:42:09.375682 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9c73176-f957-4270-9751-0ff08e6f84de None admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:42:09.385240 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f969c36-79bd-4dbd-a65a-13e9a6ed0023 None None] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:42:09.412891 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32e0cf1e-903e-4d39-9cde-4cb9de1deecd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:42:09.413649 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32e0cf1e-903e-4d39-9cde-4cb9de1deecd None admin] REQUEST_METHOD: `POST` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:42:09.413958 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32e0cf1e-903e-4d39-9cde-4cb9de1deecd None admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:42:09.414231 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32e0cf1e-903e-4d39-9cde-4cb9de1deecd None admin] PATH_INFO: `/v3/endpoints` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:42:09.414977 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32e0cf1e-903e-4d39-9cde-4cb9de1deecd None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:42:09.416341 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32e0cf1e-903e-4d39-9cde-4cb9de1deecd None admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:42:09.431201 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b2691a8-c289-41fc-a358-774e2f68b969 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:42:09.432291 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b2691a8-c289-41fc-a358-774e2f68b969 None admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:42:09.433114 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b2691a8-c289-41fc-a358-774e2f68b969 None admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:42:09.433114 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b2691a8-c289-41fc-a358-774e2f68b969 None admin] PATH_INFO: `/v3/roles/ResellerAdmin` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:42:09.444422 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b2691a8-c289-41fc-a358-774e2f68b969 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} May 23 12:42:09.453005 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application [None req-3b2691a8-c289-41fc-a358-774e2f68b969 None admin] Could not find role: ResellerAdmin.: keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. May 23 12:42:09.453005 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application Traceback (most recent call last): May 23 12:42:09.453005 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request May 23 12:42:09.453005 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application rv = self.dispatch_request() May 23 12:42:09.453005 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request May 23 12:42:09.453005 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) May 23 12:42:09.453005 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper May 23 12:42:09.453005 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) May 23 12:42:09.453005 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view May 23 12:42:09.453005 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) May 23 12:42:09.453005 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request May 23 12:42:09.453005 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) May 23 12:42:09.453005 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get May 23 12:42:09.453005 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self._get_role(role_id) May 23 12:42:09.453005 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role May 23 12:42:09.453005 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') May 23 12:42:09.454105 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call May 23 12:42:09.454105 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( May 23 12:42:09.454105 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data May 23 12:42:09.454105 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) May 23 12:42:09.454105 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped May 23 12:42:09.454105 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) May 23 12:42:09.454105 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun May 23 12:42:09.454105 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) May 23 12:42:09.454105 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func May 23 12:42:09.454105 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self.get_or_create( May 23 12:42:09.454105 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create May 23 12:42:09.454105 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application with Lock( May 23 12:42:09.454105 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ May 23 12:42:09.454105 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self._enter() May 23 12:42:09.454105 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter May 23 12:42:09.454105 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) May 23 12:42:09.455190 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create May 23 12:42:09.455190 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self.creator() May 23 12:42:09.455190 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value May 23 12:42:09.455190 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application created_value = creator( May 23 12:42:09.455190 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role May 23 12:42:09.455190 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) May 23 12:42:09.455190 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role May 23 12:42:09.455190 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() May 23 12:42:09.455190 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role May 23 12:42:09.455190 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) May 23 12:42:09.455190 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. May 23 12:42:09.455190 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application May 23 12:42:09.455190 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 465/933] 10.222.0.30 () {56 vars in 1120 bytes} [Mon May 23 12:42:09 2022] GET /identity/v3/roles/ResellerAdmin => generated 91 bytes in 196 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) May 23 12:42:09.462761 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b2691a8-c289-41fc-a358-774e2f68b969 None admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:42:09.494109 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 469/934] 10.222.0.30 () {60 vars in 1144 bytes} [Mon May 23 12:42:09 2022] POST /identity/v3/endpoints => generated 347 bytes in 265 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) May 23 12:42:09.587378 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b2691a8-c289-41fc-a358-774e2f68b969 None admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:42:09.614046 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=100 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:42:09.637311 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39f8a8ea-1bb7-4661-8600-3128135cf992 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:42:09.638029 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39f8a8ea-1bb7-4661-8600-3128135cf992 None admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:42:09.638377 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39f8a8ea-1bb7-4661-8600-3128135cf992 None admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:42:09.638650 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39f8a8ea-1bb7-4661-8600-3128135cf992 None admin] PATH_INFO: `/v3/roles` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:42:09.639432 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39f8a8ea-1bb7-4661-8600-3128135cf992 None admin] RBAC: Adding query filter params (name=ResellerAdmin) {{(pid=92057) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} May 23 12:42:09.639802 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39f8a8ea-1bb7-4661-8600-3128135cf992 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:42:09.641293 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39f8a8ea-1bb7-4661-8600-3128135cf992 None admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:42:09.653084 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 466/935] 10.222.0.30 () {56 vars in 1115 bytes} [Mon May 23 12:42:09 2022] GET /identity/v3/roles?name=ResellerAdmin => generated 338 bytes in 190 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) May 23 12:42:09.658750 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32e0cf1e-903e-4d39-9cde-4cb9de1deecd None admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:42:09.776292 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32e0cf1e-903e-4d39-9cde-4cb9de1deecd None admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:42:09.806485 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:42:09.814807 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69d55ac7-e08a-4608-82db-d8a8753bc9ff None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:42:09.815725 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69d55ac7-e08a-4608-82db-d8a8753bc9ff None admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:42:09.816607 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69d55ac7-e08a-4608-82db-d8a8753bc9ff None admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:42:09.816970 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69d55ac7-e08a-4608-82db-d8a8753bc9ff None admin] PATH_INFO: `/v3/domains/Default` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:42:09.828093 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69d55ac7-e08a-4608-82db-d8a8753bc9ff None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:42:09.832767 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69d55ac7-e08a-4608-82db-d8a8753bc9ff None admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:42:09.848414 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application [None req-69d55ac7-e08a-4608-82db-d8a8753bc9ff None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. May 23 12:42:09.848414 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application Traceback (most recent call last): May 23 12:42:09.848414 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request May 23 12:42:09.848414 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application rv = self.dispatch_request() May 23 12:42:09.848414 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request May 23 12:42:09.848414 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) May 23 12:42:09.848414 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper May 23 12:42:09.848414 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) May 23 12:42:09.848414 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view May 23 12:42:09.848414 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) May 23 12:42:09.848414 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request May 23 12:42:09.848414 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) May 23 12:42:09.848414 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get May 23 12:42:09.848414 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self._get_domain(domain_id) May 23 12:42:09.848414 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain May 23 12:42:09.849412 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) May 23 12:42:09.849412 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped May 23 12:42:09.849412 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) May 23 12:42:09.849412 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun May 23 12:42:09.849412 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) May 23 12:42:09.849412 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func May 23 12:42:09.849412 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self.get_or_create( May 23 12:42:09.849412 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create May 23 12:42:09.849412 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application with Lock( May 23 12:42:09.849412 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ May 23 12:42:09.849412 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self._enter() May 23 12:42:09.849412 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter May 23 12:42:09.849412 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) May 23 12:42:09.849412 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create May 23 12:42:09.849412 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self.creator() May 23 12:42:09.849412 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value May 23 12:42:09.849412 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application created_value = creator( May 23 12:42:09.850364 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain May 23 12:42:09.850364 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) May 23 12:42:09.850364 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. May 23 12:42:09.850364 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application May 23 12:42:09.854103 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 470/936] 10.222.0.30 () {56 vars in 1108 bytes} [Mon May 23 12:42:09 2022] GET /identity/v3/domains/Default => generated 87 bytes in 192 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) May 23 12:42:09.855456 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39f8a8ea-1bb7-4661-8600-3128135cf992 None admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:42:09.965784 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39f8a8ea-1bb7-4661-8600-3128135cf992 None admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:42:10.007857 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff819cb7-51a5-4686-a183-9b6ee8db8216 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:42:10.008618 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff819cb7-51a5-4686-a183-9b6ee8db8216 None admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:42:10.008934 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff819cb7-51a5-4686-a183-9b6ee8db8216 None admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:42:10.009232 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff819cb7-51a5-4686-a183-9b6ee8db8216 None admin] PATH_INFO: `/v3/domains` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:42:10.010057 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff819cb7-51a5-4686-a183-9b6ee8db8216 None admin] RBAC: Adding query filter params (name=Default) {{(pid=92057) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} May 23 12:42:10.010372 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff819cb7-51a5-4686-a183-9b6ee8db8216 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:42:10.011940 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff819cb7-51a5-4686-a183-9b6ee8db8216 None admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:42:10.026652 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 467/937] 10.222.0.30 () {56 vars in 1109 bytes} [Mon May 23 12:42:09 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 172 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) May 23 12:42:10.033174 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69d55ac7-e08a-4608-82db-d8a8753bc9ff None admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:42:10.153484 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69d55ac7-e08a-4608-82db-d8a8753bc9ff None admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:42:10.195282 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1c5b027-72a0-4e09-899c-9158a61a5e7f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:42:10.196243 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1c5b027-72a0-4e09-899c-9158a61a5e7f None admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:42:10.196528 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1c5b027-72a0-4e09-899c-9158a61a5e7f None admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:42:10.196902 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1c5b027-72a0-4e09-899c-9158a61a5e7f None admin] PATH_INFO: `/v3/users/glance-swift` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:42:10.206067 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1c5b027-72a0-4e09-899c-9158a61a5e7f None admin] RBAC: Authorizing `identity:get_user(user_id=glance-swift)` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:42:10.208110 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1c5b027-72a0-4e09-899c-9158a61a5e7f None admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:42:10.224629 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application [None req-f1c5b027-72a0-4e09-899c-9158a61a5e7f None admin] Could not find user: glance-swift.: keystone.exception.UserNotFound: Could not find user: glance-swift. May 23 12:42:10.224629 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application Traceback (most recent call last): May 23 12:42:10.224629 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request May 23 12:42:10.224629 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application rv = self.dispatch_request() May 23 12:42:10.224629 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request May 23 12:42:10.224629 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) May 23 12:42:10.224629 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper May 23 12:42:10.224629 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) May 23 12:42:10.224629 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view May 23 12:42:10.224629 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) May 23 12:42:10.224629 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request May 23 12:42:10.224629 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) May 23 12:42:10.224629 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get May 23 12:42:10.224629 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self._get_user(user_id) May 23 12:42:10.224629 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user May 23 12:42:10.224629 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) May 23 12:42:10.225660 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped May 23 12:42:10.225660 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) May 23 12:42:10.225660 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper May 23 12:42:10.225660 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) May 23 12:42:10.225660 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper May 23 12:42:10.225660 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) May 23 12:42:10.225660 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun May 23 12:42:10.225660 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) May 23 12:42:10.225660 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func May 23 12:42:10.225660 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self.get_or_create( May 23 12:42:10.225660 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create May 23 12:42:10.225660 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application with Lock( May 23 12:42:10.225660 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ May 23 12:42:10.225660 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self._enter() May 23 12:42:10.225660 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter May 23 12:42:10.225660 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) May 23 12:42:10.225660 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create May 23 12:42:10.225660 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self.creator() May 23 12:42:10.227077 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value May 23 12:42:10.227077 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application created_value = creator( May 23 12:42:10.227077 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user May 23 12:42:10.227077 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) May 23 12:42:10.227077 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user May 23 12:42:10.227077 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) May 23 12:42:10.227077 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user May 23 12:42:10.227077 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) May 23 12:42:10.227077 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: glance-swift. May 23 12:42:10.227077 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application May 23 12:42:10.227077 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 471/938] 10.222.0.30 () {56 vars in 1117 bytes} [Mon May 23 12:42:10 2022] GET /identity/v3/users/glance-swift => generated 90 bytes in 194 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) May 23 12:42:10.232034 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff819cb7-51a5-4686-a183-9b6ee8db8216 None admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:42:10.278443 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=100 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:42:10.341634 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a481084-a412-4077-8c5b-a4ced36c2fc4 None None] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:42:10.341984 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a481084-a412-4077-8c5b-a4ced36c2fc4 None None] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:42:10.342209 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a481084-a412-4077-8c5b-a4ced36c2fc4 None None] PATH_INFO: `/` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:42:10.343826 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 472/939] 10.222.0.30 () {54 vars in 926 bytes} [Mon May 23 12:42:10 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) May 23 12:42:10.344642 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff819cb7-51a5-4686-a183-9b6ee8db8216 None admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:42:10.349743 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0335b4d-08d4-479a-9538-86eb770c531c None None] REQUEST_METHOD: `POST` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:42:10.350070 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0335b4d-08d4-479a-9538-86eb770c531c None None] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:42:10.350351 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0335b4d-08d4-479a-9538-86eb770c531c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:42:10.367754 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.auth.core [None req-e0335b4d-08d4-479a-9538-86eb770c531c None None] MFA Rules not processed for user `0f7119f8fa5c4445977671140ded91a4`. Rule list: `[]` (Enabled: `True`). {{(pid=92056) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 12:42:10.368959 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.fernet_utils [None req-e0335b4d-08d4-479a-9538-86eb770c531c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92056) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 12:42:10.391407 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2edc220d-4c72-4f24-b3f4-7b926371ccd6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:42:10.392680 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2edc220d-4c72-4f24-b3f4-7b926371ccd6 None admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:42:10.393009 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2edc220d-4c72-4f24-b3f4-7b926371ccd6 None admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:42:10.393691 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2edc220d-4c72-4f24-b3f4-7b926371ccd6 None admin] PATH_INFO: `/v3/users` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:42:10.395432 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2edc220d-4c72-4f24-b3f4-7b926371ccd6 None admin] RBAC: Adding query filter params (domain_id=default, name=glance-swift) {{(pid=92057) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} May 23 12:42:10.396055 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2edc220d-4c72-4f24-b3f4-7b926371ccd6 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:42:10.398479 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2edc220d-4c72-4f24-b3f4-7b926371ccd6 None admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:42:10.425629 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 468/940] 10.222.0.30 () {56 vars in 1149 bytes} [Mon May 23 12:42:10 2022] GET /identity/v3/users?domain_id=default&name=glance-swift => generated 384 bytes in 194 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) May 23 12:42:10.430161 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=100 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:42:10.431494 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2edc220d-4c72-4f24-b3f4-7b926371ccd6 None admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:42:10.453580 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 473/941] 10.222.0.30 () {58 vars in 1023 bytes} [Mon May 23 12:42:10 2022] POST /identity/v3/auth/tokens => generated 1464 bytes in 97 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) May 23 12:42:10.471611 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0335b4d-08d4-479a-9538-86eb770c531c None None] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:42:10.558265 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=100 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:42:10.564130 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2edc220d-4c72-4f24-b3f4-7b926371ccd6 None admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:42:10.599548 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0335b4d-08d4-479a-9538-86eb770c531c None None] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:42:10.605851 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4a8bd35-7f7b-495c-a9eb-89a59d5563b7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:42:10.606682 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4a8bd35-7f7b-495c-a9eb-89a59d5563b7 None admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:42:10.606962 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4a8bd35-7f7b-495c-a9eb-89a59d5563b7 None admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:42:10.607232 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4a8bd35-7f7b-495c-a9eb-89a59d5563b7 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:42:10.621621 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4a8bd35-7f7b-495c-a9eb-89a59d5563b7 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:42:10.623426 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4a8bd35-7f7b-495c-a9eb-89a59d5563b7 None admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:42:10.646314 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application [None req-e4a8bd35-7f7b-495c-a9eb-89a59d5563b7 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. May 23 12:42:10.646314 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application Traceback (most recent call last): May 23 12:42:10.646314 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request May 23 12:42:10.646314 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application rv = self.dispatch_request() May 23 12:42:10.646314 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request May 23 12:42:10.646314 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) May 23 12:42:10.646314 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper May 23 12:42:10.646314 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) May 23 12:42:10.646314 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view May 23 12:42:10.646314 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) May 23 12:42:10.646314 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request May 23 12:42:10.646314 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) May 23 12:42:10.646314 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get May 23 12:42:10.646314 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self._get_domain(domain_id) May 23 12:42:10.646314 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain May 23 12:42:10.648185 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) May 23 12:42:10.648185 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped May 23 12:42:10.648185 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) May 23 12:42:10.648185 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun May 23 12:42:10.648185 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) May 23 12:42:10.648185 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func May 23 12:42:10.648185 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self.get_or_create( May 23 12:42:10.648185 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create May 23 12:42:10.648185 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application with Lock( May 23 12:42:10.648185 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ May 23 12:42:10.648185 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self._enter() May 23 12:42:10.648185 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter May 23 12:42:10.648185 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) May 23 12:42:10.648185 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create May 23 12:42:10.648185 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self.creator() May 23 12:42:10.648185 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value May 23 12:42:10.648185 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application created_value = creator( May 23 12:42:10.649233 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain May 23 12:42:10.649233 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) May 23 12:42:10.649233 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. May 23 12:42:10.649233 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application May 23 12:42:10.650419 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eac6752d-d88c-47ca-b316-aa37dcb66673 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:42:10.652367 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eac6752d-d88c-47ca-b316-aa37dcb66673 None admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:42:10.652957 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 469/942] 10.222.0.30 () {56 vars in 1108 bytes} [Mon May 23 12:42:10 2022] GET /identity/v3/domains/Default => generated 87 bytes in 217 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) May 23 12:42:10.653437 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eac6752d-d88c-47ca-b316-aa37dcb66673 None admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:42:10.654234 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eac6752d-d88c-47ca-b316-aa37dcb66673 None admin] PATH_INFO: `/v3/services/object-store` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:42:10.655427 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4a8bd35-7f7b-495c-a9eb-89a59d5563b7 None admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:42:10.656177 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eac6752d-d88c-47ca-b316-aa37dcb66673 None admin] RBAC: Authorizing `identity:get_service(service_id=object-store)` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:42:10.659614 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eac6752d-d88c-47ca-b316-aa37dcb66673 None admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:42:10.680488 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application [None req-eac6752d-d88c-47ca-b316-aa37dcb66673 None admin] Could not find service: object-store.: keystone.exception.ServiceNotFound: Could not find service: object-store. May 23 12:42:10.680488 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application Traceback (most recent call last): May 23 12:42:10.680488 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service May 23 12:42:10.680488 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) May 23 12:42:10.680488 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service May 23 12:42:10.680488 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() May 23 12:42:10.680488 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service May 23 12:42:10.680488 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) May 23 12:42:10.680488 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. May 23 12:42:10.680488 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application May 23 12:42:10.680488 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: May 23 12:42:10.680488 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application May 23 12:42:10.680488 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application Traceback (most recent call last): May 23 12:42:10.680488 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request May 23 12:42:10.680488 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application rv = self.dispatch_request() May 23 12:42:10.680488 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request May 23 12:42:10.680488 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) May 23 12:42:10.681552 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper May 23 12:42:10.681552 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) May 23 12:42:10.681552 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view May 23 12:42:10.681552 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) May 23 12:42:10.681552 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request May 23 12:42:10.681552 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) May 23 12:42:10.681552 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get May 23 12:42:10.681552 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self._get_service(service_id) May 23 12:42:10.681552 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service May 23 12:42:10.681552 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) May 23 12:42:10.681552 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped May 23 12:42:10.681552 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) May 23 12:42:10.681552 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun May 23 12:42:10.681552 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) May 23 12:42:10.681552 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func May 23 12:42:10.681552 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self.get_or_create( May 23 12:42:10.682464 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create May 23 12:42:10.682464 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application with Lock( May 23 12:42:10.682464 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ May 23 12:42:10.682464 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self._enter() May 23 12:42:10.682464 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter May 23 12:42:10.682464 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) May 23 12:42:10.682464 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create May 23 12:42:10.682464 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self.creator() May 23 12:42:10.682464 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value May 23 12:42:10.682464 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application created_value = creator( May 23 12:42:10.682464 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service May 23 12:42:10.682464 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) May 23 12:42:10.682464 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. May 23 12:42:10.682464 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application May 23 12:42:10.686872 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 474/943] 10.222.0.30 () {56 vars in 1126 bytes} [Mon May 23 12:42:10 2022] GET /identity/v3/services/object-store => generated 93 bytes in 216 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) May 23 12:42:10.694748 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eac6752d-d88c-47ca-b316-aa37dcb66673 None admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:42:10.748882 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=100 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:42:10.798813 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4a8bd35-7f7b-495c-a9eb-89a59d5563b7 None admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:42:10.824883 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eac6752d-d88c-47ca-b316-aa37dcb66673 None admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:42:10.835565 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-838b4f33-0e9a-4fa7-81d2-2331bbab2517 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:42:10.835988 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-838b4f33-0e9a-4fa7-81d2-2331bbab2517 None admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:42:10.836270 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-838b4f33-0e9a-4fa7-81d2-2331bbab2517 None admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:42:10.836547 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-838b4f33-0e9a-4fa7-81d2-2331bbab2517 None admin] PATH_INFO: `/v3/domains` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:42:10.837265 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-838b4f33-0e9a-4fa7-81d2-2331bbab2517 None admin] RBAC: Adding query filter params (name=Default) {{(pid=92057) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} May 23 12:42:10.837572 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-838b4f33-0e9a-4fa7-81d2-2331bbab2517 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:42:10.838982 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-838b4f33-0e9a-4fa7-81d2-2331bbab2517 None admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:42:10.856098 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 470/944] 10.222.0.30 () {56 vars in 1109 bytes} [Mon May 23 12:42:10 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 197 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) May 23 12:42:10.857087 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-838b4f33-0e9a-4fa7-81d2-2331bbab2517 None admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:42:10.865337 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e4887b3-c1e7-485e-b622-f1548f75c374 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:42:10.866419 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e4887b3-c1e7-485e-b622-f1548f75c374 None admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:42:10.866906 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e4887b3-c1e7-485e-b622-f1548f75c374 None admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:42:10.867380 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e4887b3-c1e7-485e-b622-f1548f75c374 None admin] PATH_INFO: `/v3/services` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:42:10.868327 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e4887b3-c1e7-485e-b622-f1548f75c374 None admin] RBAC: Adding query filter params (name=object-store) {{(pid=92056) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} May 23 12:42:10.868865 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e4887b3-c1e7-485e-b622-f1548f75c374 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:42:10.869275 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=100 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:42:10.871242 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e4887b3-c1e7-485e-b622-f1548f75c374 None admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:42:10.877207 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 475/945] 10.222.0.30 () {56 vars in 1122 bytes} [Mon May 23 12:42:10 2022] GET /identity/v3/services?name=object-store => generated 129 bytes in 183 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) May 23 12:42:10.882943 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e4887b3-c1e7-485e-b622-f1548f75c374 None admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:42:10.968178 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-838b4f33-0e9a-4fa7-81d2-2331bbab2517 None admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:42:11.007878 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20cafd69-5d49-4962-8126-4c0a1304e852 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:42:11.008769 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20cafd69-5d49-4962-8126-4c0a1304e852 None admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:42:11.009063 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20cafd69-5d49-4962-8126-4c0a1304e852 None admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:42:11.009339 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20cafd69-5d49-4962-8126-4c0a1304e852 None admin] PATH_INFO: `/v3/projects/service` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:42:11.012739 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=100 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:42:11.017459 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20cafd69-5d49-4962-8126-4c0a1304e852 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:42:11.017961 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e4887b3-c1e7-485e-b622-f1548f75c374 None admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:42:11.018973 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20cafd69-5d49-4962-8126-4c0a1304e852 None admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:42:11.034480 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application [None req-20cafd69-5d49-4962-8126-4c0a1304e852 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. May 23 12:42:11.034480 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application Traceback (most recent call last): May 23 12:42:11.034480 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request May 23 12:42:11.034480 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application rv = self.dispatch_request() May 23 12:42:11.034480 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request May 23 12:42:11.034480 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) May 23 12:42:11.034480 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper May 23 12:42:11.034480 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) May 23 12:42:11.034480 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view May 23 12:42:11.034480 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) May 23 12:42:11.034480 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request May 23 12:42:11.034480 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) May 23 12:42:11.034480 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get May 23 12:42:11.034480 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self._get_project(project_id) May 23 12:42:11.034480 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project May 23 12:42:11.035503 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) May 23 12:42:11.035503 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped May 23 12:42:11.035503 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) May 23 12:42:11.035503 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun May 23 12:42:11.035503 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) May 23 12:42:11.035503 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func May 23 12:42:11.035503 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self.get_or_create( May 23 12:42:11.035503 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create May 23 12:42:11.035503 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application with Lock( May 23 12:42:11.035503 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ May 23 12:42:11.035503 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self._enter() May 23 12:42:11.035503 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter May 23 12:42:11.035503 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) May 23 12:42:11.035503 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create May 23 12:42:11.035503 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self.creator() May 23 12:42:11.035503 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value May 23 12:42:11.035503 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application created_value = creator( May 23 12:42:11.036437 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project May 23 12:42:11.036437 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) May 23 12:42:11.036437 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project May 23 12:42:11.036437 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() May 23 12:42:11.036437 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project May 23 12:42:11.036437 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) May 23 12:42:11.036437 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. May 23 12:42:11.036437 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application May 23 12:42:11.036437 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 471/946] 10.222.0.30 () {56 vars in 1111 bytes} [Mon May 23 12:42:10 2022] GET /identity/v3/projects/service => generated 88 bytes in 179 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) May 23 12:42:11.041626 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20cafd69-5d49-4962-8126-4c0a1304e852 None admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:42:11.059348 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9318f132-fe50-4b31-a355-39a196133866 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:42:11.060355 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9318f132-fe50-4b31-a355-39a196133866 None admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:42:11.060847 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9318f132-fe50-4b31-a355-39a196133866 None admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:42:11.061325 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9318f132-fe50-4b31-a355-39a196133866 None admin] PATH_INFO: `/v3/services` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:42:11.062294 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9318f132-fe50-4b31-a355-39a196133866 None admin] RBAC: Adding query filter params (type=object-store) {{(pid=92056) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} May 23 12:42:11.062830 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9318f132-fe50-4b31-a355-39a196133866 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:42:11.064434 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9318f132-fe50-4b31-a355-39a196133866 None admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:42:11.070898 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 476/947] 10.222.0.30 () {56 vars in 1122 bytes} [Mon May 23 12:42:10 2022] GET /identity/v3/services?type=object-store => generated 356 bytes in 188 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) May 23 12:42:11.076317 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9318f132-fe50-4b31-a355-39a196133866 None admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:42:11.119151 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=100 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:42:11.159057 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20cafd69-5d49-4962-8126-4c0a1304e852 None admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:42:11.200134 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0475aa52-3c6b-487b-b045-2a790ea7a881 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:42:11.200874 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0475aa52-3c6b-487b-b045-2a790ea7a881 None admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:42:11.201353 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0475aa52-3c6b-487b-b045-2a790ea7a881 None admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:42:11.201594 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0475aa52-3c6b-487b-b045-2a790ea7a881 None admin] PATH_INFO: `/v3/projects` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:42:11.202512 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0475aa52-3c6b-487b-b045-2a790ea7a881 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=92057) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} May 23 12:42:11.203179 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0475aa52-3c6b-487b-b045-2a790ea7a881 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:42:11.204696 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0475aa52-3c6b-487b-b045-2a790ea7a881 None admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:42:11.222921 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9318f132-fe50-4b31-a355-39a196133866 None admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:42:11.224253 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 472/948] 10.222.0.30 () {56 vars in 1148 bytes} [Mon May 23 12:42:11 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 178 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) May 23 12:42:11.225134 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0475aa52-3c6b-487b-b045-2a790ea7a881 None admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:42:11.263533 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ee545b5-9d2f-4597-8b8c-6c519ecad4ea None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:42:11.264192 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ee545b5-9d2f-4597-8b8c-6c519ecad4ea None admin] REQUEST_METHOD: `POST` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:42:11.264477 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ee545b5-9d2f-4597-8b8c-6c519ecad4ea None admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:42:11.264789 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ee545b5-9d2f-4597-8b8c-6c519ecad4ea None admin] PATH_INFO: `/v3/endpoints` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:42:11.265534 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ee545b5-9d2f-4597-8b8c-6c519ecad4ea None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:42:11.267279 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ee545b5-9d2f-4597-8b8c-6c519ecad4ea None admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:42:11.352988 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 477/949] 10.222.0.30 () {60 vars in 1144 bytes} [Mon May 23 12:42:11 2022] POST /identity/v3/endpoints => generated 349 bytes in 272 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) May 23 12:42:11.366885 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0475aa52-3c6b-487b-b045-2a790ea7a881 None admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:42:11.405095 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=100 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:42:11.422168 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-959a6ef0-96b0-409f-ab25-805d8f561030 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:42:11.423491 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-959a6ef0-96b0-409f-ab25-805d8f561030 None admin] REQUEST_METHOD: `PUT` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:42:11.423993 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-959a6ef0-96b0-409f-ab25-805d8f561030 None admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:42:11.424437 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-959a6ef0-96b0-409f-ab25-805d8f561030 None admin] PATH_INFO: `/v3/projects/9c389ae65aaa4542bb5a5c4fd0bacf5b/users/6d21a56126fd4ccd837c33ccc08ab609/roles/940396c67e6744c784e95e01fe84ad45` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:42:11.427386 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-959a6ef0-96b0-409f-ab25-805d8f561030 None admin] RBAC: Authorizing `identity:create_grant(project_id=9c389ae65aaa4542bb5a5c4fd0bacf5b, user_id=6d21a56126fd4ccd837c33ccc08ab609, role_id=940396c67e6744c784e95e01fe84ad45)` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:42:11.429267 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-959a6ef0-96b0-409f-ab25-805d8f561030 None admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:42:11.508264 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 473/950] 10.222.0.30 () {58 vars in 1439 bytes} [Mon May 23 12:42:11 2022] PUT /identity/v3/projects/9c389ae65aaa4542bb5a5c4fd0bacf5b/users/6d21a56126fd4ccd837c33ccc08ab609/roles/940396c67e6744c784e95e01fe84ad45 => generated 0 bytes in 283 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) May 23 12:42:11.639161 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7ad2462-669e-4aba-9877-ec53b5e19d98 None None] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:42:11.639495 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7ad2462-669e-4aba-9877-ec53b5e19d98 None None] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:42:11.639762 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7ad2462-669e-4aba-9877-ec53b5e19d98 None None] PATH_INFO: `/` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:42:11.641455 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 478/951] 10.222.0.30 () {54 vars in 926 bytes} [Mon May 23 12:42:11 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) May 23 12:42:11.647871 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-456bfdd3-6b1f-4a21-af2f-e10f5f1d57ce None None] REQUEST_METHOD: `POST` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:42:11.648197 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-456bfdd3-6b1f-4a21-af2f-e10f5f1d57ce None None] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:42:11.648441 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-456bfdd3-6b1f-4a21-af2f-e10f5f1d57ce None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:42:11.667889 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.auth.core [None req-456bfdd3-6b1f-4a21-af2f-e10f5f1d57ce None None] MFA Rules not processed for user `0f7119f8fa5c4445977671140ded91a4`. Rule list: `[]` (Enabled: `True`). {{(pid=92057) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 12:42:11.669145 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.fernet_utils [None req-456bfdd3-6b1f-4a21-af2f-e10f5f1d57ce None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92057) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 12:42:11.745328 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 474/952] 10.222.0.30 () {58 vars in 1023 bytes} [Mon May 23 12:42:11 2022] POST /identity/v3/auth/tokens => generated 1464 bytes in 99 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) May 23 12:42:11.769777 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7ad2462-669e-4aba-9877-ec53b5e19d98 None None] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:42:11.791995 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:42:11.870350 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-245d77d6-ca56-4c01-8871-6aace7979988 None None] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:42:11.870687 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-245d77d6-ca56-4c01-8871-6aace7979988 None None] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:42:11.870908 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-245d77d6-ca56-4c01-8871-6aace7979988 None None] PATH_INFO: `/` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:42:11.872278 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 475/953] 10.222.0.30 () {54 vars in 926 bytes} [Mon May 23 12:42:11 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) May 23 12:42:11.877277 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00a094d7-2cc4-4556-9fb3-9a0b8e9f10d2 None None] REQUEST_METHOD: `POST` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:42:11.877563 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00a094d7-2cc4-4556-9fb3-9a0b8e9f10d2 None None] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:42:11.877842 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00a094d7-2cc4-4556-9fb3-9a0b8e9f10d2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:42:11.879559 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7ad2462-669e-4aba-9877-ec53b5e19d98 None None] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:42:11.894904 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.auth.core [None req-00a094d7-2cc4-4556-9fb3-9a0b8e9f10d2 None None] MFA Rules not processed for user `0f7119f8fa5c4445977671140ded91a4`. Rule list: `[]` (Enabled: `True`). {{(pid=92057) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 12:42:11.896080 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.fernet_utils [None req-00a094d7-2cc4-4556-9fb3-9a0b8e9f10d2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92057) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 12:42:11.915329 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93a4dc1e-3507-4e25-a4ea-f366b429872f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:42:11.915985 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93a4dc1e-3507-4e25-a4ea-f366b429872f None admin] REQUEST_METHOD: `POST` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:42:11.916268 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93a4dc1e-3507-4e25-a4ea-f366b429872f None admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:42:11.916543 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93a4dc1e-3507-4e25-a4ea-f366b429872f None admin] PATH_INFO: `/v3/projects` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:42:11.917244 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93a4dc1e-3507-4e25-a4ea-f366b429872f None admin] RBAC: Authorizing `identity:create_project()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:42:11.918786 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93a4dc1e-3507-4e25-a4ea-f366b429872f None admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:42:11.920146 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: WARNING keystone.server.flask.common [None req-93a4dc1e-3507-4e25-a4ea-f366b429872f None admin] Deprecated: Not specifying a domain during a create user, group or project call, and relying on falling back to the default domain, is deprecated as of Liberty. There is no plan to remove this compatibility, however, future API versions may remove this, so please specify the domain explicitly or use a domain-scoped token. May 23 12:42:11.966569 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.sql.core [None req-93a4dc1e-3507-4e25-a4ea-f366b429872f None admin] Conflict project: (pymysql.err.IntegrityError) (1062, "Duplicate entry 'default-service' for key 'project.ixu_project_name_domain_id'") May 23 12:42:11.966569 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [SQL: INSERT INTO project (id, name, domain_id, description, enabled, extra, parent_id, is_domain) VALUES (%(id)s, %(name)s, %(domain_id)s, %(description)s, %(enabled)s, %(extra)s, %(parent_id)s, %(is_domain)s)] May 23 12:42:11.966569 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [parameters: {'id': '5a072a95923948f78dc2b654fddc7dec', 'name': 'service', 'domain_id': 'default', 'description': '', 'enabled': 1, 'extra': '{}', 'parent_id': 'default', 'is_domain': 0}] May 23 12:42:11.966569 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: (Background on this error at: https://sqlalche.me/e/14/gkpj) {{(pid=92056) wrapper /opt/stack/keystone/keystone/common/sql/core.py:533}} May 23 12:42:11.981218 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 476/954] 10.222.0.30 () {58 vars in 1023 bytes} [Mon May 23 12:42:11 2022] POST /identity/v3/auth/tokens => generated 1464 bytes in 104 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) May 23 12:42:11.983380 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application [None req-93a4dc1e-3507-4e25-a4ea-f366b429872f None admin] Conflict occurred attempting to store project - it is not permitted to have two projects with either the same name or same id in the same domain: name is service, project id 5a072a95923948f78dc2b654fddc7dec.: keystone.exception.Conflict: Conflict occurred attempting to store project - it is not permitted to have two projects with either the same name or same id in the same domain: name is service, project id 5a072a95923948f78dc2b654fddc7dec. May 23 12:42:11.983380 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application Traceback (most recent call last): May 23 12:42:11.983380 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1819, in _execute_context May 23 12:42:11.983380 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application self.dialect.do_execute( May 23 12:42:11.983380 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/default.py", line 732, in do_execute May 23 12:42:11.983380 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application cursor.execute(statement, parameters) May 23 12:42:11.983380 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 148, in execute May 23 12:42:11.983380 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application result = self._query(query) May 23 12:42:11.983380 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 310, in _query May 23 12:42:11.983380 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application conn.query(q) May 23 12:42:11.983380 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 548, in query May 23 12:42:11.983380 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application self._affected_rows = self._read_query_result(unbuffered=unbuffered) May 23 12:42:11.983380 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 775, in _read_query_result May 23 12:42:11.984257 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application result.read() May 23 12:42:11.984257 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 1156, in read May 23 12:42:11.984257 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application first_packet = self.connection._read_packet() May 23 12:42:11.984257 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 725, in _read_packet May 23 12:42:11.984257 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application packet.raise_for_error() May 23 12:42:11.984257 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/protocol.py", line 221, in raise_for_error May 23 12:42:11.984257 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application err.raise_mysql_exception(self._data) May 23 12:42:11.984257 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/err.py", line 143, in raise_mysql_exception May 23 12:42:11.984257 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application raise errorclass(errno, errval) May 23 12:42:11.984257 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application pymysql.err.IntegrityError: (1062, "Duplicate entry 'default-service' for key 'project.ixu_project_name_domain_id'") May 23 12:42:11.984257 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application May 23 12:42:11.984257 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application The above exception was the direct cause of the following exception: May 23 12:42:11.984257 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application May 23 12:42:11.984257 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application Traceback (most recent call last): May 23 12:42:11.984257 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/sql/core.py", line 528, in wrapper May 23 12:42:11.984257 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return method(*args, **kwargs) May 23 12:42:11.984257 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 241, in create_project May 23 12:42:11.984257 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return project_ref.to_dict() May 23 12:42:11.985247 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ May 23 12:42:11.985247 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application next(self.gen) May 23 12:42:11.985247 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 1060, in _transaction_scope May 23 12:42:11.985247 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application yield resource May 23 12:42:11.985247 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ May 23 12:42:11.985247 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application next(self.gen) May 23 12:42:11.985247 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 662, in _session May 23 12:42:11.985247 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application self.session.rollback() May 23 12:42:11.985247 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ May 23 12:42:11.985247 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application self.force_reraise() May 23 12:42:11.985247 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise May 23 12:42:11.985247 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application raise self.value May 23 12:42:11.985247 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 659, in _session May 23 12:42:11.985247 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application self._end_session_transaction(self.session) May 23 12:42:11.985247 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 687, in _end_session_transaction May 23 12:42:11.985247 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application session.commit() May 23 12:42:11.985247 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 1435, in commit May 23 12:42:11.986227 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application self._transaction.commit(_to_root=self.future) May 23 12:42:11.986227 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 829, in commit May 23 12:42:11.986227 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application self._prepare_impl() May 23 12:42:11.986227 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 808, in _prepare_impl May 23 12:42:11.986227 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application self.session.flush() May 23 12:42:11.986227 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3367, in flush May 23 12:42:11.986227 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application self._flush(objects) May 23 12:42:11.986227 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3507, in _flush May 23 12:42:11.986227 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application transaction.rollback(_capture_exception=True) May 23 12:42:11.986227 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/langhelpers.py", line 70, in __exit__ May 23 12:42:11.986227 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application compat.raise_( May 23 12:42:11.986227 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/compat.py", line 207, in raise_ May 23 12:42:11.986227 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application raise exception May 23 12:42:11.986227 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3467, in _flush May 23 12:42:11.986227 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application flush_context.execute() May 23 12:42:11.986227 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/unitofwork.py", line 456, in execute May 23 12:42:11.986227 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application rec.execute(self) May 23 12:42:11.987299 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/unitofwork.py", line 630, in execute May 23 12:42:11.987299 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application util.preloaded.orm_persistence.save_obj( May 23 12:42:11.987299 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/persistence.py", line 245, in save_obj May 23 12:42:11.987299 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application _emit_insert_statements( May 23 12:42:11.987299 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/persistence.py", line 1097, in _emit_insert_statements May 23 12:42:11.987299 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application c = connection._execute_20( May 23 12:42:11.987299 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1631, in _execute_20 May 23 12:42:11.987299 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return meth(self, args_10style, kwargs_10style, execution_options) May 23 12:42:11.987299 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/sql/elements.py", line 325, in _execute_on_connection May 23 12:42:11.987299 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return connection._execute_clauseelement( May 23 12:42:11.987299 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1498, in _execute_clauseelement May 23 12:42:11.987299 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application ret = self._execute_context( May 23 12:42:11.987299 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1862, in _execute_context May 23 12:42:11.987299 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application self._handle_dbapi_exception( May 23 12:42:11.987299 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 2041, in _handle_dbapi_exception May 23 12:42:11.988775 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application util.raise_(newraise, with_traceback=exc_info[2], from_=e) May 23 12:42:11.988775 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/compat.py", line 207, in raise_ May 23 12:42:11.988775 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application raise exception May 23 12:42:11.988775 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1819, in _execute_context May 23 12:42:11.988775 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application self.dialect.do_execute( May 23 12:42:11.988775 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/default.py", line 732, in do_execute May 23 12:42:11.988775 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application cursor.execute(statement, parameters) May 23 12:42:11.988775 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 148, in execute May 23 12:42:11.988775 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application result = self._query(query) May 23 12:42:11.988775 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 310, in _query May 23 12:42:11.988775 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application conn.query(q) May 23 12:42:11.988775 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 548, in query May 23 12:42:11.988775 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application self._affected_rows = self._read_query_result(unbuffered=unbuffered) May 23 12:42:11.988775 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 775, in _read_query_result May 23 12:42:11.988775 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application result.read() May 23 12:42:11.988775 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 1156, in read May 23 12:42:11.988775 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application first_packet = self.connection._read_packet() May 23 12:42:11.989825 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 725, in _read_packet May 23 12:42:11.989825 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application packet.raise_for_error() May 23 12:42:11.989825 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/protocol.py", line 221, in raise_for_error May 23 12:42:11.989825 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application err.raise_mysql_exception(self._data) May 23 12:42:11.989825 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/err.py", line 143, in raise_mysql_exception May 23 12:42:11.989825 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application raise errorclass(errno, errval) May 23 12:42:11.989825 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application oslo_db.exception.DBDuplicateEntry: (pymysql.err.IntegrityError) (1062, "Duplicate entry 'default-service' for key 'project.ixu_project_name_domain_id'") May 23 12:42:11.989825 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application [SQL: INSERT INTO project (id, name, domain_id, description, enabled, extra, parent_id, is_domain) VALUES (%(id)s, %(name)s, %(domain_id)s, %(description)s, %(enabled)s, %(extra)s, %(parent_id)s, %(is_domain)s)] May 23 12:42:11.989825 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application [parameters: {'id': '5a072a95923948f78dc2b654fddc7dec', 'name': 'service', 'domain_id': 'default', 'description': '', 'enabled': 1, 'extra': '{}', 'parent_id': 'default', 'is_domain': 0}] May 23 12:42:11.989825 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application (Background on this error at: https://sqlalche.me/e/14/gkpj) May 23 12:42:11.989825 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application May 23 12:42:11.989825 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: May 23 12:42:11.989825 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application May 23 12:42:11.989825 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application Traceback (most recent call last): May 23 12:42:11.989825 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 216, in create_project May 23 12:42:11.996269 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application ret = self.driver.create_project(project_id, project) May 23 12:42:11.996269 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/sql/core.py", line 563, in wrapper May 23 12:42:11.996269 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application raise exception.Conflict(type=conflict_type, May 23 12:42:11.996269 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application keystone.exception.Conflict: Conflict occurred attempting to store project - Duplicate entry found with name service at domain ID default. May 23 12:42:11.996269 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application May 23 12:42:11.996269 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: May 23 12:42:11.996269 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application May 23 12:42:11.996269 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application Traceback (most recent call last): May 23 12:42:11.996269 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request May 23 12:42:11.996269 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application rv = self.dispatch_request() May 23 12:42:11.996269 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request May 23 12:42:11.996269 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) May 23 12:42:11.996269 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper May 23 12:42:11.996269 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) May 23 12:42:11.996269 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view May 23 12:42:11.996269 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) May 23 12:42:11.996269 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request May 23 12:42:11.997574 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) May 23 12:42:11.997574 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 176, in post May 23 12:42:11.997574 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application ref = PROVIDERS.resource_api.create_project( May 23 12:42:11.997574 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped May 23 12:42:11.997574 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) May 23 12:42:11.997574 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 218, in create_project May 23 12:42:11.997574 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application raise exception.Conflict( May 23 12:42:11.997574 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application keystone.exception.Conflict: Conflict occurred attempting to store project - it is not permitted to have two projects with either the same name or same id in the same domain: name is service, project id 5a072a95923948f78dc2b654fddc7dec. May 23 12:42:11.997574 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application May 23 12:42:11.997574 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 479/955] 10.222.0.30 () {60 vars in 1140 bytes} [Mon May 23 12:42:11 2022] POST /identity/v3/projects => generated 262 bytes in 216 msecs (HTTP/1.1 409) 5 headers in 183 bytes (1 switches on core 0) May 23 12:42:11.998347 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00a094d7-2cc4-4556-9fb3-9a0b8e9f10d2 None None] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:42:12.004805 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93a4dc1e-3507-4e25-a4ea-f366b429872f None admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:42:12.051458 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:42:12.107498 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00a094d7-2cc4-4556-9fb3-9a0b8e9f10d2 None None] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:42:12.121832 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:42:12.152692 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93a4dc1e-3507-4e25-a4ea-f366b429872f None admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:42:12.153292 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3537222-5234-4ddc-9135-3c6e8bcade40 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:42:12.154149 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3537222-5234-4ddc-9135-3c6e8bcade40 None admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:42:12.154599 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3537222-5234-4ddc-9135-3c6e8bcade40 None admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:42:12.154885 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3537222-5234-4ddc-9135-3c6e8bcade40 None admin] PATH_INFO: `/v3/projects/service` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:42:12.164013 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3537222-5234-4ddc-9135-3c6e8bcade40 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:42:12.165997 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3537222-5234-4ddc-9135-3c6e8bcade40 None admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:42:12.184219 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application [None req-b3537222-5234-4ddc-9135-3c6e8bcade40 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. May 23 12:42:12.184219 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application Traceback (most recent call last): May 23 12:42:12.184219 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request May 23 12:42:12.184219 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application rv = self.dispatch_request() May 23 12:42:12.184219 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request May 23 12:42:12.184219 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) May 23 12:42:12.184219 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper May 23 12:42:12.184219 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) May 23 12:42:12.184219 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view May 23 12:42:12.184219 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) May 23 12:42:12.184219 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request May 23 12:42:12.184219 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) May 23 12:42:12.184219 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get May 23 12:42:12.184219 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self._get_project(project_id) May 23 12:42:12.184219 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project May 23 12:42:12.185268 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) May 23 12:42:12.185268 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped May 23 12:42:12.185268 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) May 23 12:42:12.185268 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun May 23 12:42:12.185268 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) May 23 12:42:12.185268 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func May 23 12:42:12.185268 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self.get_or_create( May 23 12:42:12.185268 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create May 23 12:42:12.185268 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application with Lock( May 23 12:42:12.185268 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ May 23 12:42:12.185268 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self._enter() May 23 12:42:12.185268 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter May 23 12:42:12.185268 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) May 23 12:42:12.185268 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create May 23 12:42:12.185268 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self.creator() May 23 12:42:12.185268 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value May 23 12:42:12.185268 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application created_value = creator( May 23 12:42:12.186382 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project May 23 12:42:12.186382 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) May 23 12:42:12.186382 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project May 23 12:42:12.186382 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() May 23 12:42:12.186382 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project May 23 12:42:12.186382 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) May 23 12:42:12.186382 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. May 23 12:42:12.186382 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application May 23 12:42:12.186832 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 477/956] 10.222.0.30 () {56 vars in 1111 bytes} [Mon May 23 12:42:11 2022] GET /identity/v3/projects/service => generated 88 bytes in 195 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) May 23 12:42:12.193421 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7093060f-a76f-4645-8ee9-451425d9d5cd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:42:12.194149 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3537222-5234-4ddc-9135-3c6e8bcade40 None admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:42:12.194257 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7093060f-a76f-4645-8ee9-451425d9d5cd None admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:42:12.194526 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7093060f-a76f-4645-8ee9-451425d9d5cd None admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:42:12.194800 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7093060f-a76f-4645-8ee9-451425d9d5cd None admin] PATH_INFO: `/v3/services/compute` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:42:12.195464 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7093060f-a76f-4645-8ee9-451425d9d5cd None admin] RBAC: Authorizing `identity:get_service(service_id=compute)` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:42:12.197051 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7093060f-a76f-4645-8ee9-451425d9d5cd None admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:42:12.216412 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application [None req-7093060f-a76f-4645-8ee9-451425d9d5cd None admin] Could not find service: compute.: keystone.exception.ServiceNotFound: Could not find service: compute. May 23 12:42:12.216412 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application Traceback (most recent call last): May 23 12:42:12.216412 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service May 23 12:42:12.216412 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) May 23 12:42:12.216412 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service May 23 12:42:12.216412 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() May 23 12:42:12.216412 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service May 23 12:42:12.216412 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) May 23 12:42:12.216412 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute. May 23 12:42:12.216412 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application May 23 12:42:12.216412 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: May 23 12:42:12.216412 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application May 23 12:42:12.216412 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application Traceback (most recent call last): May 23 12:42:12.216412 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request May 23 12:42:12.216412 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application rv = self.dispatch_request() May 23 12:42:12.216412 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request May 23 12:42:12.216412 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) May 23 12:42:12.217913 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper May 23 12:42:12.217913 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) May 23 12:42:12.217913 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view May 23 12:42:12.217913 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) May 23 12:42:12.217913 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request May 23 12:42:12.217913 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) May 23 12:42:12.217913 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get May 23 12:42:12.217913 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self._get_service(service_id) May 23 12:42:12.217913 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service May 23 12:42:12.217913 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) May 23 12:42:12.217913 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped May 23 12:42:12.217913 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) May 23 12:42:12.217913 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun May 23 12:42:12.217913 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) May 23 12:42:12.217913 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func May 23 12:42:12.217913 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self.get_or_create( May 23 12:42:12.218817 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create May 23 12:42:12.218817 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application with Lock( May 23 12:42:12.218817 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ May 23 12:42:12.218817 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self._enter() May 23 12:42:12.218817 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter May 23 12:42:12.218817 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) May 23 12:42:12.218817 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create May 23 12:42:12.218817 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self.creator() May 23 12:42:12.218817 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value May 23 12:42:12.218817 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application created_value = creator( May 23 12:42:12.218817 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service May 23 12:42:12.218817 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) May 23 12:42:12.218817 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute. May 23 12:42:12.218817 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application May 23 12:42:12.221944 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 480/957] 10.222.0.30 () {56 vars in 1111 bytes} [Mon May 23 12:42:12 2022] GET /identity/v3/services/compute => generated 88 bytes in 214 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) May 23 12:42:12.224458 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7093060f-a76f-4645-8ee9-451425d9d5cd None admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:42:12.317537 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3537222-5234-4ddc-9135-3c6e8bcade40 None admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:42:12.320353 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=100 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:42:12.353970 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0db2681-9201-4db2-b28f-09e7957609a9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:42:12.354584 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0db2681-9201-4db2-b28f-09e7957609a9 None admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:42:12.354869 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0db2681-9201-4db2-b28f-09e7957609a9 None admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:42:12.355145 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0db2681-9201-4db2-b28f-09e7957609a9 None admin] PATH_INFO: `/v3/projects` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:42:12.355890 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0db2681-9201-4db2-b28f-09e7957609a9 None admin] RBAC: Adding query filter params (name=service) {{(pid=92057) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} May 23 12:42:12.356199 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0db2681-9201-4db2-b28f-09e7957609a9 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:42:12.357758 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0db2681-9201-4db2-b28f-09e7957609a9 None admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:42:12.361953 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7093060f-a76f-4645-8ee9-451425d9d5cd None admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:42:12.382193 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 478/958] 10.222.0.30 () {56 vars in 1112 bytes} [Mon May 23 12:42:12 2022] GET /identity/v3/projects?name=service => generated 411 bytes in 184 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) May 23 12:42:12.402854 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2b8911f-2101-417e-98b6-25a3139d933c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:42:12.403976 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2b8911f-2101-417e-98b6-25a3139d933c None admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:42:12.404490 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2b8911f-2101-417e-98b6-25a3139d933c None admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:42:12.405021 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2b8911f-2101-417e-98b6-25a3139d933c None admin] PATH_INFO: `/v3/services` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:42:12.406006 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2b8911f-2101-417e-98b6-25a3139d933c None admin] RBAC: Adding query filter params (name=compute) {{(pid=92056) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} May 23 12:42:12.406524 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2b8911f-2101-417e-98b6-25a3139d933c None admin] RBAC: Authorizing `identity:list_services()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:42:12.408215 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2b8911f-2101-417e-98b6-25a3139d933c None admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:42:12.424199 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 481/959] 10.222.0.30 () {56 vars in 1112 bytes} [Mon May 23 12:42:12 2022] GET /identity/v3/services?name=compute => generated 124 bytes in 195 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) May 23 12:42:12.427092 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0db2681-9201-4db2-b28f-09e7957609a9 None admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:42:12.545912 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0db2681-9201-4db2-b28f-09e7957609a9 None admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:42:12.587122 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e78923f-556a-4905-a78e-2b9d5dbf5c43 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:42:12.587122 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e78923f-556a-4905-a78e-2b9d5dbf5c43 None admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:42:12.587122 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e78923f-556a-4905-a78e-2b9d5dbf5c43 None admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:42:12.587122 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e78923f-556a-4905-a78e-2b9d5dbf5c43 None admin] PATH_INFO: `/v3/services` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:42:12.587533 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e78923f-556a-4905-a78e-2b9d5dbf5c43 None admin] RBAC: Adding query filter params (type=compute) {{(pid=92057) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} May 23 12:42:12.587774 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e78923f-556a-4905-a78e-2b9d5dbf5c43 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:42:12.589185 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e78923f-556a-4905-a78e-2b9d5dbf5c43 None admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:42:12.602010 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 479/960] 10.222.0.30 () {56 vars in 1112 bytes} [Mon May 23 12:42:12 2022] GET /identity/v3/services?type=compute => generated 124 bytes in 168 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) May 23 12:42:13.783140 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3df07ed0-c7fc-4db9-a7e5-80857f421182 None None] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:42:13.783654 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3df07ed0-c7fc-4db9-a7e5-80857f421182 None None] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:42:13.783769 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3df07ed0-c7fc-4db9-a7e5-80857f421182 None None] PATH_INFO: `/` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:42:13.785570 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 482/961] 10.222.0.30 () {54 vars in 926 bytes} [Mon May 23 12:42:13 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) May 23 12:42:13.791201 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdd4ef63-da9a-4bd4-8bb6-e239a070cfd8 None None] REQUEST_METHOD: `POST` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:42:13.791633 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdd4ef63-da9a-4bd4-8bb6-e239a070cfd8 None None] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:42:13.791808 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdd4ef63-da9a-4bd4-8bb6-e239a070cfd8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:42:13.809395 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.auth.core [None req-fdd4ef63-da9a-4bd4-8bb6-e239a070cfd8 None None] MFA Rules not processed for user `0f7119f8fa5c4445977671140ded91a4`. Rule list: `[]` (Enabled: `True`). {{(pid=92057) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 12:42:13.810610 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.fernet_utils [None req-fdd4ef63-da9a-4bd4-8bb6-e239a070cfd8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92057) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 12:42:13.836327 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=100 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:42:13.885827 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 480/962] 10.222.0.30 () {58 vars in 1023 bytes} [Mon May 23 12:42:13 2022] POST /identity/v3/auth/tokens => generated 1464 bytes in 96 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) May 23 12:42:13.894200 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-593f26d7-c2a3-4c5a-8652-24ade3e6f4d9 None None] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:42:13.894200 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-593f26d7-c2a3-4c5a-8652-24ade3e6f4d9 None None] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:42:13.894770 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-593f26d7-c2a3-4c5a-8652-24ade3e6f4d9 None None] PATH_INFO: `/` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:42:13.896340 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 483/963] 10.222.0.30 () {54 vars in 926 bytes} [Mon May 23 12:42:13 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) May 23 12:42:13.903250 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85dea70b-bd0d-4b8c-b9b8-620ffd8ff1ac None None] REQUEST_METHOD: `POST` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:42:13.903542 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85dea70b-bd0d-4b8c-b9b8-620ffd8ff1ac None None] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:42:13.903822 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85dea70b-bd0d-4b8c-b9b8-620ffd8ff1ac None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:42:13.914416 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-593f26d7-c2a3-4c5a-8652-24ade3e6f4d9 None None] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:42:13.917342 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=100 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:42:13.922570 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.auth.core [None req-85dea70b-bd0d-4b8c-b9b8-620ffd8ff1ac None None] MFA Rules not processed for user `0f7119f8fa5c4445977671140ded91a4`. Rule list: `[]` (Enabled: `True`). {{(pid=92057) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 12:42:13.923750 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.fernet_utils [None req-85dea70b-bd0d-4b8c-b9b8-620ffd8ff1ac None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92057) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 12:42:14.014364 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 481/964] 10.222.0.30 () {58 vars in 1023 bytes} [Mon May 23 12:42:13 2022] POST /identity/v3/auth/tokens => generated 1464 bytes in 112 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) May 23 12:42:14.038966 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85dea70b-bd0d-4b8c-b9b8-620ffd8ff1ac None None] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:42:14.043059 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-593f26d7-c2a3-4c5a-8652-24ade3e6f4d9 None None] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:42:14.079701 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5943da1f-c9de-4fb7-bc0b-03f7611b9842 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:42:14.080590 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5943da1f-c9de-4fb7-bc0b-03f7611b9842 None admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:42:14.080900 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5943da1f-c9de-4fb7-bc0b-03f7611b9842 None admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:42:14.081186 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5943da1f-c9de-4fb7-bc0b-03f7611b9842 None admin] PATH_INFO: `/v3/services/object-store` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:42:14.082149 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5943da1f-c9de-4fb7-bc0b-03f7611b9842 None admin] RBAC: Authorizing `identity:get_service(service_id=object-store)` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:42:14.083500 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5943da1f-c9de-4fb7-bc0b-03f7611b9842 None admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:42:14.096267 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application [None req-5943da1f-c9de-4fb7-bc0b-03f7611b9842 None admin] Could not find service: object-store.: keystone.exception.ServiceNotFound: Could not find service: object-store. May 23 12:42:14.096267 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application Traceback (most recent call last): May 23 12:42:14.096267 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service May 23 12:42:14.096267 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) May 23 12:42:14.096267 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service May 23 12:42:14.096267 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() May 23 12:42:14.096267 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service May 23 12:42:14.096267 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) May 23 12:42:14.096267 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. May 23 12:42:14.096267 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application May 23 12:42:14.096267 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: May 23 12:42:14.096267 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application May 23 12:42:14.096267 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application Traceback (most recent call last): May 23 12:42:14.096267 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request May 23 12:42:14.096267 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application rv = self.dispatch_request() May 23 12:42:14.096267 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request May 23 12:42:14.096267 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) May 23 12:42:14.098585 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper May 23 12:42:14.098585 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) May 23 12:42:14.098585 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view May 23 12:42:14.098585 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) May 23 12:42:14.098585 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request May 23 12:42:14.098585 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) May 23 12:42:14.098585 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get May 23 12:42:14.098585 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self._get_service(service_id) May 23 12:42:14.098585 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service May 23 12:42:14.098585 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) May 23 12:42:14.098585 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped May 23 12:42:14.098585 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) May 23 12:42:14.098585 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun May 23 12:42:14.098585 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) May 23 12:42:14.098585 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func May 23 12:42:14.098585 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self.get_or_create( May 23 12:42:14.099453 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create May 23 12:42:14.099453 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application with Lock( May 23 12:42:14.099453 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ May 23 12:42:14.099453 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self._enter() May 23 12:42:14.099453 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter May 23 12:42:14.099453 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) May 23 12:42:14.099453 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create May 23 12:42:14.099453 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self.creator() May 23 12:42:14.099453 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value May 23 12:42:14.099453 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application created_value = creator( May 23 12:42:14.099453 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service May 23 12:42:14.099453 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) May 23 12:42:14.099453 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. May 23 12:42:14.099453 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application May 23 12:42:14.099453 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 484/965] 10.222.0.30 () {56 vars in 1126 bytes} [Mon May 23 12:42:13 2022] GET /identity/v3/services/object-store => generated 93 bytes in 184 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) May 23 12:42:14.104611 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5943da1f-c9de-4fb7-bc0b-03f7611b9842 None admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:42:14.141102 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=100 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:42:14.154284 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85dea70b-bd0d-4b8c-b9b8-620ffd8ff1ac None None] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:42:14.177143 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=100 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:42:14.192980 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f729e8e-55d3-43b9-b8f2-7dec6756bcc6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:42:14.194397 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f729e8e-55d3-43b9-b8f2-7dec6756bcc6 None admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:42:14.194850 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f729e8e-55d3-43b9-b8f2-7dec6756bcc6 None admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:42:14.195470 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f729e8e-55d3-43b9-b8f2-7dec6756bcc6 None admin] PATH_INFO: `/v3/roles/ResellerAdmin` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:42:14.210240 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f729e8e-55d3-43b9-b8f2-7dec6756bcc6 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} May 23 12:42:14.214867 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5943da1f-c9de-4fb7-bc0b-03f7611b9842 None admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:42:14.220763 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application [None req-6f729e8e-55d3-43b9-b8f2-7dec6756bcc6 None admin] Could not find role: ResellerAdmin.: keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. May 23 12:42:14.220763 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application Traceback (most recent call last): May 23 12:42:14.220763 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request May 23 12:42:14.220763 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application rv = self.dispatch_request() May 23 12:42:14.220763 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request May 23 12:42:14.220763 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) May 23 12:42:14.220763 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper May 23 12:42:14.220763 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) May 23 12:42:14.220763 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view May 23 12:42:14.220763 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) May 23 12:42:14.220763 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request May 23 12:42:14.220763 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) May 23 12:42:14.220763 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get May 23 12:42:14.220763 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self._get_role(role_id) May 23 12:42:14.220763 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role May 23 12:42:14.220763 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') May 23 12:42:14.221739 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call May 23 12:42:14.221739 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( May 23 12:42:14.221739 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data May 23 12:42:14.221739 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) May 23 12:42:14.221739 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped May 23 12:42:14.221739 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) May 23 12:42:14.221739 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun May 23 12:42:14.221739 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) May 23 12:42:14.221739 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func May 23 12:42:14.221739 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self.get_or_create( May 23 12:42:14.221739 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create May 23 12:42:14.221739 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application with Lock( May 23 12:42:14.221739 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ May 23 12:42:14.221739 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self._enter() May 23 12:42:14.221739 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter May 23 12:42:14.221739 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) May 23 12:42:14.223205 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create May 23 12:42:14.223205 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self.creator() May 23 12:42:14.223205 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value May 23 12:42:14.223205 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application created_value = creator( May 23 12:42:14.223205 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role May 23 12:42:14.223205 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) May 23 12:42:14.223205 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role May 23 12:42:14.223205 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() May 23 12:42:14.223205 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role May 23 12:42:14.223205 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) May 23 12:42:14.223205 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. May 23 12:42:14.223205 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application May 23 12:42:14.230910 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 482/966] 10.222.0.30 () {56 vars in 1120 bytes} [Mon May 23 12:42:14 2022] GET /identity/v3/roles/ResellerAdmin => generated 91 bytes in 192 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) May 23 12:42:14.231440 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f729e8e-55d3-43b9-b8f2-7dec6756bcc6 None admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:42:14.260572 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b4b7f8f-0001-415b-aec7-dd453263b352 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:42:14.261247 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b4b7f8f-0001-415b-aec7-dd453263b352 None admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:42:14.261535 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b4b7f8f-0001-415b-aec7-dd453263b352 None admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:42:14.261867 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b4b7f8f-0001-415b-aec7-dd453263b352 None admin] PATH_INFO: `/v3/services` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:42:14.262605 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b4b7f8f-0001-415b-aec7-dd453263b352 None admin] RBAC: Adding query filter params (name=object-store) {{(pid=92056) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} May 23 12:42:14.262964 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b4b7f8f-0001-415b-aec7-dd453263b352 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:42:14.264356 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b4b7f8f-0001-415b-aec7-dd453263b352 None admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:42:14.274111 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 485/967] 10.222.0.30 () {56 vars in 1122 bytes} [Mon May 23 12:42:14 2022] GET /identity/v3/services?name=object-store => generated 129 bytes in 166 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) May 23 12:42:14.277168 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b4b7f8f-0001-415b-aec7-dd453263b352 None admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:42:14.371626 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f729e8e-55d3-43b9-b8f2-7dec6756bcc6 None admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:42:14.388612 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b4b7f8f-0001-415b-aec7-dd453263b352 None admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:42:14.417582 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-faf56393-8f98-4108-921d-e81cdeefc2fa None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:42:14.418835 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-faf56393-8f98-4108-921d-e81cdeefc2fa None admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:42:14.419272 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-faf56393-8f98-4108-921d-e81cdeefc2fa None admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:42:14.419957 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-faf56393-8f98-4108-921d-e81cdeefc2fa None admin] PATH_INFO: `/v3/roles` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:42:14.420882 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-faf56393-8f98-4108-921d-e81cdeefc2fa None admin] RBAC: Adding query filter params (name=ResellerAdmin) {{(pid=92057) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} May 23 12:42:14.421155 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=100 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:42:14.421494 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-faf56393-8f98-4108-921d-e81cdeefc2fa None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:42:14.423536 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-faf56393-8f98-4108-921d-e81cdeefc2fa None admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:42:14.428319 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a719eb7f-84d6-4c97-a95b-6b95b48bbffa None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:42:14.429109 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a719eb7f-84d6-4c97-a95b-6b95b48bbffa None admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:42:14.429619 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a719eb7f-84d6-4c97-a95b-6b95b48bbffa None admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:42:14.430026 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a719eb7f-84d6-4c97-a95b-6b95b48bbffa None admin] PATH_INFO: `/v3/services` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:42:14.430819 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a719eb7f-84d6-4c97-a95b-6b95b48bbffa None admin] RBAC: Adding query filter params (type=object-store) {{(pid=92056) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} May 23 12:42:14.431167 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a719eb7f-84d6-4c97-a95b-6b95b48bbffa None admin] RBAC: Authorizing `identity:list_services()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:42:14.437720 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a719eb7f-84d6-4c97-a95b-6b95b48bbffa None admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:42:14.439277 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 483/968] 10.222.0.30 () {56 vars in 1115 bytes} [Mon May 23 12:42:14 2022] GET /identity/v3/roles?name=ResellerAdmin => generated 338 bytes in 203 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) May 23 12:42:14.440219 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-faf56393-8f98-4108-921d-e81cdeefc2fa None admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:42:14.448301 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 486/969] 10.222.0.30 () {56 vars in 1122 bytes} [Mon May 23 12:42:14 2022] GET /identity/v3/services?type=object-store => generated 356 bytes in 168 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) May 23 12:42:14.449525 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a719eb7f-84d6-4c97-a95b-6b95b48bbffa None admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:42:14.455763 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=100 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:42:14.565315 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a719eb7f-84d6-4c97-a95b-6b95b48bbffa None admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:42:14.575424 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-faf56393-8f98-4108-921d-e81cdeefc2fa None admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:42:14.605754 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2eb683df-4b1c-45c3-bc14-dfa8966fe426 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:42:14.606883 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2eb683df-4b1c-45c3-bc14-dfa8966fe426 None admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:42:14.607401 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2eb683df-4b1c-45c3-bc14-dfa8966fe426 None admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:42:14.607900 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2eb683df-4b1c-45c3-bc14-dfa8966fe426 None admin] PATH_INFO: `/v3/endpoints` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:42:14.609056 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2eb683df-4b1c-45c3-bc14-dfa8966fe426 None admin] RBAC: Adding query filter params (interface=admin, service_id=24b330c81aa74757ba04d66aa30e333c, region_id=RegionOne) {{(pid=92056) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} May 23 12:42:14.609872 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2eb683df-4b1c-45c3-bc14-dfa8966fe426 None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:42:14.611570 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2eb683df-4b1c-45c3-bc14-dfa8966fe426 None admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:42:14.624071 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 487/970] 10.222.0.30 () {56 vars in 1249 bytes} [Mon May 23 12:42:14 2022] GET /identity/v3/endpoints?service_id=24b330c81aa74757ba04d66aa30e333c&interface=admin®ion_id=RegionOne => generated 193 bytes in 170 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) May 23 12:42:14.624860 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2eb683df-4b1c-45c3-bc14-dfa8966fe426 None admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:42:14.630285 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-edffd361-cb91-4dd4-bee4-519fbab8be97 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:42:14.632340 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edffd361-cb91-4dd4-bee4-519fbab8be97 None admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:42:14.632633 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edffd361-cb91-4dd4-bee4-519fbab8be97 None admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:42:14.632913 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edffd361-cb91-4dd4-bee4-519fbab8be97 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:42:14.649749 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-edffd361-cb91-4dd4-bee4-519fbab8be97 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:42:14.652419 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-edffd361-cb91-4dd4-bee4-519fbab8be97 None admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:42:14.678158 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application [None req-edffd361-cb91-4dd4-bee4-519fbab8be97 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. May 23 12:42:14.678158 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application Traceback (most recent call last): May 23 12:42:14.678158 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request May 23 12:42:14.678158 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application rv = self.dispatch_request() May 23 12:42:14.678158 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request May 23 12:42:14.678158 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) May 23 12:42:14.678158 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper May 23 12:42:14.678158 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) May 23 12:42:14.678158 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view May 23 12:42:14.678158 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) May 23 12:42:14.678158 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request May 23 12:42:14.678158 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) May 23 12:42:14.678158 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get May 23 12:42:14.678158 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self._get_domain(domain_id) May 23 12:42:14.678158 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain May 23 12:42:14.679179 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) May 23 12:42:14.679179 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped May 23 12:42:14.679179 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) May 23 12:42:14.679179 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun May 23 12:42:14.679179 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) May 23 12:42:14.679179 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func May 23 12:42:14.679179 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self.get_or_create( May 23 12:42:14.679179 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create May 23 12:42:14.679179 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application with Lock( May 23 12:42:14.679179 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ May 23 12:42:14.679179 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self._enter() May 23 12:42:14.679179 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter May 23 12:42:14.679179 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) May 23 12:42:14.679179 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create May 23 12:42:14.679179 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self.creator() May 23 12:42:14.679179 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value May 23 12:42:14.679179 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application created_value = creator( May 23 12:42:14.680118 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain May 23 12:42:14.680118 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) May 23 12:42:14.680118 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. May 23 12:42:14.680118 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application May 23 12:42:14.680118 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 484/971] 10.222.0.30 () {56 vars in 1108 bytes} [Mon May 23 12:42:14 2022] GET /identity/v3/domains/Default => generated 87 bytes in 240 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) May 23 12:42:14.686657 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-edffd361-cb91-4dd4-bee4-519fbab8be97 None admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:42:14.716230 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=100 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:42:14.763801 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2eb683df-4b1c-45c3-bc14-dfa8966fe426 None admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:42:14.773085 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=100 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:42:14.803476 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27734036-6bdd-472f-90e6-31c225e41d76 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:42:14.804829 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27734036-6bdd-472f-90e6-31c225e41d76 None admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:42:14.805359 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27734036-6bdd-472f-90e6-31c225e41d76 None admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:42:14.806088 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27734036-6bdd-472f-90e6-31c225e41d76 None admin] PATH_INFO: `/v3/services` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:42:14.807027 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-27734036-6bdd-472f-90e6-31c225e41d76 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:42:14.809029 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-27734036-6bdd-472f-90e6-31c225e41d76 None admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:42:14.815747 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 488/972] 10.222.0.30 () {56 vars in 1087 bytes} [Mon May 23 12:42:14 2022] GET /identity/v3/services => generated 1497 bytes in 191 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) May 23 12:42:14.817617 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08dcc9e7-fa06-40f2-9feb-bdbfeb4f0dd5 None None] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:42:14.817912 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08dcc9e7-fa06-40f2-9feb-bdbfeb4f0dd5 None None] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:42:14.818194 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08dcc9e7-fa06-40f2-9feb-bdbfeb4f0dd5 None None] PATH_INFO: `/` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:42:14.819630 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 489/973] 10.222.0.30 () {54 vars in 926 bytes} [Mon May 23 12:42:14 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) May 23 12:42:14.825118 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b16398e3-6508-4b51-83c8-ee5b7279bee2 None None] REQUEST_METHOD: `POST` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:42:14.825407 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b16398e3-6508-4b51-83c8-ee5b7279bee2 None None] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:42:14.825681 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b16398e3-6508-4b51-83c8-ee5b7279bee2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:42:14.830233 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-edffd361-cb91-4dd4-bee4-519fbab8be97 None admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:42:14.844952 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.auth.core [None req-b16398e3-6508-4b51-83c8-ee5b7279bee2 None None] MFA Rules not processed for user `0f7119f8fa5c4445977671140ded91a4`. Rule list: `[]` (Enabled: `True`). {{(pid=92056) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 12:42:14.846538 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.fernet_utils [None req-b16398e3-6508-4b51-83c8-ee5b7279bee2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92056) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 12:42:14.874566 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ea71a4d-4262-4787-b128-869c236e2d26 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:42:14.875869 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ea71a4d-4262-4787-b128-869c236e2d26 None admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:42:14.876341 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ea71a4d-4262-4787-b128-869c236e2d26 None admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:42:14.876964 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ea71a4d-4262-4787-b128-869c236e2d26 None admin] PATH_INFO: `/v3/domains` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:42:14.880373 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ea71a4d-4262-4787-b128-869c236e2d26 None admin] RBAC: Adding query filter params (name=Default) {{(pid=92057) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} May 23 12:42:14.881110 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ea71a4d-4262-4787-b128-869c236e2d26 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:42:14.883592 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ea71a4d-4262-4787-b128-869c236e2d26 None admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:42:14.903061 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 485/974] 10.222.0.30 () {56 vars in 1109 bytes} [Mon May 23 12:42:14 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 217 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) May 23 12:42:14.905304 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a69c995-96df-4c3a-b049-79de31c1bdf4 None None] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:42:14.905756 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a69c995-96df-4c3a-b049-79de31c1bdf4 None None] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:42:14.906634 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a69c995-96df-4c3a-b049-79de31c1bdf4 None None] PATH_INFO: `/` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:42:14.908485 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 486/975] 10.222.0.30 () {54 vars in 926 bytes} [Mon May 23 12:42:14 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) May 23 12:42:14.909485 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a69c995-96df-4c3a-b049-79de31c1bdf4 None None] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:42:14.926325 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 490/976] 10.222.0.30 () {58 vars in 1023 bytes} [Mon May 23 12:42:14 2022] POST /identity/v3/auth/tokens => generated 1464 bytes in 102 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) May 23 12:42:14.928831 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2dfe77ae-f73a-4d52-a162-2ff6f288dd1d None None] REQUEST_METHOD: `POST` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:42:14.929120 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2dfe77ae-f73a-4d52-a162-2ff6f288dd1d None None] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:42:14.929396 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2dfe77ae-f73a-4d52-a162-2ff6f288dd1d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:42:14.948682 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.auth.core [None req-2dfe77ae-f73a-4d52-a162-2ff6f288dd1d None None] MFA Rules not processed for user `0f7119f8fa5c4445977671140ded91a4`. Rule list: `[]` (Enabled: `True`). {{(pid=92056) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 12:42:14.949905 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.fernet_utils [None req-2dfe77ae-f73a-4d52-a162-2ff6f288dd1d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92056) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 12:42:15.003869 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=100 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:42:15.027685 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 491/977] 10.222.0.30 () {58 vars in 1023 bytes} [Mon May 23 12:42:14 2022] POST /identity/v3/auth/tokens => generated 1464 bytes in 101 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) May 23 12:42:15.028723 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2dfe77ae-f73a-4d52-a162-2ff6f288dd1d None None] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:42:15.037422 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a69c995-96df-4c3a-b049-79de31c1bdf4 None None] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:42:15.065119 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=100 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:42:15.079663 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d384aed-254d-4cd3-abf0-93ffee03dc89 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:42:15.080402 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d384aed-254d-4cd3-abf0-93ffee03dc89 None admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:42:15.080696 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d384aed-254d-4cd3-abf0-93ffee03dc89 None admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:42:15.080956 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d384aed-254d-4cd3-abf0-93ffee03dc89 None admin] PATH_INFO: `/v3/users/glance-swift` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:42:15.089365 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d384aed-254d-4cd3-abf0-93ffee03dc89 None admin] RBAC: Authorizing `identity:get_user(user_id=glance-swift)` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:42:15.090974 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d384aed-254d-4cd3-abf0-93ffee03dc89 None admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:42:15.108718 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application [None req-8d384aed-254d-4cd3-abf0-93ffee03dc89 None admin] Could not find user: glance-swift.: keystone.exception.UserNotFound: Could not find user: glance-swift. May 23 12:42:15.108718 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application Traceback (most recent call last): May 23 12:42:15.108718 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request May 23 12:42:15.108718 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application rv = self.dispatch_request() May 23 12:42:15.108718 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request May 23 12:42:15.108718 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) May 23 12:42:15.108718 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper May 23 12:42:15.108718 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) May 23 12:42:15.108718 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view May 23 12:42:15.108718 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) May 23 12:42:15.108718 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request May 23 12:42:15.108718 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) May 23 12:42:15.108718 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get May 23 12:42:15.108718 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self._get_user(user_id) May 23 12:42:15.108718 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user May 23 12:42:15.108718 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) May 23 12:42:15.109738 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped May 23 12:42:15.109738 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) May 23 12:42:15.109738 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper May 23 12:42:15.109738 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) May 23 12:42:15.109738 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper May 23 12:42:15.109738 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) May 23 12:42:15.109738 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun May 23 12:42:15.109738 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) May 23 12:42:15.109738 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func May 23 12:42:15.109738 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self.get_or_create( May 23 12:42:15.109738 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create May 23 12:42:15.109738 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application with Lock( May 23 12:42:15.109738 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ May 23 12:42:15.109738 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self._enter() May 23 12:42:15.109738 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter May 23 12:42:15.109738 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) May 23 12:42:15.109738 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create May 23 12:42:15.109738 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self.creator() May 23 12:42:15.110759 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value May 23 12:42:15.110759 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application created_value = creator( May 23 12:42:15.110759 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user May 23 12:42:15.110759 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) May 23 12:42:15.110759 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user May 23 12:42:15.110759 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) May 23 12:42:15.110759 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user May 23 12:42:15.110759 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) May 23 12:42:15.110759 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: glance-swift. May 23 12:42:15.110759 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application May 23 12:42:15.110759 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 487/978] 10.222.0.30 () {56 vars in 1117 bytes} [Mon May 23 12:42:14 2022] GET /identity/v3/users/glance-swift => generated 90 bytes in 201 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) May 23 12:42:15.112039 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d384aed-254d-4cd3-abf0-93ffee03dc89 None admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:42:15.144969 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2dfe77ae-f73a-4d52-a162-2ff6f288dd1d None None] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:42:15.181287 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8bce98c8-d0d3-41a2-bf9b-f640e781e2e1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:42:15.181989 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8bce98c8-d0d3-41a2-bf9b-f640e781e2e1 None admin] REQUEST_METHOD: `POST` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:42:15.182278 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8bce98c8-d0d3-41a2-bf9b-f640e781e2e1 None admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:42:15.182556 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8bce98c8-d0d3-41a2-bf9b-f640e781e2e1 None admin] PATH_INFO: `/v3/users` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:42:15.183261 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8bce98c8-d0d3-41a2-bf9b-f640e781e2e1 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:42:15.184944 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8bce98c8-d0d3-41a2-bf9b-f640e781e2e1 None admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:42:15.223576 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d384aed-254d-4cd3-abf0-93ffee03dc89 None admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:42:15.226845 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.sql.core [None req-8bce98c8-d0d3-41a2-bf9b-f640e781e2e1 None admin] Conflict user: (pymysql.err.IntegrityError) (1062, "Duplicate entry 'default-cinder' for key 'local_user.domain_id'") May 23 12:42:15.226845 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [SQL: INSERT INTO local_user (user_id, domain_id, name, failed_auth_count, failed_auth_at) VALUES (%(user_id)s, %(domain_id)s, %(name)s, %(failed_auth_count)s, %(failed_auth_at)s)] May 23 12:42:15.226845 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [parameters: {'user_id': '1ddac02f9b344d118e680249fea0754b', 'domain_id': 'default', 'name': 'cinder', 'failed_auth_count': 0, 'failed_auth_at': None}] May 23 12:42:15.226845 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: (Background on this error at: https://sqlalche.me/e/14/gkpj) {{(pid=92056) wrapper /opt/stack/keystone/keystone/common/sql/core.py:533}} May 23 12:42:15.235819 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application [None req-8bce98c8-d0d3-41a2-bf9b-f640e781e2e1 None admin] Conflict occurred attempting to store user - Duplicate entry found with name cinder at domain ID default.: keystone.exception.Conflict: Conflict occurred attempting to store user - Duplicate entry found with name cinder at domain ID default. May 23 12:42:15.235819 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application Traceback (most recent call last): May 23 12:42:15.235819 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1819, in _execute_context May 23 12:42:15.235819 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application self.dialect.do_execute( May 23 12:42:15.235819 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/default.py", line 732, in do_execute May 23 12:42:15.235819 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application cursor.execute(statement, parameters) May 23 12:42:15.235819 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 148, in execute May 23 12:42:15.235819 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application result = self._query(query) May 23 12:42:15.235819 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 310, in _query May 23 12:42:15.235819 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application conn.query(q) May 23 12:42:15.235819 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 548, in query May 23 12:42:15.235819 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application self._affected_rows = self._read_query_result(unbuffered=unbuffered) May 23 12:42:15.235819 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 775, in _read_query_result May 23 12:42:15.235819 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application result.read() May 23 12:42:15.235819 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 1156, in read May 23 12:42:15.236754 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application first_packet = self.connection._read_packet() May 23 12:42:15.236754 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 725, in _read_packet May 23 12:42:15.236754 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application packet.raise_for_error() May 23 12:42:15.236754 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/protocol.py", line 221, in raise_for_error May 23 12:42:15.236754 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application err.raise_mysql_exception(self._data) May 23 12:42:15.236754 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/err.py", line 143, in raise_mysql_exception May 23 12:42:15.236754 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application raise errorclass(errno, errval) May 23 12:42:15.236754 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application pymysql.err.IntegrityError: (1062, "Duplicate entry 'default-cinder' for key 'local_user.domain_id'") May 23 12:42:15.236754 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application May 23 12:42:15.236754 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application The above exception was the direct cause of the following exception: May 23 12:42:15.236754 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application May 23 12:42:15.236754 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application Traceback (most recent call last): May 23 12:42:15.236754 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/sql/core.py", line 528, in wrapper May 23 12:42:15.236754 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return method(*args, **kwargs) May 23 12:42:15.236754 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 141, in create_user May 23 12:42:15.236754 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return base.filter_user(user_ref.to_dict()) May 23 12:42:15.236754 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ May 23 12:42:15.236754 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application next(self.gen) May 23 12:42:15.238555 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 1060, in _transaction_scope May 23 12:42:15.238555 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application yield resource May 23 12:42:15.238555 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ May 23 12:42:15.238555 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application next(self.gen) May 23 12:42:15.238555 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 662, in _session May 23 12:42:15.238555 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application self.session.rollback() May 23 12:42:15.238555 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ May 23 12:42:15.238555 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application self.force_reraise() May 23 12:42:15.238555 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise May 23 12:42:15.238555 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application raise self.value May 23 12:42:15.238555 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 659, in _session May 23 12:42:15.238555 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application self._end_session_transaction(self.session) May 23 12:42:15.238555 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 687, in _end_session_transaction May 23 12:42:15.238555 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application session.commit() May 23 12:42:15.238555 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 1435, in commit May 23 12:42:15.238555 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application self._transaction.commit(_to_root=self.future) May 23 12:42:15.238555 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 829, in commit May 23 12:42:15.239527 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application self._prepare_impl() May 23 12:42:15.239527 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 808, in _prepare_impl May 23 12:42:15.239527 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application self.session.flush() May 23 12:42:15.239527 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3367, in flush May 23 12:42:15.239527 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application self._flush(objects) May 23 12:42:15.239527 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3507, in _flush May 23 12:42:15.239527 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application transaction.rollback(_capture_exception=True) May 23 12:42:15.239527 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/langhelpers.py", line 70, in __exit__ May 23 12:42:15.239527 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application compat.raise_( May 23 12:42:15.239527 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/compat.py", line 207, in raise_ May 23 12:42:15.239527 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application raise exception May 23 12:42:15.239527 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3467, in _flush May 23 12:42:15.239527 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application flush_context.execute() May 23 12:42:15.239527 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/unitofwork.py", line 456, in execute May 23 12:42:15.239527 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application rec.execute(self) May 23 12:42:15.239527 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/unitofwork.py", line 630, in execute May 23 12:42:15.239527 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application util.preloaded.orm_persistence.save_obj( May 23 12:42:15.240483 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/persistence.py", line 245, in save_obj May 23 12:42:15.240483 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application _emit_insert_statements( May 23 12:42:15.240483 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/persistence.py", line 1238, in _emit_insert_statements May 23 12:42:15.240483 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application result = connection._execute_20( May 23 12:42:15.240483 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1631, in _execute_20 May 23 12:42:15.240483 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return meth(self, args_10style, kwargs_10style, execution_options) May 23 12:42:15.240483 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/sql/elements.py", line 325, in _execute_on_connection May 23 12:42:15.240483 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return connection._execute_clauseelement( May 23 12:42:15.240483 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1498, in _execute_clauseelement May 23 12:42:15.240483 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application ret = self._execute_context( May 23 12:42:15.240483 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1862, in _execute_context May 23 12:42:15.240483 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application self._handle_dbapi_exception( May 23 12:42:15.240483 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 2041, in _handle_dbapi_exception May 23 12:42:15.240483 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application util.raise_(newraise, with_traceback=exc_info[2], from_=e) May 23 12:42:15.240483 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/compat.py", line 207, in raise_ May 23 12:42:15.240483 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application raise exception May 23 12:42:15.241353 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1819, in _execute_context May 23 12:42:15.241353 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application self.dialect.do_execute( May 23 12:42:15.241353 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/default.py", line 732, in do_execute May 23 12:42:15.241353 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application cursor.execute(statement, parameters) May 23 12:42:15.241353 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 148, in execute May 23 12:42:15.241353 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application result = self._query(query) May 23 12:42:15.241353 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 310, in _query May 23 12:42:15.241353 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application conn.query(q) May 23 12:42:15.241353 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 548, in query May 23 12:42:15.241353 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application self._affected_rows = self._read_query_result(unbuffered=unbuffered) May 23 12:42:15.241353 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 775, in _read_query_result May 23 12:42:15.241353 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application result.read() May 23 12:42:15.241353 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 1156, in read May 23 12:42:15.241353 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application first_packet = self.connection._read_packet() May 23 12:42:15.241353 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 725, in _read_packet May 23 12:42:15.241353 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application packet.raise_for_error() May 23 12:42:15.249489 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/protocol.py", line 221, in raise_for_error May 23 12:42:15.249489 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application err.raise_mysql_exception(self._data) May 23 12:42:15.249489 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/err.py", line 143, in raise_mysql_exception May 23 12:42:15.249489 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application raise errorclass(errno, errval) May 23 12:42:15.249489 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application oslo_db.exception.DBDuplicateEntry: (pymysql.err.IntegrityError) (1062, "Duplicate entry 'default-cinder' for key 'local_user.domain_id'") May 23 12:42:15.249489 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application [SQL: INSERT INTO local_user (user_id, domain_id, name, failed_auth_count, failed_auth_at) VALUES (%(user_id)s, %(domain_id)s, %(name)s, %(failed_auth_count)s, %(failed_auth_at)s)] May 23 12:42:15.249489 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application [parameters: {'user_id': '1ddac02f9b344d118e680249fea0754b', 'domain_id': 'default', 'name': 'cinder', 'failed_auth_count': 0, 'failed_auth_at': None}] May 23 12:42:15.249489 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application (Background on this error at: https://sqlalche.me/e/14/gkpj) May 23 12:42:15.249489 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application May 23 12:42:15.249489 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: May 23 12:42:15.249489 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application May 23 12:42:15.249489 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application Traceback (most recent call last): May 23 12:42:15.249489 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request May 23 12:42:15.249489 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application rv = self.dispatch_request() May 23 12:42:15.249489 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request May 23 12:42:15.251433 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) May 23 12:42:15.251433 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper May 23 12:42:15.251433 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) May 23 12:42:15.251433 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view May 23 12:42:15.251433 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) May 23 12:42:15.251433 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request May 23 12:42:15.251433 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) May 23 12:42:15.251433 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 244, in post May 23 12:42:15.251433 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.create_user( May 23 12:42:15.251433 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped May 23 12:42:15.251433 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) May 23 12:42:15.251433 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper May 23 12:42:15.251433 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) May 23 12:42:15.251433 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper May 23 12:42:15.251433 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) May 23 12:42:15.251433 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1000, in create_user May 23 12:42:15.251433 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application ref = self._create_user_with_federated_objects(user, driver) May 23 12:42:15.254211 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 966, in _create_user_with_federated_objects May 23 12:42:15.254211 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application user = driver.create_user(user['id'], user) May 23 12:42:15.254211 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/sql/core.py", line 563, in wrapper May 23 12:42:15.254211 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application raise exception.Conflict(type=conflict_type, May 23 12:42:15.254211 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application keystone.exception.Conflict: Conflict occurred attempting to store user - Duplicate entry found with name cinder at domain ID default. May 23 12:42:15.254211 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application May 23 12:42:15.254211 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 492/979] 10.222.0.30 () {60 vars in 1131 bytes} [Mon May 23 12:42:15 2022] POST /identity/v3/users => generated 160 bytes in 210 msecs (HTTP/1.1 409) 5 headers in 183 bytes (1 switches on core 0) May 23 12:42:15.254211 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8bce98c8-d0d3-41a2-bf9b-f640e781e2e1 None admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:42:15.266031 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7675d865-6867-4878-8cda-af7ec61c236b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:42:15.267005 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7675d865-6867-4878-8cda-af7ec61c236b None admin] REQUEST_METHOD: `POST` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:42:15.267124 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7675d865-6867-4878-8cda-af7ec61c236b None admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:42:15.267452 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7675d865-6867-4878-8cda-af7ec61c236b None admin] PATH_INFO: `/v3/services` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:42:15.268248 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7675d865-6867-4878-8cda-af7ec61c236b None admin] RBAC: Authorizing `identity:create_service()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:42:15.270619 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7675d865-6867-4878-8cda-af7ec61c236b None admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:42:15.295543 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:42:15.350551 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8bce98c8-d0d3-41a2-bf9b-f640e781e2e1 None admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:42:15.352954 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 488/980] 10.222.0.30 () {60 vars in 1141 bytes} [Mon May 23 12:42:15 2022] POST /identity/v3/services => generated 242 bytes in 242 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) May 23 12:42:15.354021 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7675d865-6867-4878-8cda-af7ec61c236b None admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:42:15.414198 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:42:15.429872 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b3f3b8f-35f7-49cb-8ad1-ed414d4910d7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:42:15.430757 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b3f3b8f-35f7-49cb-8ad1-ed414d4910d7 None admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:42:15.431041 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b3f3b8f-35f7-49cb-8ad1-ed414d4910d7 None admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:42:15.432511 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b3f3b8f-35f7-49cb-8ad1-ed414d4910d7 None admin] PATH_INFO: `/v3/users` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:42:15.433863 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b3f3b8f-35f7-49cb-8ad1-ed414d4910d7 None admin] RBAC: Adding query filter params (domain_id=default, name=glance-swift) {{(pid=92056) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} May 23 12:42:15.434240 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b3f3b8f-35f7-49cb-8ad1-ed414d4910d7 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:42:15.436521 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b3f3b8f-35f7-49cb-8ad1-ed414d4910d7 None admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:42:15.466058 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 493/981] 10.222.0.30 () {56 vars in 1149 bytes} [Mon May 23 12:42:15 2022] GET /identity/v3/users?domain_id=default&name=glance-swift => generated 384 bytes in 227 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) May 23 12:42:15.475626 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b3f3b8f-35f7-49cb-8ad1-ed414d4910d7 None admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:42:15.493586 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7675d865-6867-4878-8cda-af7ec61c236b None admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:42:15.535996 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb035805-b95c-466f-9c14-49cae524591b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:42:15.536806 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb035805-b95c-466f-9c14-49cae524591b None admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:42:15.537088 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb035805-b95c-466f-9c14-49cae524591b None admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:42:15.537365 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb035805-b95c-466f-9c14-49cae524591b None admin] PATH_INFO: `/v3/users/cinder` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:42:15.546056 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb035805-b95c-466f-9c14-49cae524591b None admin] RBAC: Authorizing `identity:get_user(user_id=cinder)` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:42:15.547532 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb035805-b95c-466f-9c14-49cae524591b None admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:42:15.562990 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application [None req-bb035805-b95c-466f-9c14-49cae524591b None admin] Could not find user: cinder.: keystone.exception.UserNotFound: Could not find user: cinder. May 23 12:42:15.562990 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application Traceback (most recent call last): May 23 12:42:15.562990 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request May 23 12:42:15.562990 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application rv = self.dispatch_request() May 23 12:42:15.562990 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request May 23 12:42:15.562990 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) May 23 12:42:15.562990 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper May 23 12:42:15.562990 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) May 23 12:42:15.562990 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view May 23 12:42:15.562990 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) May 23 12:42:15.562990 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request May 23 12:42:15.562990 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) May 23 12:42:15.562990 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get May 23 12:42:15.562990 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self._get_user(user_id) May 23 12:42:15.562990 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user May 23 12:42:15.562990 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) May 23 12:42:15.564106 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped May 23 12:42:15.564106 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) May 23 12:42:15.564106 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper May 23 12:42:15.564106 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) May 23 12:42:15.564106 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper May 23 12:42:15.564106 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) May 23 12:42:15.564106 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun May 23 12:42:15.564106 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) May 23 12:42:15.564106 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func May 23 12:42:15.564106 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self.get_or_create( May 23 12:42:15.564106 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create May 23 12:42:15.564106 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application with Lock( May 23 12:42:15.564106 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ May 23 12:42:15.564106 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self._enter() May 23 12:42:15.564106 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter May 23 12:42:15.564106 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) May 23 12:42:15.564106 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create May 23 12:42:15.564106 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self.creator() May 23 12:42:15.565130 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value May 23 12:42:15.565130 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application created_value = creator( May 23 12:42:15.565130 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user May 23 12:42:15.565130 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) May 23 12:42:15.565130 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user May 23 12:42:15.565130 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) May 23 12:42:15.565130 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user May 23 12:42:15.565130 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) May 23 12:42:15.565130 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: cinder. May 23 12:42:15.565130 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application May 23 12:42:15.565130 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 489/982] 10.222.0.30 () {56 vars in 1099 bytes} [Mon May 23 12:42:15 2022] GET /identity/v3/users/cinder => generated 84 bytes in 211 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) May 23 12:42:15.570364 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb035805-b95c-466f-9c14-49cae524591b None admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:42:15.603641 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=100 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:42:15.608959 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b3f3b8f-35f7-49cb-8ad1-ed414d4910d7 None admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:42:15.655518 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a395b869-4da5-4c5f-aa81-e9ebdda3b11d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:42:15.656331 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a395b869-4da5-4c5f-aa81-e9ebdda3b11d None admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:42:15.656617 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a395b869-4da5-4c5f-aa81-e9ebdda3b11d None admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:42:15.656888 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a395b869-4da5-4c5f-aa81-e9ebdda3b11d None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:42:15.716883 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb035805-b95c-466f-9c14-49cae524591b None admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:42:15.735838 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=100 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:42:15.741474 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a395b869-4da5-4c5f-aa81-e9ebdda3b11d None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:42:15.743032 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a395b869-4da5-4c5f-aa81-e9ebdda3b11d None admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:42:15.755671 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a928f564-09f8-4c4f-9579-f79e16546318 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:42:15.756665 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a928f564-09f8-4c4f-9579-f79e16546318 None admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:42:15.757279 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a928f564-09f8-4c4f-9579-f79e16546318 None admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:42:15.757773 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a928f564-09f8-4c4f-9579-f79e16546318 None admin] PATH_INFO: `/v3/users` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:42:15.758950 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a928f564-09f8-4c4f-9579-f79e16546318 None admin] RBAC: Adding query filter params (name=cinder) {{(pid=92057) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} May 23 12:42:15.759474 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a928f564-09f8-4c4f-9579-f79e16546318 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:42:15.761363 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a928f564-09f8-4c4f-9579-f79e16546318 None admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:42:15.799765 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 490/983] 10.222.0.30 () {56 vars in 1101 bytes} [Mon May 23 12:42:15 2022] GET /identity/v3/users?name=cinder => generated 354 bytes in 230 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) May 23 12:42:15.852524 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=100 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:42:15.858646 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 494/984] 10.222.0.30 () {58 vars in 1282 bytes} [Mon May 23 12:42:15 2022] GET /identity/v3/auth/tokens => generated 1560 bytes in 387 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) May 23 12:42:15.867068 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a928f564-09f8-4c4f-9579-f79e16546318 None admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:42:15.986016 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a928f564-09f8-4c4f-9579-f79e16546318 None admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:42:16.025985 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d7d1ecc-c314-4050-9b81-052699cd3f60 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:42:16.030082 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d7d1ecc-c314-4050-9b81-052699cd3f60 None admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:42:16.030082 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d7d1ecc-c314-4050-9b81-052699cd3f60 None admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:42:16.030082 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d7d1ecc-c314-4050-9b81-052699cd3f60 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:42:16.038513 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d7d1ecc-c314-4050-9b81-052699cd3f60 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:42:16.040006 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d7d1ecc-c314-4050-9b81-052699cd3f60 None admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:42:16.059513 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application [None req-8d7d1ecc-c314-4050-9b81-052699cd3f60 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. May 23 12:42:16.059513 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application Traceback (most recent call last): May 23 12:42:16.059513 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request May 23 12:42:16.059513 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application rv = self.dispatch_request() May 23 12:42:16.059513 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request May 23 12:42:16.059513 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) May 23 12:42:16.059513 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper May 23 12:42:16.059513 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) May 23 12:42:16.059513 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view May 23 12:42:16.059513 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) May 23 12:42:16.059513 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request May 23 12:42:16.059513 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) May 23 12:42:16.059513 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get May 23 12:42:16.059513 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self._get_domain(domain_id) May 23 12:42:16.059513 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain May 23 12:42:16.060491 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) May 23 12:42:16.060491 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped May 23 12:42:16.060491 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) May 23 12:42:16.060491 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun May 23 12:42:16.060491 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) May 23 12:42:16.060491 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func May 23 12:42:16.060491 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self.get_or_create( May 23 12:42:16.060491 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create May 23 12:42:16.060491 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application with Lock( May 23 12:42:16.060491 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ May 23 12:42:16.060491 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self._enter() May 23 12:42:16.060491 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter May 23 12:42:16.060491 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) May 23 12:42:16.060491 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create May 23 12:42:16.060491 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self.creator() May 23 12:42:16.060491 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value May 23 12:42:16.060491 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application created_value = creator( May 23 12:42:16.061903 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain May 23 12:42:16.061903 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) May 23 12:42:16.061903 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. May 23 12:42:16.061903 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application May 23 12:42:16.061903 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 491/985] 10.222.0.30 () {56 vars in 1108 bytes} [Mon May 23 12:42:15 2022] GET /identity/v3/domains/Default => generated 87 bytes in 195 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) May 23 12:42:16.068158 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a395b869-4da5-4c5f-aa81-e9ebdda3b11d None admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:42:16.179849 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a395b869-4da5-4c5f-aa81-e9ebdda3b11d None admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:42:16.217717 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0f27924-17a0-4a53-a5cc-503130686195 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:42:16.218401 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0f27924-17a0-4a53-a5cc-503130686195 None admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:42:16.218678 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0f27924-17a0-4a53-a5cc-503130686195 None admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:42:16.218951 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0f27924-17a0-4a53-a5cc-503130686195 None admin] PATH_INFO: `/v3/domains` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:42:16.219662 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0f27924-17a0-4a53-a5cc-503130686195 None admin] RBAC: Adding query filter params (name=Default) {{(pid=92056) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} May 23 12:42:16.219966 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0f27924-17a0-4a53-a5cc-503130686195 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:42:16.221322 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0f27924-17a0-4a53-a5cc-503130686195 None admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:42:16.233359 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 495/986] 10.222.0.30 () {56 vars in 1109 bytes} [Mon May 23 12:42:16 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 166 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) May 23 12:42:16.239670 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d7d1ecc-c314-4050-9b81-052699cd3f60 None admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:42:16.255227 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=100 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:42:16.343089 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d7d1ecc-c314-4050-9b81-052699cd3f60 None admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:42:16.378117 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd339f62-a203-49ad-b3ff-0ed76448f7c7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:42:16.378903 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd339f62-a203-49ad-b3ff-0ed76448f7c7 None admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:42:16.379185 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd339f62-a203-49ad-b3ff-0ed76448f7c7 None admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:42:16.379457 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd339f62-a203-49ad-b3ff-0ed76448f7c7 None admin] PATH_INFO: `/v3/projects/service` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:42:16.386102 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd339f62-a203-49ad-b3ff-0ed76448f7c7 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:42:16.387539 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd339f62-a203-49ad-b3ff-0ed76448f7c7 None admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:42:16.401259 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application [None req-bd339f62-a203-49ad-b3ff-0ed76448f7c7 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. May 23 12:42:16.401259 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application Traceback (most recent call last): May 23 12:42:16.401259 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request May 23 12:42:16.401259 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application rv = self.dispatch_request() May 23 12:42:16.401259 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request May 23 12:42:16.401259 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) May 23 12:42:16.401259 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper May 23 12:42:16.401259 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) May 23 12:42:16.401259 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view May 23 12:42:16.401259 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) May 23 12:42:16.401259 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request May 23 12:42:16.401259 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) May 23 12:42:16.401259 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get May 23 12:42:16.401259 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self._get_project(project_id) May 23 12:42:16.401259 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project May 23 12:42:16.402218 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) May 23 12:42:16.402218 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped May 23 12:42:16.402218 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) May 23 12:42:16.402218 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun May 23 12:42:16.402218 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) May 23 12:42:16.402218 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func May 23 12:42:16.402218 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self.get_or_create( May 23 12:42:16.402218 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create May 23 12:42:16.402218 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application with Lock( May 23 12:42:16.402218 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ May 23 12:42:16.402218 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self._enter() May 23 12:42:16.402218 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter May 23 12:42:16.402218 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) May 23 12:42:16.402218 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create May 23 12:42:16.402218 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self.creator() May 23 12:42:16.402218 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value May 23 12:42:16.402218 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application created_value = creator( May 23 12:42:16.403150 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project May 23 12:42:16.403150 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) May 23 12:42:16.403150 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project May 23 12:42:16.403150 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() May 23 12:42:16.403150 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project May 23 12:42:16.403150 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) May 23 12:42:16.403150 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. May 23 12:42:16.403150 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application May 23 12:42:16.403150 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 492/987] 10.222.0.30 () {56 vars in 1111 bytes} [Mon May 23 12:42:16 2022] GET /identity/v3/projects/service => generated 88 bytes in 163 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) May 23 12:42:16.410584 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0f27924-17a0-4a53-a5cc-503130686195 None admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:42:16.467911 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=100 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:42:16.509907 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0f27924-17a0-4a53-a5cc-503130686195 None admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:42:16.544072 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2a05eef-13ec-4676-a796-2c214a55c9a7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:42:16.544723 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2a05eef-13ec-4676-a796-2c214a55c9a7 None admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:42:16.545003 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2a05eef-13ec-4676-a796-2c214a55c9a7 None admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:42:16.545277 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2a05eef-13ec-4676-a796-2c214a55c9a7 None admin] PATH_INFO: `/v3/projects` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:42:16.546079 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f2a05eef-13ec-4676-a796-2c214a55c9a7 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=92056) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} May 23 12:42:16.546387 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f2a05eef-13ec-4676-a796-2c214a55c9a7 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:42:16.547811 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f2a05eef-13ec-4676-a796-2c214a55c9a7 None admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:42:16.559590 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 496/988] 10.222.0.30 () {56 vars in 1148 bytes} [Mon May 23 12:42:16 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 150 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) May 23 12:42:16.565352 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd339f62-a203-49ad-b3ff-0ed76448f7c7 None admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:42:16.638365 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=100 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:42:16.670613 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd339f62-a203-49ad-b3ff-0ed76448f7c7 None admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:42:16.714783 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62cf80b3-3cc2-4740-a63c-90b8ab0e000e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:42:16.715452 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62cf80b3-3cc2-4740-a63c-90b8ab0e000e None admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:42:16.715740 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62cf80b3-3cc2-4740-a63c-90b8ab0e000e None admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:42:16.716019 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62cf80b3-3cc2-4740-a63c-90b8ab0e000e None admin] PATH_INFO: `/v3/role_assignments` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:42:16.716972 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-62cf80b3-3cc2-4740-a63c-90b8ab0e000e None admin] RBAC: Adding query filter params (role.id=940396c67e6744c784e95e01fe84ad45, scope.project.id=9c389ae65aaa4542bb5a5c4fd0bacf5b, user.id=6d21a56126fd4ccd837c33ccc08ab609) {{(pid=92057) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} May 23 12:42:16.717293 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-62cf80b3-3cc2-4740-a63c-90b8ab0e000e None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:42:16.718821 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-62cf80b3-3cc2-4740-a63c-90b8ab0e000e None admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:42:16.733976 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 493/989] 10.222.0.30 () {56 vars in 1374 bytes} [Mon May 23 12:42:16 2022] GET /identity/v3/role_assignments?user.id=6d21a56126fd4ccd837c33ccc08ab609&scope.project.id=9c389ae65aaa4542bb5a5c4fd0bacf5b&role.id=940396c67e6744c784e95e01fe84ad45 => generated 608 bytes in 160 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) May 23 12:42:17.178233 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52fb5984-f680-45b5-a66e-f9e9db3caa1d None None] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:42:17.178796 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52fb5984-f680-45b5-a66e-f9e9db3caa1d None None] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:42:17.178796 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52fb5984-f680-45b5-a66e-f9e9db3caa1d None None] PATH_INFO: `/` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:42:17.180306 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 497/990] 10.222.0.30 () {54 vars in 926 bytes} [Mon May 23 12:42:17 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) May 23 12:42:17.185428 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d736996-3db9-4335-96c7-6ed97bda6f70 None None] REQUEST_METHOD: `POST` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:42:17.185828 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d736996-3db9-4335-96c7-6ed97bda6f70 None None] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:42:17.186014 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d736996-3db9-4335-96c7-6ed97bda6f70 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:42:17.203438 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.auth.core [None req-1d736996-3db9-4335-96c7-6ed97bda6f70 None None] MFA Rules not processed for user `0f7119f8fa5c4445977671140ded91a4`. Rule list: `[]` (Enabled: `True`). {{(pid=92057) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 12:42:17.204616 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.fernet_utils [None req-1d736996-3db9-4335-96c7-6ed97bda6f70 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92057) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 12:42:17.280268 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 494/991] 10.222.0.30 () {58 vars in 1023 bytes} [Mon May 23 12:42:17 2022] POST /identity/v3/auth/tokens => generated 1560 bytes in 96 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) May 23 12:42:17.304538 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52fb5984-f680-45b5-a66e-f9e9db3caa1d None None] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:42:17.412849 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52fb5984-f680-45b5-a66e-f9e9db3caa1d None None] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:42:17.431061 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=100 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:42:17.449594 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-671b6d79-d6bd-4209-8ce4-914a1453a69b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:42:17.450520 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-671b6d79-d6bd-4209-8ce4-914a1453a69b None admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:42:17.452623 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-671b6d79-d6bd-4209-8ce4-914a1453a69b None admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:42:17.452623 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-671b6d79-d6bd-4209-8ce4-914a1453a69b None admin] PATH_INFO: `/v3/services/object-store` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:42:17.452623 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-671b6d79-d6bd-4209-8ce4-914a1453a69b None admin] RBAC: Authorizing `identity:get_service(service_id=object-store)` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:42:17.453461 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-671b6d79-d6bd-4209-8ce4-914a1453a69b None admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:42:17.466070 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application [None req-671b6d79-d6bd-4209-8ce4-914a1453a69b None admin] Could not find service: object-store.: keystone.exception.ServiceNotFound: Could not find service: object-store. May 23 12:42:17.466070 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application Traceback (most recent call last): May 23 12:42:17.466070 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service May 23 12:42:17.466070 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) May 23 12:42:17.466070 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service May 23 12:42:17.466070 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() May 23 12:42:17.466070 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service May 23 12:42:17.466070 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) May 23 12:42:17.466070 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. May 23 12:42:17.466070 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application May 23 12:42:17.466070 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: May 23 12:42:17.466070 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application May 23 12:42:17.466070 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application Traceback (most recent call last): May 23 12:42:17.466070 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request May 23 12:42:17.466070 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application rv = self.dispatch_request() May 23 12:42:17.466070 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request May 23 12:42:17.466070 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) May 23 12:42:17.467218 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper May 23 12:42:17.467218 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) May 23 12:42:17.467218 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view May 23 12:42:17.467218 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) May 23 12:42:17.467218 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request May 23 12:42:17.467218 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) May 23 12:42:17.467218 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get May 23 12:42:17.467218 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self._get_service(service_id) May 23 12:42:17.467218 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service May 23 12:42:17.467218 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) May 23 12:42:17.467218 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped May 23 12:42:17.467218 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) May 23 12:42:17.467218 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun May 23 12:42:17.467218 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) May 23 12:42:17.467218 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func May 23 12:42:17.467218 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self.get_or_create( May 23 12:42:17.474064 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create May 23 12:42:17.474064 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application with Lock( May 23 12:42:17.474064 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ May 23 12:42:17.474064 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self._enter() May 23 12:42:17.474064 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter May 23 12:42:17.474064 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) May 23 12:42:17.474064 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create May 23 12:42:17.474064 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self.creator() May 23 12:42:17.474064 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value May 23 12:42:17.474064 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application created_value = creator( May 23 12:42:17.474064 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service May 23 12:42:17.474064 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) May 23 12:42:17.474064 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. May 23 12:42:17.474064 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application May 23 12:42:17.474064 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 498/992] 10.222.0.30 () {56 vars in 1126 bytes} [Mon May 23 12:42:17 2022] GET /identity/v3/services/object-store => generated 93 bytes in 164 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) May 23 12:42:17.476505 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d736996-3db9-4335-96c7-6ed97bda6f70 None None] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:42:17.545067 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=100 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:42:17.580522 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d736996-3db9-4335-96c7-6ed97bda6f70 None None] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:42:17.621654 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f556fcab-cb0a-44d8-9fa0-0e5460921331 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:42:17.622365 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f556fcab-cb0a-44d8-9fa0-0e5460921331 None admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:42:17.622646 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f556fcab-cb0a-44d8-9fa0-0e5460921331 None admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:42:17.622917 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f556fcab-cb0a-44d8-9fa0-0e5460921331 None admin] PATH_INFO: `/v3/services` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:42:17.623630 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f556fcab-cb0a-44d8-9fa0-0e5460921331 None admin] RBAC: Adding query filter params (name=object-store) {{(pid=92057) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} May 23 12:42:17.623933 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f556fcab-cb0a-44d8-9fa0-0e5460921331 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:42:17.625323 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f556fcab-cb0a-44d8-9fa0-0e5460921331 None admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:42:17.635649 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 495/993] 10.222.0.30 () {56 vars in 1122 bytes} [Mon May 23 12:42:17 2022] GET /identity/v3/services?name=object-store => generated 129 bytes in 155 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) May 23 12:42:17.636580 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-671b6d79-d6bd-4209-8ce4-914a1453a69b None admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:42:17.777496 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-671b6d79-d6bd-4209-8ce4-914a1453a69b None admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:42:17.803304 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d85d894-a90b-4340-8025-54419d5d18bf None None] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:42:17.803632 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d85d894-a90b-4340-8025-54419d5d18bf None None] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:42:17.804092 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d85d894-a90b-4340-8025-54419d5d18bf None None] PATH_INFO: `/` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:42:17.806622 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 496/994] 10.222.0.30 () {54 vars in 926 bytes} [Mon May 23 12:42:17 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) May 23 12:42:17.812118 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32c67484-4555-4eea-850d-e4d236ff1137 None None] REQUEST_METHOD: `POST` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:42:17.812677 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32c67484-4555-4eea-850d-e4d236ff1137 None None] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:42:17.812960 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32c67484-4555-4eea-850d-e4d236ff1137 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:42:17.827349 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-659a4200-20a2-48d1-99aa-74387f9c9145 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:42:17.828088 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-659a4200-20a2-48d1-99aa-74387f9c9145 None admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:42:17.828365 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-659a4200-20a2-48d1-99aa-74387f9c9145 None admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:42:17.828634 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-659a4200-20a2-48d1-99aa-74387f9c9145 None admin] PATH_INFO: `/v3/services` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:42:17.829338 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-659a4200-20a2-48d1-99aa-74387f9c9145 None admin] RBAC: Adding query filter params (type=object-store) {{(pid=92056) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} May 23 12:42:17.829640 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-659a4200-20a2-48d1-99aa-74387f9c9145 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:42:17.832293 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-659a4200-20a2-48d1-99aa-74387f9c9145 None admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:42:17.836799 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.auth.core [None req-32c67484-4555-4eea-850d-e4d236ff1137 None None] MFA Rules not processed for user `0f7119f8fa5c4445977671140ded91a4`. Rule list: `[]` (Enabled: `True`). {{(pid=92057) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 12:42:17.837607 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 499/995] 10.222.0.30 () {56 vars in 1122 bytes} [Mon May 23 12:42:17 2022] GET /identity/v3/services?type=object-store => generated 356 bytes in 201 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) May 23 12:42:17.838529 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.fernet_utils [None req-32c67484-4555-4eea-850d-e4d236ff1137 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92057) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 12:42:17.843465 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-659a4200-20a2-48d1-99aa-74387f9c9145 None admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:42:17.911624 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=100 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:42:17.936502 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 497/996] 10.222.0.30 () {58 vars in 1023 bytes} [Mon May 23 12:42:17 2022] POST /identity/v3/auth/tokens => generated 1560 bytes in 123 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) May 23 12:42:17.958681 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32c67484-4555-4eea-850d-e4d236ff1137 None None] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:42:17.974204 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-659a4200-20a2-48d1-99aa-74387f9c9145 None admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:42:18.033107 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54cfd3a2-4c77-4c48-9da6-f60ee62a572c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:42:18.034839 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54cfd3a2-4c77-4c48-9da6-f60ee62a572c None admin] REQUEST_METHOD: `POST` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:42:18.035634 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54cfd3a2-4c77-4c48-9da6-f60ee62a572c None admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:42:18.036102 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54cfd3a2-4c77-4c48-9da6-f60ee62a572c None admin] PATH_INFO: `/v3/endpoints` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:42:18.037246 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=100 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:42:18.039524 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54cfd3a2-4c77-4c48-9da6-f60ee62a572c None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:42:18.043002 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54cfd3a2-4c77-4c48-9da6-f60ee62a572c None admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:42:18.088935 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32c67484-4555-4eea-850d-e4d236ff1137 None None] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:42:18.122511 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54855d56-b5f1-4491-857f-0b13fd820e58 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:42:18.123333 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54855d56-b5f1-4491-857f-0b13fd820e58 None admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:42:18.123661 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54855d56-b5f1-4491-857f-0b13fd820e58 None admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:42:18.123981 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54855d56-b5f1-4491-857f-0b13fd820e58 None admin] PATH_INFO: `/v3/services/compute` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:42:18.124631 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54855d56-b5f1-4491-857f-0b13fd820e58 None admin] RBAC: Authorizing `identity:get_service(service_id=compute)` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:42:18.126045 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54855d56-b5f1-4491-857f-0b13fd820e58 None admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:42:18.130992 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 500/997] 10.222.0.30 () {60 vars in 1144 bytes} [Mon May 23 12:42:17 2022] POST /identity/v3/endpoints => generated 325 bytes in 288 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) May 23 12:42:18.142100 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application [None req-54855d56-b5f1-4491-857f-0b13fd820e58 None admin] Could not find service: compute.: keystone.exception.ServiceNotFound: Could not find service: compute. May 23 12:42:18.142100 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application Traceback (most recent call last): May 23 12:42:18.142100 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service May 23 12:42:18.142100 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) May 23 12:42:18.142100 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service May 23 12:42:18.142100 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() May 23 12:42:18.142100 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service May 23 12:42:18.142100 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) May 23 12:42:18.142100 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute. May 23 12:42:18.142100 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application May 23 12:42:18.142100 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: May 23 12:42:18.142100 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application May 23 12:42:18.142100 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application Traceback (most recent call last): May 23 12:42:18.142100 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request May 23 12:42:18.142100 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application rv = self.dispatch_request() May 23 12:42:18.142100 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request May 23 12:42:18.142100 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) May 23 12:42:18.143309 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper May 23 12:42:18.143309 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) May 23 12:42:18.143309 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view May 23 12:42:18.143309 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) May 23 12:42:18.143309 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request May 23 12:42:18.143309 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) May 23 12:42:18.143309 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get May 23 12:42:18.143309 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self._get_service(service_id) May 23 12:42:18.143309 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service May 23 12:42:18.143309 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) May 23 12:42:18.143309 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped May 23 12:42:18.143309 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) May 23 12:42:18.143309 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun May 23 12:42:18.143309 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) May 23 12:42:18.143309 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func May 23 12:42:18.143309 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self.get_or_create( May 23 12:42:18.145255 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create May 23 12:42:18.145255 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application with Lock( May 23 12:42:18.145255 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ May 23 12:42:18.145255 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self._enter() May 23 12:42:18.145255 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter May 23 12:42:18.145255 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) May 23 12:42:18.145255 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create May 23 12:42:18.145255 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self.creator() May 23 12:42:18.145255 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value May 23 12:42:18.145255 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application created_value = creator( May 23 12:42:18.145255 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service May 23 12:42:18.145255 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) May 23 12:42:18.145255 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute. May 23 12:42:18.145255 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application May 23 12:42:18.145255 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 498/998] 10.222.0.30 () {56 vars in 1111 bytes} [Mon May 23 12:42:17 2022] GET /identity/v3/services/compute => generated 88 bytes in 182 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) May 23 12:42:18.146692 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54855d56-b5f1-4491-857f-0b13fd820e58 None admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:42:18.260985 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54855d56-b5f1-4491-857f-0b13fd820e58 None admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:42:18.286114 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=100 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:42:18.304599 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c63bd016-8e57-4575-a01d-8aec26cb5819 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:42:18.305203 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c63bd016-8e57-4575-a01d-8aec26cb5819 None admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:42:18.305485 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c63bd016-8e57-4575-a01d-8aec26cb5819 None admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:42:18.305758 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c63bd016-8e57-4575-a01d-8aec26cb5819 None admin] PATH_INFO: `/v3/services` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:42:18.306494 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c63bd016-8e57-4575-a01d-8aec26cb5819 None admin] RBAC: Adding query filter params (name=compute) {{(pid=92057) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} May 23 12:42:18.306798 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c63bd016-8e57-4575-a01d-8aec26cb5819 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:42:18.308175 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c63bd016-8e57-4575-a01d-8aec26cb5819 None admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:42:18.313528 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 499/999] 10.222.0.30 () {56 vars in 1112 bytes} [Mon May 23 12:42:18 2022] GET /identity/v3/services?name=compute => generated 124 bytes in 167 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) May 23 12:42:18.320341 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54cfd3a2-4c77-4c48-9da6-f60ee62a572c None admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:42:18.431300 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54cfd3a2-4c77-4c48-9da6-f60ee62a572c None admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:42:18.438920 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:42:18.467327 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0e9afc5-db56-4e31-954c-105af519d6a9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:42:18.468041 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0e9afc5-db56-4e31-954c-105af519d6a9 None admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:42:18.468320 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0e9afc5-db56-4e31-954c-105af519d6a9 None admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:42:18.468639 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0e9afc5-db56-4e31-954c-105af519d6a9 None admin] PATH_INFO: `/v3/services` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:42:18.469346 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0e9afc5-db56-4e31-954c-105af519d6a9 None admin] RBAC: Adding query filter params (type=compute) {{(pid=92056) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} May 23 12:42:18.469647 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0e9afc5-db56-4e31-954c-105af519d6a9 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:42:18.471062 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0e9afc5-db56-4e31-954c-105af519d6a9 None admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:42:18.481279 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 501/1000] 10.222.0.30 () {56 vars in 1112 bytes} [Mon May 23 12:42:18 2022] GET /identity/v3/services?type=compute => generated 352 bytes in 157 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) May 23 12:42:18.482557 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c63bd016-8e57-4575-a01d-8aec26cb5819 None admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:42:18.587825 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c63bd016-8e57-4575-a01d-8aec26cb5819 None admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:42:18.623382 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d72eea44-d810-4466-8f1d-c7a18db79ad4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:42:18.624393 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d72eea44-d810-4466-8f1d-c7a18db79ad4 None admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:42:18.624891 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d72eea44-d810-4466-8f1d-c7a18db79ad4 None admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:42:18.625361 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d72eea44-d810-4466-8f1d-c7a18db79ad4 None admin] PATH_INFO: `/v3/endpoints` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:42:18.626385 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d72eea44-d810-4466-8f1d-c7a18db79ad4 None admin] RBAC: Adding query filter params (interface=public, service_id=842ffb3bd8994cb8a73a74e5a0ffd3ff, region_id=RegionOne) {{(pid=92057) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} May 23 12:42:18.626957 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d72eea44-d810-4466-8f1d-c7a18db79ad4 None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:42:18.628857 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d72eea44-d810-4466-8f1d-c7a18db79ad4 None admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:42:18.634752 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 500/1001] 10.222.0.30 () {56 vars in 1251 bytes} [Mon May 23 12:42:18 2022] GET /identity/v3/endpoints?service_id=842ffb3bd8994cb8a73a74e5a0ffd3ff&interface=public®ion_id=RegionOne => generated 194 bytes in 153 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) May 23 12:42:18.640324 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0e9afc5-db56-4e31-954c-105af519d6a9 None admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:42:18.752409 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0e9afc5-db56-4e31-954c-105af519d6a9 None admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:42:18.787002 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d43f5291-3f35-44a0-bfd9-7f19d0d88506 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:42:18.787986 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d43f5291-3f35-44a0-bfd9-7f19d0d88506 None admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:42:18.788483 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d43f5291-3f35-44a0-bfd9-7f19d0d88506 None admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:42:18.788952 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d43f5291-3f35-44a0-bfd9-7f19d0d88506 None admin] PATH_INFO: `/v3/services` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:42:18.789831 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d43f5291-3f35-44a0-bfd9-7f19d0d88506 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:42:18.791416 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d43f5291-3f35-44a0-bfd9-7f19d0d88506 None admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:42:18.798121 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 502/1002] 10.222.0.30 () {56 vars in 1087 bytes} [Mon May 23 12:42:18 2022] GET /identity/v3/services => generated 1727 bytes in 158 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) May 23 12:42:19.059839 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-012c6d6f-bd00-4eb5-814d-1f92b86773e6 None None] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:42:19.060180 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-012c6d6f-bd00-4eb5-814d-1f92b86773e6 None None] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:42:19.060435 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-012c6d6f-bd00-4eb5-814d-1f92b86773e6 None None] PATH_INFO: `/` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:42:19.062164 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 501/1003] 10.222.0.30 () {54 vars in 926 bytes} [Mon May 23 12:42:19 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) May 23 12:42:19.067368 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfae9ff8-4c85-457a-a5bf-5581ae70bb30 None None] REQUEST_METHOD: `POST` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:42:19.067666 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfae9ff8-4c85-457a-a5bf-5581ae70bb30 None None] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:42:19.067946 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfae9ff8-4c85-457a-a5bf-5581ae70bb30 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:42:19.085773 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.auth.core [None req-cfae9ff8-4c85-457a-a5bf-5581ae70bb30 None None] MFA Rules not processed for user `0f7119f8fa5c4445977671140ded91a4`. Rule list: `[]` (Enabled: `True`). {{(pid=92056) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 12:42:19.087672 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.fernet_utils [None req-cfae9ff8-4c85-457a-a5bf-5581ae70bb30 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92056) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 12:42:19.119481 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=100 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:42:19.162425 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 503/1004] 10.222.0.30 () {58 vars in 1023 bytes} [Mon May 23 12:42:19 2022] POST /identity/v3/auth/tokens => generated 1707 bytes in 96 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) May 23 12:42:19.184104 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-012c6d6f-bd00-4eb5-814d-1f92b86773e6 None None] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:42:19.237003 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=100 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:42:19.295195 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-012c6d6f-bd00-4eb5-814d-1f92b86773e6 None None] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:42:19.332004 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1804cacc-9848-48d5-b771-3b80c2ab9a13 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:42:19.332889 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1804cacc-9848-48d5-b771-3b80c2ab9a13 None admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:42:19.333212 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1804cacc-9848-48d5-b771-3b80c2ab9a13 None admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:42:19.333493 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1804cacc-9848-48d5-b771-3b80c2ab9a13 None admin] PATH_INFO: `/v3/services/image` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:42:19.334239 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1804cacc-9848-48d5-b771-3b80c2ab9a13 None admin] RBAC: Authorizing `identity:get_service(service_id=image)` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:42:19.335750 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1804cacc-9848-48d5-b771-3b80c2ab9a13 None admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:42:19.350563 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application [None req-1804cacc-9848-48d5-b771-3b80c2ab9a13 None admin] Could not find service: image.: keystone.exception.ServiceNotFound: Could not find service: image. May 23 12:42:19.350563 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application Traceback (most recent call last): May 23 12:42:19.350563 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service May 23 12:42:19.350563 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) May 23 12:42:19.350563 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service May 23 12:42:19.350563 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() May 23 12:42:19.350563 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service May 23 12:42:19.350563 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) May 23 12:42:19.350563 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: image. May 23 12:42:19.350563 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application May 23 12:42:19.350563 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: May 23 12:42:19.350563 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application May 23 12:42:19.350563 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application Traceback (most recent call last): May 23 12:42:19.350563 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request May 23 12:42:19.350563 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application rv = self.dispatch_request() May 23 12:42:19.350563 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request May 23 12:42:19.350563 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) May 23 12:42:19.351810 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper May 23 12:42:19.351810 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) May 23 12:42:19.351810 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view May 23 12:42:19.351810 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) May 23 12:42:19.351810 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request May 23 12:42:19.351810 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) May 23 12:42:19.351810 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get May 23 12:42:19.351810 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self._get_service(service_id) May 23 12:42:19.351810 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service May 23 12:42:19.351810 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) May 23 12:42:19.351810 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped May 23 12:42:19.351810 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) May 23 12:42:19.351810 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun May 23 12:42:19.351810 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) May 23 12:42:19.351810 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func May 23 12:42:19.351810 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self.get_or_create( May 23 12:42:19.352763 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create May 23 12:42:19.352763 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application with Lock( May 23 12:42:19.352763 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ May 23 12:42:19.352763 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self._enter() May 23 12:42:19.352763 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter May 23 12:42:19.352763 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) May 23 12:42:19.352763 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create May 23 12:42:19.352763 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self.creator() May 23 12:42:19.352763 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value May 23 12:42:19.352763 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application created_value = creator( May 23 12:42:19.352763 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service May 23 12:42:19.352763 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) May 23 12:42:19.352763 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: image. May 23 12:42:19.352763 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application May 23 12:42:19.357213 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 502/1005] 10.222.0.30 () {56 vars in 1105 bytes} [Mon May 23 12:42:19 2022] GET /identity/v3/services/image => generated 86 bytes in 173 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) May 23 12:42:19.358093 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cfae9ff8-4c85-457a-a5bf-5581ae70bb30 None None] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:42:19.460683 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cfae9ff8-4c85-457a-a5bf-5581ae70bb30 None None] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:42:19.498151 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7bfa6dc-94b9-484c-ad30-a0ac7f4311f6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:42:19.498884 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7bfa6dc-94b9-484c-ad30-a0ac7f4311f6 None admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:42:19.499193 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7bfa6dc-94b9-484c-ad30-a0ac7f4311f6 None admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:42:19.499466 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7bfa6dc-94b9-484c-ad30-a0ac7f4311f6 None admin] PATH_INFO: `/v3/services` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:42:19.500246 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7bfa6dc-94b9-484c-ad30-a0ac7f4311f6 None admin] RBAC: Adding query filter params (name=image) {{(pid=92056) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} May 23 12:42:19.500548 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7bfa6dc-94b9-484c-ad30-a0ac7f4311f6 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:42:19.501927 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7bfa6dc-94b9-484c-ad30-a0ac7f4311f6 None admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:42:19.512317 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 504/1006] 10.222.0.30 () {56 vars in 1108 bytes} [Mon May 23 12:42:19 2022] GET /identity/v3/services?name=image => generated 122 bytes in 150 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) May 23 12:42:19.513898 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1804cacc-9848-48d5-b771-3b80c2ab9a13 None admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:42:19.618446 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1804cacc-9848-48d5-b771-3b80c2ab9a13 None admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:42:19.631845 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=100 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:42:19.653787 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b871cbe9-6cd8-4189-9434-aa53a12fe220 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:42:19.654481 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b871cbe9-6cd8-4189-9434-aa53a12fe220 None admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:42:19.654769 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b871cbe9-6cd8-4189-9434-aa53a12fe220 None admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:42:19.655047 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b871cbe9-6cd8-4189-9434-aa53a12fe220 None admin] PATH_INFO: `/v3/services` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:42:19.655746 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b871cbe9-6cd8-4189-9434-aa53a12fe220 None admin] RBAC: Adding query filter params (type=image) {{(pid=92057) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} May 23 12:42:19.656054 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b871cbe9-6cd8-4189-9434-aa53a12fe220 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:42:19.657495 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b871cbe9-6cd8-4189-9434-aa53a12fe220 None admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:42:19.670178 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 503/1007] 10.222.0.30 () {56 vars in 1108 bytes} [Mon May 23 12:42:19 2022] GET /identity/v3/services?type=image => generated 122 bytes in 149 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) May 23 12:42:20.419582 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7f8b162-f6c8-4a66-99f4-b1d3091cd53c None None] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:42:20.420177 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7f8b162-f6c8-4a66-99f4-b1d3091cd53c None None] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:42:20.420177 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7f8b162-f6c8-4a66-99f4-b1d3091cd53c None None] PATH_INFO: `/` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:42:20.421646 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 505/1008] 10.222.0.30 () {54 vars in 926 bytes} [Mon May 23 12:42:20 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) May 23 12:42:20.427538 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-903cbb40-7afa-43c1-96fd-ceeb8d8843d5 None None] REQUEST_METHOD: `POST` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:42:20.427970 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-903cbb40-7afa-43c1-96fd-ceeb8d8843d5 None None] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:42:20.428157 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-903cbb40-7afa-43c1-96fd-ceeb8d8843d5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:42:20.446052 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.auth.core [None req-903cbb40-7afa-43c1-96fd-ceeb8d8843d5 None None] MFA Rules not processed for user `0f7119f8fa5c4445977671140ded91a4`. Rule list: `[]` (Enabled: `True`). {{(pid=92057) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 12:42:20.447233 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.fernet_utils [None req-903cbb40-7afa-43c1-96fd-ceeb8d8843d5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92057) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 12:42:20.517694 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 504/1009] 10.222.0.30 () {58 vars in 1023 bytes} [Mon May 23 12:42:20 2022] POST /identity/v3/auth/tokens => generated 1707 bytes in 91 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) May 23 12:42:20.541985 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7f8b162-f6c8-4a66-99f4-b1d3091cd53c None None] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:42:20.573436 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=100 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:42:20.648681 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7f8b162-f6c8-4a66-99f4-b1d3091cd53c None None] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:42:20.688843 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dfad1527-42fc-4eec-888b-3bb850d95815 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:42:20.693153 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfad1527-42fc-4eec-888b-3bb850d95815 None admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:42:20.693153 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfad1527-42fc-4eec-888b-3bb850d95815 None admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:42:20.693153 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfad1527-42fc-4eec-888b-3bb850d95815 None admin] PATH_INFO: `/v3/domains/default` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:42:20.693153 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dfad1527-42fc-4eec-888b-3bb850d95815 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:42:20.693903 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dfad1527-42fc-4eec-888b-3bb850d95815 None admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:42:20.700499 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 506/1010] 10.222.0.30 () {56 vars in 1108 bytes} [Mon May 23 12:42:20 2022] GET /identity/v3/domains/default => generated 199 bytes in 155 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) May 23 12:42:20.701844 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-903cbb40-7afa-43c1-96fd-ceeb8d8843d5 None None] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:42:20.805904 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-903cbb40-7afa-43c1-96fd-ceeb8d8843d5 None None] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:42:20.813431 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=100 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:42:20.841822 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60096030-bb08-4563-ac58-a34d81c2f779 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:42:20.842923 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60096030-bb08-4563-ac58-a34d81c2f779 None admin] REQUEST_METHOD: `POST` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:42:20.843458 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60096030-bb08-4563-ac58-a34d81c2f779 None admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:42:20.844116 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60096030-bb08-4563-ac58-a34d81c2f779 None admin] PATH_INFO: `/v3/projects` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:42:20.845052 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60096030-bb08-4563-ac58-a34d81c2f779 None admin] RBAC: Authorizing `identity:create_project()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:42:20.847339 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60096030-bb08-4563-ac58-a34d81c2f779 None admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:42:20.965133 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 505/1011] 10.222.0.30 () {60 vars in 1141 bytes} [Mon May 23 12:42:20 2022] POST /identity/v3/projects => generated 311 bytes in 264 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) May 23 12:42:21.045966 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9ea37dc-b3a2-4585-9520-a417ca95016b None None] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:42:21.046322 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9ea37dc-b3a2-4585-9520-a417ca95016b None None] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:42:21.046533 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9ea37dc-b3a2-4585-9520-a417ca95016b None None] PATH_INFO: `/` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:42:21.048141 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 507/1012] 10.222.0.30 () {54 vars in 926 bytes} [Mon May 23 12:42:21 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) May 23 12:42:21.054126 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d07c4ec4-dd4a-495e-84a6-52758dd2046b None None] REQUEST_METHOD: `POST` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:42:21.054418 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d07c4ec4-dd4a-495e-84a6-52758dd2046b None None] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:42:21.054694 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d07c4ec4-dd4a-495e-84a6-52758dd2046b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:42:21.073397 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.auth.core [None req-d07c4ec4-dd4a-495e-84a6-52758dd2046b None None] MFA Rules not processed for user `0f7119f8fa5c4445977671140ded91a4`. Rule list: `[]` (Enabled: `True`). {{(pid=92057) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 12:42:21.074758 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.fernet_utils [None req-d07c4ec4-dd4a-495e-84a6-52758dd2046b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92057) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 12:42:21.153464 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 506/1013] 10.222.0.30 () {58 vars in 1023 bytes} [Mon May 23 12:42:21 2022] POST /identity/v3/auth/tokens => generated 1707 bytes in 101 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) May 23 12:42:21.176922 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9ea37dc-b3a2-4585-9520-a417ca95016b None None] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:42:21.275199 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=100 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:42:21.281895 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9ea37dc-b3a2-4585-9520-a417ca95016b None None] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:42:21.318374 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be55619c-08cc-4a33-b34b-c18754fdd918 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:42:21.319180 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be55619c-08cc-4a33-b34b-c18754fdd918 None admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:42:21.319462 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be55619c-08cc-4a33-b34b-c18754fdd918 None admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:42:21.319744 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be55619c-08cc-4a33-b34b-c18754fdd918 None admin] PATH_INFO: `/v3/services/compute` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:42:21.320390 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be55619c-08cc-4a33-b34b-c18754fdd918 None admin] RBAC: Authorizing `identity:get_service(service_id=compute)` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:42:21.321767 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be55619c-08cc-4a33-b34b-c18754fdd918 None admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:42:21.334308 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application [None req-be55619c-08cc-4a33-b34b-c18754fdd918 None admin] Could not find service: compute.: keystone.exception.ServiceNotFound: Could not find service: compute. May 23 12:42:21.334308 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application Traceback (most recent call last): May 23 12:42:21.334308 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service May 23 12:42:21.334308 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) May 23 12:42:21.334308 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service May 23 12:42:21.334308 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() May 23 12:42:21.334308 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service May 23 12:42:21.334308 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) May 23 12:42:21.334308 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute. May 23 12:42:21.334308 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application May 23 12:42:21.334308 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: May 23 12:42:21.334308 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application May 23 12:42:21.334308 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application Traceback (most recent call last): May 23 12:42:21.334308 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request May 23 12:42:21.334308 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application rv = self.dispatch_request() May 23 12:42:21.334308 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request May 23 12:42:21.334308 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) May 23 12:42:21.335372 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper May 23 12:42:21.335372 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) May 23 12:42:21.335372 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view May 23 12:42:21.335372 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) May 23 12:42:21.335372 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request May 23 12:42:21.335372 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) May 23 12:42:21.335372 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get May 23 12:42:21.335372 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self._get_service(service_id) May 23 12:42:21.335372 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service May 23 12:42:21.335372 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) May 23 12:42:21.335372 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped May 23 12:42:21.335372 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) May 23 12:42:21.335372 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun May 23 12:42:21.335372 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) May 23 12:42:21.335372 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func May 23 12:42:21.335372 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self.get_or_create( May 23 12:42:21.336245 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create May 23 12:42:21.336245 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application with Lock( May 23 12:42:21.336245 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ May 23 12:42:21.336245 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self._enter() May 23 12:42:21.336245 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter May 23 12:42:21.336245 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) May 23 12:42:21.336245 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create May 23 12:42:21.336245 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self.creator() May 23 12:42:21.336245 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value May 23 12:42:21.336245 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application created_value = creator( May 23 12:42:21.336245 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service May 23 12:42:21.336245 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) May 23 12:42:21.336245 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute. May 23 12:42:21.336245 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application May 23 12:42:21.336245 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 508/1014] 10.222.0.30 () {56 vars in 1111 bytes} [Mon May 23 12:42:21 2022] GET /identity/v3/services/compute => generated 88 bytes in 159 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) May 23 12:42:21.343010 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d07c4ec4-dd4a-495e-84a6-52758dd2046b None None] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:42:21.438826 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:42:21.452157 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d07c4ec4-dd4a-495e-84a6-52758dd2046b None None] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:42:21.491043 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ec301b2-d2a2-47db-a777-3d66637b1a85 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:42:21.491717 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ec301b2-d2a2-47db-a777-3d66637b1a85 None admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:42:21.492002 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ec301b2-d2a2-47db-a777-3d66637b1a85 None admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:42:21.492281 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ec301b2-d2a2-47db-a777-3d66637b1a85 None admin] PATH_INFO: `/v3/services` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:42:21.492989 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ec301b2-d2a2-47db-a777-3d66637b1a85 None admin] RBAC: Adding query filter params (name=compute) {{(pid=92057) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} May 23 12:42:21.493301 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ec301b2-d2a2-47db-a777-3d66637b1a85 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:42:21.494713 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ec301b2-d2a2-47db-a777-3d66637b1a85 None admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:42:21.504301 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 507/1015] 10.222.0.30 () {56 vars in 1112 bytes} [Mon May 23 12:42:21 2022] GET /identity/v3/services?name=compute => generated 124 bytes in 157 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) May 23 12:42:21.506139 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be55619c-08cc-4a33-b34b-c18754fdd918 None admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:42:21.615736 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be55619c-08cc-4a33-b34b-c18754fdd918 None admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:42:21.651997 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d3827c0-1106-4dc1-ba66-fb75b35f4b3f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:42:21.652658 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d3827c0-1106-4dc1-ba66-fb75b35f4b3f None admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:42:21.652940 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d3827c0-1106-4dc1-ba66-fb75b35f4b3f None admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:42:21.653214 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d3827c0-1106-4dc1-ba66-fb75b35f4b3f None admin] PATH_INFO: `/v3/services` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:42:21.653941 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d3827c0-1106-4dc1-ba66-fb75b35f4b3f None admin] RBAC: Adding query filter params (type=compute) {{(pid=92056) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} May 23 12:42:21.654249 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d3827c0-1106-4dc1-ba66-fb75b35f4b3f None admin] RBAC: Authorizing `identity:list_services()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:42:21.655615 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d3827c0-1106-4dc1-ba66-fb75b35f4b3f None admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:42:21.660964 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 509/1016] 10.222.0.30 () {56 vars in 1112 bytes} [Mon May 23 12:42:21 2022] GET /identity/v3/services?type=compute => generated 352 bytes in 155 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) May 23 12:42:21.666406 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ec301b2-d2a2-47db-a777-3d66637b1a85 None admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:42:21.784650 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ec301b2-d2a2-47db-a777-3d66637b1a85 None admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:42:21.819958 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e34a6b7-cf21-4983-8bb2-7720b559c855 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:42:21.820611 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e34a6b7-cf21-4983-8bb2-7720b559c855 None admin] REQUEST_METHOD: `POST` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:42:21.820897 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e34a6b7-cf21-4983-8bb2-7720b559c855 None admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:42:21.821173 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e34a6b7-cf21-4983-8bb2-7720b559c855 None admin] PATH_INFO: `/v3/endpoints` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:42:21.821947 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e34a6b7-cf21-4983-8bb2-7720b559c855 None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:42:21.823321 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e34a6b7-cf21-4983-8bb2-7720b559c855 None admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:42:21.913993 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 508/1017] 10.222.0.30 () {60 vars in 1144 bytes} [Mon May 23 12:42:21 2022] POST /identity/v3/endpoints => generated 334 bytes in 241 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) May 23 12:42:21.916258 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22432443-f36f-4f86-8ae2-51949d2c8661 None None] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:42:21.916562 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22432443-f36f-4f86-8ae2-51949d2c8661 None None] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:42:21.916851 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22432443-f36f-4f86-8ae2-51949d2c8661 None None] PATH_INFO: `/` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:42:21.918551 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 510/1018] 10.222.0.30 () {54 vars in 926 bytes} [Mon May 23 12:42:21 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) May 23 12:42:21.924649 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d553f776-5633-4eaf-92d8-b2af58aed0aa None None] REQUEST_METHOD: `POST` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:42:21.925036 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d553f776-5633-4eaf-92d8-b2af58aed0aa None None] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:42:21.925325 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d553f776-5633-4eaf-92d8-b2af58aed0aa None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:42:21.943757 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.auth.core [None req-d553f776-5633-4eaf-92d8-b2af58aed0aa None None] MFA Rules not processed for user `0f7119f8fa5c4445977671140ded91a4`. Rule list: `[]` (Enabled: `True`). {{(pid=92057) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 12:42:21.945067 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.fernet_utils [None req-d553f776-5633-4eaf-92d8-b2af58aed0aa None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92057) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 12:42:21.947618 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:42:22.027924 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 509/1019] 10.222.0.30 () {58 vars in 1023 bytes} [Mon May 23 12:42:21 2022] POST /identity/v3/auth/tokens => generated 1863 bytes in 104 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) May 23 12:42:22.051765 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22432443-f36f-4f86-8ae2-51949d2c8661 None None] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:42:22.079854 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=100 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:42:22.169004 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22432443-f36f-4f86-8ae2-51949d2c8661 None None] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:42:22.207304 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f365efb6-3c23-4105-99e3-3e00507c9dd8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:42:22.207977 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f365efb6-3c23-4105-99e3-3e00507c9dd8 None admin] REQUEST_METHOD: `POST` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:42:22.208302 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f365efb6-3c23-4105-99e3-3e00507c9dd8 None admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:42:22.208626 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f365efb6-3c23-4105-99e3-3e00507c9dd8 None admin] PATH_INFO: `/v3/services` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:42:22.209472 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f365efb6-3c23-4105-99e3-3e00507c9dd8 None admin] RBAC: Authorizing `identity:create_service()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:42:22.211151 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f365efb6-3c23-4105-99e3-3e00507c9dd8 None admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:42:22.290404 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 511/1020] 10.222.0.30 () {60 vars in 1141 bytes} [Mon May 23 12:42:22 2022] POST /identity/v3/services => generated 242 bytes in 239 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) May 23 12:42:23.266567 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31678ce0-9167-43e2-8bd8-4c6068339062 None None] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:42:23.267092 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31678ce0-9167-43e2-8bd8-4c6068339062 None None] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:42:23.267172 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31678ce0-9167-43e2-8bd8-4c6068339062 None None] PATH_INFO: `/` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:42:23.268528 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 510/1021] 10.222.0.30 () {54 vars in 926 bytes} [Mon May 23 12:42:23 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) May 23 12:42:23.273983 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-657ef714-1256-40fb-9619-0dba1d719132 None None] REQUEST_METHOD: `POST` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:42:23.274373 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-657ef714-1256-40fb-9619-0dba1d719132 None None] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:42:23.274615 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-657ef714-1256-40fb-9619-0dba1d719132 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:42:23.292461 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.auth.core [None req-657ef714-1256-40fb-9619-0dba1d719132 None None] MFA Rules not processed for user `0f7119f8fa5c4445977671140ded91a4`. Rule list: `[]` (Enabled: `True`). {{(pid=92056) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 12:42:23.293640 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.fernet_utils [None req-657ef714-1256-40fb-9619-0dba1d719132 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92056) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 12:42:23.367076 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 512/1022] 10.222.0.30 () {58 vars in 1023 bytes} [Mon May 23 12:42:23 2022] POST /identity/v3/auth/tokens => generated 1959 bytes in 94 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) May 23 12:42:23.391853 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31678ce0-9167-43e2-8bd8-4c6068339062 None None] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:42:23.502867 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31678ce0-9167-43e2-8bd8-4c6068339062 None None] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:42:23.539651 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f4a3b3e-3013-4497-9f65-39f670b7816f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:42:23.540363 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f4a3b3e-3013-4497-9f65-39f670b7816f None admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:42:23.540649 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f4a3b3e-3013-4497-9f65-39f670b7816f None admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:42:23.540926 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f4a3b3e-3013-4497-9f65-39f670b7816f None admin] PATH_INFO: `/v3/domains/default` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:42:23.542123 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f4a3b3e-3013-4497-9f65-39f670b7816f None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:42:23.543580 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f4a3b3e-3013-4497-9f65-39f670b7816f None admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:42:23.550028 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 511/1023] 10.222.0.30 () {56 vars in 1108 bytes} [Mon May 23 12:42:23 2022] GET /identity/v3/domains/default => generated 199 bytes in 158 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) May 23 12:42:23.551273 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-657ef714-1256-40fb-9619-0dba1d719132 None None] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:42:23.562893 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:42:23.650872 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-657ef714-1256-40fb-9619-0dba1d719132 None None] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:42:23.692452 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cea4751a-1eb3-4b1f-ae92-91127c91438d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:42:23.692452 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cea4751a-1eb3-4b1f-ae92-91127c91438d None admin] REQUEST_METHOD: `POST` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:42:23.692452 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cea4751a-1eb3-4b1f-ae92-91127c91438d None admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:42:23.692452 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cea4751a-1eb3-4b1f-ae92-91127c91438d None admin] PATH_INFO: `/v3/users` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:42:23.692868 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cea4751a-1eb3-4b1f-ae92-91127c91438d None admin] RBAC: Authorizing `identity:create_user()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:42:23.693791 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cea4751a-1eb3-4b1f-ae92-91127c91438d None admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:42:23.810010 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 513/1024] 10.222.0.30 () {60 vars in 1132 bytes} [Mon May 23 12:42:23 2022] POST /identity/v3/users => generated 285 bytes in 252 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) May 23 12:42:24.245512 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37deca29-9498-4e6b-a093-13e15afdf647 None None] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:42:24.245804 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37deca29-9498-4e6b-a093-13e15afdf647 None None] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:42:24.246282 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37deca29-9498-4e6b-a093-13e15afdf647 None None] PATH_INFO: `/` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:42:24.247801 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 512/1025] 10.222.0.30 () {54 vars in 926 bytes} [Mon May 23 12:42:24 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) May 23 12:42:24.253335 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44db92f6-4394-4f6e-b723-4de5ea1b7186 None None] REQUEST_METHOD: `POST` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:42:24.253626 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44db92f6-4394-4f6e-b723-4de5ea1b7186 None None] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:42:24.253923 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44db92f6-4394-4f6e-b723-4de5ea1b7186 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:42:24.270640 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.auth.core [None req-44db92f6-4394-4f6e-b723-4de5ea1b7186 None None] MFA Rules not processed for user `0f7119f8fa5c4445977671140ded91a4`. Rule list: `[]` (Enabled: `True`). {{(pid=92056) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 12:42:24.271820 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.fernet_utils [None req-44db92f6-4394-4f6e-b723-4de5ea1b7186 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92056) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 12:42:24.332433 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:42:24.341756 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 514/1026] 10.222.0.30 () {58 vars in 1023 bytes} [Mon May 23 12:42:24 2022] POST /identity/v3/auth/tokens => generated 1959 bytes in 89 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) May 23 12:42:24.364707 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37deca29-9498-4e6b-a093-13e15afdf647 None None] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:42:24.367735 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=100 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:42:24.479971 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37deca29-9498-4e6b-a093-13e15afdf647 None None] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:42:24.513869 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d294bc98-148b-449d-b050-8ea21ced0087 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:42:24.514628 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d294bc98-148b-449d-b050-8ea21ced0087 None admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:42:24.514910 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d294bc98-148b-449d-b050-8ea21ced0087 None admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:42:24.515184 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d294bc98-148b-449d-b050-8ea21ced0087 None admin] PATH_INFO: `/v3/roles/ResellerAdmin` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:42:24.526219 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d294bc98-148b-449d-b050-8ea21ced0087 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} May 23 12:42:24.535471 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application [None req-d294bc98-148b-449d-b050-8ea21ced0087 None admin] Could not find role: ResellerAdmin.: keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. May 23 12:42:24.535471 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application Traceback (most recent call last): May 23 12:42:24.535471 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request May 23 12:42:24.535471 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application rv = self.dispatch_request() May 23 12:42:24.535471 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request May 23 12:42:24.535471 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) May 23 12:42:24.535471 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper May 23 12:42:24.535471 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) May 23 12:42:24.535471 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view May 23 12:42:24.535471 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) May 23 12:42:24.535471 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request May 23 12:42:24.535471 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) May 23 12:42:24.535471 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get May 23 12:42:24.535471 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self._get_role(role_id) May 23 12:42:24.535471 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role May 23 12:42:24.535471 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') May 23 12:42:24.536423 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call May 23 12:42:24.536423 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( May 23 12:42:24.536423 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data May 23 12:42:24.536423 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) May 23 12:42:24.536423 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped May 23 12:42:24.536423 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) May 23 12:42:24.536423 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun May 23 12:42:24.536423 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) May 23 12:42:24.536423 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func May 23 12:42:24.536423 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self.get_or_create( May 23 12:42:24.536423 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create May 23 12:42:24.536423 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application with Lock( May 23 12:42:24.536423 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ May 23 12:42:24.536423 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self._enter() May 23 12:42:24.536423 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter May 23 12:42:24.536423 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) May 23 12:42:24.537297 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create May 23 12:42:24.537297 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self.creator() May 23 12:42:24.537297 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value May 23 12:42:24.537297 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application created_value = creator( May 23 12:42:24.537297 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role May 23 12:42:24.537297 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) May 23 12:42:24.537297 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role May 23 12:42:24.537297 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() May 23 12:42:24.537297 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role May 23 12:42:24.537297 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) May 23 12:42:24.537297 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. May 23 12:42:24.537297 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application May 23 12:42:24.538727 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 513/1027] 10.222.0.30 () {56 vars in 1120 bytes} [Mon May 23 12:42:24 2022] GET /identity/v3/roles/ResellerAdmin => generated 91 bytes in 175 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) May 23 12:42:24.542885 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44db92f6-4394-4f6e-b723-4de5ea1b7186 None None] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:42:24.576341 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41e47488-af45-49fe-a1d0-34bcdbfada32 None None] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:42:24.576667 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41e47488-af45-49fe-a1d0-34bcdbfada32 None None] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:42:24.576988 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41e47488-af45-49fe-a1d0-34bcdbfada32 None None] PATH_INFO: `/` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:42:24.578638 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 514/1028] 10.222.0.30 () {54 vars in 926 bytes} [Mon May 23 12:42:24 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) May 23 12:42:24.584558 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a94928cf-07e9-43d7-ac61-ed8826ecb672 None None] REQUEST_METHOD: `POST` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:42:24.584890 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a94928cf-07e9-43d7-ac61-ed8826ecb672 None None] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:42:24.585217 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a94928cf-07e9-43d7-ac61-ed8826ecb672 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:42:24.603249 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.auth.core [None req-a94928cf-07e9-43d7-ac61-ed8826ecb672 None None] MFA Rules not processed for user `0f7119f8fa5c4445977671140ded91a4`. Rule list: `[]` (Enabled: `True`). {{(pid=92057) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 12:42:24.604494 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.fernet_utils [None req-a94928cf-07e9-43d7-ac61-ed8826ecb672 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92057) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 12:42:24.647036 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44db92f6-4394-4f6e-b723-4de5ea1b7186 None None] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:42:24.665496 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=100 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:42:24.682626 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 515/1029] 10.222.0.30 () {58 vars in 1023 bytes} [Mon May 23 12:42:24 2022] POST /identity/v3/auth/tokens => generated 1959 bytes in 99 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) May 23 12:42:24.687848 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a32428f1-36ff-40bb-8aa2-99d01d25aeba None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:42:24.687848 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a32428f1-36ff-40bb-8aa2-99d01d25aeba None admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:42:24.687848 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a32428f1-36ff-40bb-8aa2-99d01d25aeba None admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:42:24.687848 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a32428f1-36ff-40bb-8aa2-99d01d25aeba None admin] PATH_INFO: `/v3/roles` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:42:24.688154 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a32428f1-36ff-40bb-8aa2-99d01d25aeba None admin] RBAC: Adding query filter params (name=ResellerAdmin) {{(pid=92056) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} May 23 12:42:24.688154 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a32428f1-36ff-40bb-8aa2-99d01d25aeba None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:42:24.688654 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a32428f1-36ff-40bb-8aa2-99d01d25aeba None admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:42:24.704655 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 515/1030] 10.222.0.30 () {56 vars in 1115 bytes} [Mon May 23 12:42:24 2022] GET /identity/v3/roles?name=ResellerAdmin => generated 338 bytes in 158 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) May 23 12:42:24.706819 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a32428f1-36ff-40bb-8aa2-99d01d25aeba None admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:42:24.707097 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a94928cf-07e9-43d7-ac61-ed8826ecb672 None None] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:42:24.772871 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=100 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:42:24.816314 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a32428f1-36ff-40bb-8aa2-99d01d25aeba None admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:42:24.818675 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a94928cf-07e9-43d7-ac61-ed8826ecb672 None None] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:42:24.852617 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-314af3ae-f81e-4d0d-905b-8e54402ced62 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:42:24.853366 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-314af3ae-f81e-4d0d-905b-8e54402ced62 None admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:42:24.853643 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-314af3ae-f81e-4d0d-905b-8e54402ced62 None admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:42:24.853808 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2281c37-bd91-4bf2-9737-52cf186b3667 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:42:24.854021 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-314af3ae-f81e-4d0d-905b-8e54402ced62 None admin] PATH_INFO: `/v3/services/image` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:42:24.854574 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-314af3ae-f81e-4d0d-905b-8e54402ced62 None admin] RBAC: Authorizing `identity:get_service(service_id=image)` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:42:24.854669 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2281c37-bd91-4bf2-9737-52cf186b3667 None admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:42:24.854857 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2281c37-bd91-4bf2-9737-52cf186b3667 None admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:42:24.855212 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2281c37-bd91-4bf2-9737-52cf186b3667 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:42:24.855936 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-314af3ae-f81e-4d0d-905b-8e54402ced62 None admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:42:24.866036 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f2281c37-bd91-4bf2-9737-52cf186b3667 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:42:24.867301 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application [None req-314af3ae-f81e-4d0d-905b-8e54402ced62 None admin] Could not find service: image.: keystone.exception.ServiceNotFound: Could not find service: image. May 23 12:42:24.867301 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application Traceback (most recent call last): May 23 12:42:24.867301 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service May 23 12:42:24.867301 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) May 23 12:42:24.867301 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service May 23 12:42:24.867301 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() May 23 12:42:24.867301 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service May 23 12:42:24.867301 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) May 23 12:42:24.867301 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: image. May 23 12:42:24.867301 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application May 23 12:42:24.867301 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: May 23 12:42:24.867301 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application May 23 12:42:24.867301 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application Traceback (most recent call last): May 23 12:42:24.867301 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request May 23 12:42:24.867301 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application rv = self.dispatch_request() May 23 12:42:24.867301 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request May 23 12:42:24.867301 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) May 23 12:42:24.868289 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper May 23 12:42:24.868289 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) May 23 12:42:24.868289 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view May 23 12:42:24.868289 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) May 23 12:42:24.868289 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request May 23 12:42:24.868289 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) May 23 12:42:24.868289 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get May 23 12:42:24.868289 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self._get_service(service_id) May 23 12:42:24.868289 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service May 23 12:42:24.868289 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) May 23 12:42:24.868289 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped May 23 12:42:24.868289 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) May 23 12:42:24.868289 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun May 23 12:42:24.868289 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) May 23 12:42:24.868289 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func May 23 12:42:24.868289 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self.get_or_create( May 23 12:42:24.869169 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create May 23 12:42:24.869169 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application with Lock( May 23 12:42:24.869169 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ May 23 12:42:24.869169 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self._enter() May 23 12:42:24.869169 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter May 23 12:42:24.869169 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) May 23 12:42:24.869169 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create May 23 12:42:24.869169 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self.creator() May 23 12:42:24.869169 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value May 23 12:42:24.869169 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application created_value = creator( May 23 12:42:24.869169 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service May 23 12:42:24.869169 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) May 23 12:42:24.869169 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: image. May 23 12:42:24.869169 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application May 23 12:42:24.869962 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f2281c37-bd91-4bf2-9737-52cf186b3667 None admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:42:24.870036 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 516/1031] 10.222.0.30 () {56 vars in 1105 bytes} [Mon May 23 12:42:24 2022] GET /identity/v3/services/image => generated 86 bytes in 163 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) May 23 12:42:24.874542 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-314af3ae-f81e-4d0d-905b-8e54402ced62 None admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:42:24.889615 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application [None req-f2281c37-bd91-4bf2-9737-52cf186b3667 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. May 23 12:42:24.889615 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application Traceback (most recent call last): May 23 12:42:24.889615 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request May 23 12:42:24.889615 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application rv = self.dispatch_request() May 23 12:42:24.889615 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request May 23 12:42:24.889615 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) May 23 12:42:24.889615 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper May 23 12:42:24.889615 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) May 23 12:42:24.889615 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view May 23 12:42:24.889615 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) May 23 12:42:24.889615 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request May 23 12:42:24.889615 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) May 23 12:42:24.889615 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get May 23 12:42:24.889615 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self._get_domain(domain_id) May 23 12:42:24.889615 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain May 23 12:42:24.890753 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) May 23 12:42:24.890753 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped May 23 12:42:24.890753 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) May 23 12:42:24.890753 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun May 23 12:42:24.890753 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) May 23 12:42:24.890753 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func May 23 12:42:24.890753 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self.get_or_create( May 23 12:42:24.890753 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create May 23 12:42:24.890753 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application with Lock( May 23 12:42:24.890753 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ May 23 12:42:24.890753 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self._enter() May 23 12:42:24.890753 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter May 23 12:42:24.890753 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) May 23 12:42:24.890753 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create May 23 12:42:24.890753 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self.creator() May 23 12:42:24.890753 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value May 23 12:42:24.890753 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application created_value = creator( May 23 12:42:24.891707 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain May 23 12:42:24.891707 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) May 23 12:42:24.891707 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. May 23 12:42:24.891707 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application May 23 12:42:24.891707 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 516/1032] 10.222.0.30 () {56 vars in 1108 bytes} [Mon May 23 12:42:24 2022] GET /identity/v3/domains/Default => generated 87 bytes in 185 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) May 23 12:42:24.897250 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2281c37-bd91-4bf2-9737-52cf186b3667 None admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:42:24.945799 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=100 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:42:24.988040 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-314af3ae-f81e-4d0d-905b-8e54402ced62 None admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:42:25.006704 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2281c37-bd91-4bf2-9737-52cf186b3667 None admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:42:25.011969 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=100 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:42:25.024079 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e55ed32d-6c3a-4c3d-a003-00995b5626aa None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:42:25.024235 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e55ed32d-6c3a-4c3d-a003-00995b5626aa None admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:42:25.024528 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e55ed32d-6c3a-4c3d-a003-00995b5626aa None admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:42:25.024705 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e55ed32d-6c3a-4c3d-a003-00995b5626aa None admin] PATH_INFO: `/v3/services` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:42:25.025674 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e55ed32d-6c3a-4c3d-a003-00995b5626aa None admin] RBAC: Adding query filter params (name=image) {{(pid=92056) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} May 23 12:42:25.026007 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e55ed32d-6c3a-4c3d-a003-00995b5626aa None admin] RBAC: Authorizing `identity:list_services()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:42:25.027378 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e55ed32d-6c3a-4c3d-a003-00995b5626aa None admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:42:25.032194 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 517/1033] 10.222.0.30 () {56 vars in 1108 bytes} [Mon May 23 12:42:24 2022] GET /identity/v3/services?name=image => generated 122 bytes in 158 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) May 23 12:42:25.038063 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e55ed32d-6c3a-4c3d-a003-00995b5626aa None admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:42:25.053865 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84ca050e-874c-4c6f-9f22-56adc2e6085f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:42:25.054821 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84ca050e-874c-4c6f-9f22-56adc2e6085f None admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:42:25.055529 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84ca050e-874c-4c6f-9f22-56adc2e6085f None admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:42:25.056498 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84ca050e-874c-4c6f-9f22-56adc2e6085f None admin] PATH_INFO: `/v3/domains` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:42:25.057708 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-84ca050e-874c-4c6f-9f22-56adc2e6085f None admin] RBAC: Adding query filter params (name=Default) {{(pid=92057) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} May 23 12:42:25.058190 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-84ca050e-874c-4c6f-9f22-56adc2e6085f None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:42:25.060371 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-84ca050e-874c-4c6f-9f22-56adc2e6085f None admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:42:25.078283 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 517/1034] 10.222.0.30 () {56 vars in 1109 bytes} [Mon May 23 12:42:24 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 177 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) May 23 12:42:25.079383 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84ca050e-874c-4c6f-9f22-56adc2e6085f None admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:42:25.155495 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e55ed32d-6c3a-4c3d-a003-00995b5626aa None admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:42:25.191562 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84ca050e-874c-4c6f-9f22-56adc2e6085f None admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:42:25.194469 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-836fa98e-236e-4152-a1a6-0bbdfc8f5203 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:42:25.195139 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-836fa98e-236e-4152-a1a6-0bbdfc8f5203 None admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:42:25.195242 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=100 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:42:25.195430 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-836fa98e-236e-4152-a1a6-0bbdfc8f5203 None admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:42:25.195715 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-836fa98e-236e-4152-a1a6-0bbdfc8f5203 None admin] PATH_INFO: `/v3/services` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:42:25.196417 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-836fa98e-236e-4152-a1a6-0bbdfc8f5203 None admin] RBAC: Adding query filter params (type=image) {{(pid=92056) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} May 23 12:42:25.196728 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-836fa98e-236e-4152-a1a6-0bbdfc8f5203 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:42:25.198149 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-836fa98e-236e-4152-a1a6-0bbdfc8f5203 None admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:42:25.208163 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 518/1035] 10.222.0.30 () {56 vars in 1108 bytes} [Mon May 23 12:42:25 2022] GET /identity/v3/services?type=image => generated 350 bytes in 166 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) May 23 12:42:25.209999 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-836fa98e-236e-4152-a1a6-0bbdfc8f5203 None admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:42:25.227485 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd6efa76-7a4c-4ea8-a22a-6943cdd75778 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:42:25.228275 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd6efa76-7a4c-4ea8-a22a-6943cdd75778 None admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:42:25.228562 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd6efa76-7a4c-4ea8-a22a-6943cdd75778 None admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:42:25.228843 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd6efa76-7a4c-4ea8-a22a-6943cdd75778 None admin] PATH_INFO: `/v3/users/nova` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:42:25.237359 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd6efa76-7a4c-4ea8-a22a-6943cdd75778 None admin] RBAC: Authorizing `identity:get_user(user_id=nova)` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:42:25.239005 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd6efa76-7a4c-4ea8-a22a-6943cdd75778 None admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:42:25.255280 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application [None req-fd6efa76-7a4c-4ea8-a22a-6943cdd75778 None admin] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova. May 23 12:42:25.255280 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application Traceback (most recent call last): May 23 12:42:25.255280 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request May 23 12:42:25.255280 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application rv = self.dispatch_request() May 23 12:42:25.255280 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request May 23 12:42:25.255280 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) May 23 12:42:25.255280 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper May 23 12:42:25.255280 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) May 23 12:42:25.255280 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view May 23 12:42:25.255280 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) May 23 12:42:25.255280 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request May 23 12:42:25.255280 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) May 23 12:42:25.255280 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get May 23 12:42:25.255280 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self._get_user(user_id) May 23 12:42:25.255280 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user May 23 12:42:25.255280 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) May 23 12:42:25.256296 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped May 23 12:42:25.256296 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) May 23 12:42:25.256296 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper May 23 12:42:25.256296 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) May 23 12:42:25.256296 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper May 23 12:42:25.256296 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) May 23 12:42:25.256296 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun May 23 12:42:25.256296 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) May 23 12:42:25.256296 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func May 23 12:42:25.256296 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self.get_or_create( May 23 12:42:25.256296 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create May 23 12:42:25.256296 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application with Lock( May 23 12:42:25.256296 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ May 23 12:42:25.256296 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self._enter() May 23 12:42:25.256296 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter May 23 12:42:25.256296 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) May 23 12:42:25.256296 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create May 23 12:42:25.256296 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self.creator() May 23 12:42:25.258106 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value May 23 12:42:25.258106 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application created_value = creator( May 23 12:42:25.258106 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user May 23 12:42:25.258106 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) May 23 12:42:25.258106 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user May 23 12:42:25.258106 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) May 23 12:42:25.258106 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user May 23 12:42:25.258106 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) May 23 12:42:25.258106 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: nova. May 23 12:42:25.258106 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application May 23 12:42:25.258106 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 518/1036] 10.222.0.30 () {56 vars in 1093 bytes} [Mon May 23 12:42:25 2022] GET /identity/v3/users/nova => generated 82 bytes in 178 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) May 23 12:42:25.262116 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd6efa76-7a4c-4ea8-a22a-6943cdd75778 None admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:42:25.265416 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=100 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:42:25.320331 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-836fa98e-236e-4152-a1a6-0bbdfc8f5203 None admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:42:25.355982 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-332058e3-42c2-4841-9b50-060c467153e3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:42:25.356670 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-332058e3-42c2-4841-9b50-060c467153e3 None admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:42:25.356920 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-332058e3-42c2-4841-9b50-060c467153e3 None admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:42:25.357197 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-332058e3-42c2-4841-9b50-060c467153e3 None admin] PATH_INFO: `/v3/endpoints` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:42:25.358075 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-332058e3-42c2-4841-9b50-060c467153e3 None admin] RBAC: Adding query filter params (interface=public, service_id=fb704c307bcc4a74a0dcb21e885d3906, region_id=RegionOne) {{(pid=92056) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} May 23 12:42:25.358364 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-332058e3-42c2-4841-9b50-060c467153e3 None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:42:25.359743 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-332058e3-42c2-4841-9b50-060c467153e3 None admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:42:25.365597 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 519/1037] 10.222.0.30 () {56 vars in 1251 bytes} [Mon May 23 12:42:25 2022] GET /identity/v3/endpoints?service_id=fb704c307bcc4a74a0dcb21e885d3906&interface=public®ion_id=RegionOne => generated 194 bytes in 156 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) May 23 12:42:25.371016 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-332058e3-42c2-4841-9b50-060c467153e3 None admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:42:25.371368 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd6efa76-7a4c-4ea8-a22a-6943cdd75778 None admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:42:25.406498 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6953f41-9e9a-459b-9b10-cf289d42f6a4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:42:25.407188 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6953f41-9e9a-459b-9b10-cf289d42f6a4 None admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:42:25.407471 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6953f41-9e9a-459b-9b10-cf289d42f6a4 None admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:42:25.407752 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6953f41-9e9a-459b-9b10-cf289d42f6a4 None admin] PATH_INFO: `/v3/users` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:42:25.409126 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6953f41-9e9a-459b-9b10-cf289d42f6a4 None admin] RBAC: Adding query filter params (domain_id=default, name=nova) {{(pid=92057) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} May 23 12:42:25.409589 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6953f41-9e9a-459b-9b10-cf289d42f6a4 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:42:25.411240 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6953f41-9e9a-459b-9b10-cf289d42f6a4 None admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:42:25.429220 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 519/1038] 10.222.0.30 () {56 vars in 1133 bytes} [Mon May 23 12:42:25 2022] GET /identity/v3/users?domain_id=default&name=nova => generated 368 bytes in 167 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) May 23 12:42:25.434807 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6953f41-9e9a-459b-9b10-cf289d42f6a4 None admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:42:25.459829 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=100 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:42:25.476721 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-332058e3-42c2-4841-9b50-060c467153e3 None admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:42:25.491107 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=100 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:42:25.512744 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-963b700c-f6bd-48c0-a737-283ef783ca46 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:42:25.513416 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-963b700c-f6bd-48c0-a737-283ef783ca46 None admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:42:25.513700 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-963b700c-f6bd-48c0-a737-283ef783ca46 None admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:42:25.514008 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-963b700c-f6bd-48c0-a737-283ef783ca46 None admin] PATH_INFO: `/v3/services` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:42:25.514695 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-963b700c-f6bd-48c0-a737-283ef783ca46 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:42:25.517061 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-963b700c-f6bd-48c0-a737-283ef783ca46 None admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:42:25.526503 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 520/1039] 10.222.0.30 () {56 vars in 1087 bytes} [Mon May 23 12:42:25 2022] GET /identity/v3/services => generated 1957 bytes in 156 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) May 23 12:42:25.553900 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6953f41-9e9a-459b-9b10-cf289d42f6a4 None admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:42:25.593992 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05eb7c52-3ac9-4921-a558-8b79d90b4cd5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:42:25.594646 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05eb7c52-3ac9-4921-a558-8b79d90b4cd5 None admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:42:25.594935 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05eb7c52-3ac9-4921-a558-8b79d90b4cd5 None admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:42:25.595212 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05eb7c52-3ac9-4921-a558-8b79d90b4cd5 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:42:25.671823 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05eb7c52-3ac9-4921-a558-8b79d90b4cd5 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:42:25.673294 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05eb7c52-3ac9-4921-a558-8b79d90b4cd5 None admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:42:25.706127 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=100 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:42:25.790012 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 520/1040] 10.222.0.30 () {58 vars in 1282 bytes} [Mon May 23 12:42:25 2022] GET /identity/v3/auth/tokens => generated 1959 bytes in 356 msecs (HTTP/1.1 200) 6 headers in 359 bytes (2 switches on core 0) May 23 12:42:25.795960 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-963b700c-f6bd-48c0-a737-283ef783ca46 None admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:42:25.900207 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-963b700c-f6bd-48c0-a737-283ef783ca46 None admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:42:25.934555 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ccd593b4-1ca9-486e-b389-5d5177d66801 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:42:25.935353 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccd593b4-1ca9-486e-b389-5d5177d66801 None admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:42:25.935638 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccd593b4-1ca9-486e-b389-5d5177d66801 None admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:42:25.935917 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccd593b4-1ca9-486e-b389-5d5177d66801 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:42:25.947182 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ccd593b4-1ca9-486e-b389-5d5177d66801 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:42:25.948635 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ccd593b4-1ca9-486e-b389-5d5177d66801 None admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:42:25.966296 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application [None req-ccd593b4-1ca9-486e-b389-5d5177d66801 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. May 23 12:42:25.966296 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application Traceback (most recent call last): May 23 12:42:25.966296 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request May 23 12:42:25.966296 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application rv = self.dispatch_request() May 23 12:42:25.966296 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request May 23 12:42:25.966296 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) May 23 12:42:25.966296 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper May 23 12:42:25.966296 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) May 23 12:42:25.966296 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view May 23 12:42:25.966296 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) May 23 12:42:25.966296 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request May 23 12:42:25.966296 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) May 23 12:42:25.966296 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get May 23 12:42:25.966296 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self._get_domain(domain_id) May 23 12:42:25.966296 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain May 23 12:42:25.967298 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) May 23 12:42:25.967298 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped May 23 12:42:25.967298 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) May 23 12:42:25.967298 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun May 23 12:42:25.967298 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) May 23 12:42:25.967298 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func May 23 12:42:25.967298 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self.get_or_create( May 23 12:42:25.967298 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create May 23 12:42:25.967298 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application with Lock( May 23 12:42:25.967298 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ May 23 12:42:25.967298 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self._enter() May 23 12:42:25.967298 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter May 23 12:42:25.967298 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) May 23 12:42:25.967298 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create May 23 12:42:25.967298 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self.creator() May 23 12:42:25.967298 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value May 23 12:42:25.967298 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application created_value = creator( May 23 12:42:25.968253 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain May 23 12:42:25.968253 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) May 23 12:42:25.968253 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. May 23 12:42:25.968253 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application May 23 12:42:25.969980 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 521/1041] 10.222.0.30 () {56 vars in 1108 bytes} [Mon May 23 12:42:25 2022] GET /identity/v3/domains/Default => generated 87 bytes in 172 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) May 23 12:42:25.973725 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05eb7c52-3ac9-4921-a558-8b79d90b4cd5 None admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:42:26.079309 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05eb7c52-3ac9-4921-a558-8b79d90b4cd5 None admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:42:26.103395 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=100 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:42:26.115284 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4d4e563-57fa-474f-99c3-d1c304717afc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:42:26.116187 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4d4e563-57fa-474f-99c3-d1c304717afc None admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:42:26.116650 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4d4e563-57fa-474f-99c3-d1c304717afc None admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:42:26.117102 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4d4e563-57fa-474f-99c3-d1c304717afc None admin] PATH_INFO: `/v3/domains` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:42:26.118052 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4d4e563-57fa-474f-99c3-d1c304717afc None admin] RBAC: Adding query filter params (name=Default) {{(pid=92057) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} May 23 12:42:26.118517 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4d4e563-57fa-474f-99c3-d1c304717afc None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:42:26.120079 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4d4e563-57fa-474f-99c3-d1c304717afc None admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:42:26.132828 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 521/1042] 10.222.0.30 () {56 vars in 1109 bytes} [Mon May 23 12:42:25 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 160 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) May 23 12:42:26.138121 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ccd593b4-1ca9-486e-b389-5d5177d66801 None admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:42:26.161917 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85f8a692-7612-4043-9bab-dce73abc08a2 None None] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:42:26.162477 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85f8a692-7612-4043-9bab-dce73abc08a2 None None] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:42:26.162477 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85f8a692-7612-4043-9bab-dce73abc08a2 None None] PATH_INFO: `/` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:42:26.166089 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 522/1043] 10.222.0.30 () {54 vars in 926 bytes} [Mon May 23 12:42:26 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) May 23 12:42:26.168630 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bfa24f7-bc1d-4d28-8a63-6c28ba6059f3 None None] REQUEST_METHOD: `POST` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:42:26.168919 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bfa24f7-bc1d-4d28-8a63-6c28ba6059f3 None None] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:42:26.169274 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bfa24f7-bc1d-4d28-8a63-6c28ba6059f3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:42:26.172448 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=100 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:42:26.185717 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.auth.core [None req-9bfa24f7-bc1d-4d28-8a63-6c28ba6059f3 None None] MFA Rules not processed for user `0f7119f8fa5c4445977671140ded91a4`. Rule list: `[]` (Enabled: `True`). {{(pid=92057) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 12:42:26.186934 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.fernet_utils [None req-9bfa24f7-bc1d-4d28-8a63-6c28ba6059f3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92057) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 12:42:26.246166 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ccd593b4-1ca9-486e-b389-5d5177d66801 None admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:42:26.265535 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 523/1044] 10.222.0.30 () {58 vars in 1023 bytes} [Mon May 23 12:42:26 2022] POST /identity/v3/auth/tokens => generated 1959 bytes in 97 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) May 23 12:42:26.283429 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f72ac122-31de-483f-8456-57517be4ce9b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:42:26.284325 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f72ac122-31de-483f-8456-57517be4ce9b None admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:42:26.284611 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f72ac122-31de-483f-8456-57517be4ce9b None admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:42:26.284892 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f72ac122-31de-483f-8456-57517be4ce9b None admin] PATH_INFO: `/v3/projects/service` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:42:26.289748 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9bfa24f7-bc1d-4d28-8a63-6c28ba6059f3 None None] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:42:26.291941 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f72ac122-31de-483f-8456-57517be4ce9b None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:42:26.293413 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f72ac122-31de-483f-8456-57517be4ce9b None admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:42:26.308495 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application [None req-f72ac122-31de-483f-8456-57517be4ce9b None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. May 23 12:42:26.308495 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application Traceback (most recent call last): May 23 12:42:26.308495 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request May 23 12:42:26.308495 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application rv = self.dispatch_request() May 23 12:42:26.308495 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request May 23 12:42:26.308495 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) May 23 12:42:26.308495 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper May 23 12:42:26.308495 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) May 23 12:42:26.308495 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view May 23 12:42:26.308495 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) May 23 12:42:26.308495 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request May 23 12:42:26.308495 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) May 23 12:42:26.308495 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get May 23 12:42:26.308495 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self._get_project(project_id) May 23 12:42:26.308495 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project May 23 12:42:26.309449 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) May 23 12:42:26.309449 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped May 23 12:42:26.309449 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) May 23 12:42:26.309449 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun May 23 12:42:26.309449 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) May 23 12:42:26.309449 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func May 23 12:42:26.309449 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self.get_or_create( May 23 12:42:26.309449 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create May 23 12:42:26.309449 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application with Lock( May 23 12:42:26.309449 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ May 23 12:42:26.309449 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self._enter() May 23 12:42:26.309449 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter May 23 12:42:26.309449 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) May 23 12:42:26.309449 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create May 23 12:42:26.309449 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self.creator() May 23 12:42:26.309449 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value May 23 12:42:26.309449 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application created_value = creator( May 23 12:42:26.316393 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project May 23 12:42:26.316393 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) May 23 12:42:26.316393 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project May 23 12:42:26.316393 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() May 23 12:42:26.316393 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project May 23 12:42:26.316393 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) May 23 12:42:26.316393 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. May 23 12:42:26.316393 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application May 23 12:42:26.316393 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 522/1045] 10.222.0.30 () {56 vars in 1111 bytes} [Mon May 23 12:42:26 2022] GET /identity/v3/projects/service => generated 88 bytes in 172 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) May 23 12:42:26.316393 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f72ac122-31de-483f-8456-57517be4ce9b None admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:42:26.404173 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9bfa24f7-bc1d-4d28-8a63-6c28ba6059f3 None None] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:42:26.412276 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=100 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:42:26.419269 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=100 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:42:26.422408 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f72ac122-31de-483f-8456-57517be4ce9b None admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:42:26.440569 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a03cd82-d80f-431b-ad1a-40d9c34c75a7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:42:26.441357 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a03cd82-d80f-431b-ad1a-40d9c34c75a7 None admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:42:26.441639 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a03cd82-d80f-431b-ad1a-40d9c34c75a7 None admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:42:26.441943 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a03cd82-d80f-431b-ad1a-40d9c34c75a7 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:42:26.453054 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a03cd82-d80f-431b-ad1a-40d9c34c75a7 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} May 23 12:42:26.459168 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70a7cd4d-0cc1-42ae-afb9-7be280dba27d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:42:26.459830 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70a7cd4d-0cc1-42ae-afb9-7be280dba27d None admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:42:26.460116 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70a7cd4d-0cc1-42ae-afb9-7be280dba27d None admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:42:26.460566 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70a7cd4d-0cc1-42ae-afb9-7be280dba27d None admin] PATH_INFO: `/v3/projects` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:42:26.461364 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70a7cd4d-0cc1-42ae-afb9-7be280dba27d None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=92056) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} May 23 12:42:26.461678 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70a7cd4d-0cc1-42ae-afb9-7be280dba27d None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:42:26.461789 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application [None req-6a03cd82-d80f-431b-ad1a-40d9c34c75a7 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. May 23 12:42:26.461789 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application Traceback (most recent call last): May 23 12:42:26.461789 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request May 23 12:42:26.461789 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application rv = self.dispatch_request() May 23 12:42:26.461789 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request May 23 12:42:26.461789 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) May 23 12:42:26.461789 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper May 23 12:42:26.461789 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) May 23 12:42:26.461789 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view May 23 12:42:26.461789 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) May 23 12:42:26.461789 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request May 23 12:42:26.461789 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) May 23 12:42:26.461789 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get May 23 12:42:26.461789 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self._get_role(role_id) May 23 12:42:26.461789 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role May 23 12:42:26.461789 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') May 23 12:42:26.462711 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call May 23 12:42:26.462711 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( May 23 12:42:26.462711 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data May 23 12:42:26.462711 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) May 23 12:42:26.462711 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped May 23 12:42:26.462711 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) May 23 12:42:26.462711 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun May 23 12:42:26.462711 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) May 23 12:42:26.462711 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func May 23 12:42:26.462711 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self.get_or_create( May 23 12:42:26.462711 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create May 23 12:42:26.462711 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application with Lock( May 23 12:42:26.462711 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ May 23 12:42:26.462711 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self._enter() May 23 12:42:26.462711 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter May 23 12:42:26.462711 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) May 23 12:42:26.466936 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create May 23 12:42:26.466936 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self.creator() May 23 12:42:26.466936 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value May 23 12:42:26.466936 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application created_value = creator( May 23 12:42:26.466936 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role May 23 12:42:26.466936 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) May 23 12:42:26.466936 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role May 23 12:42:26.466936 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() May 23 12:42:26.466936 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role May 23 12:42:26.466936 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) May 23 12:42:26.466936 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. May 23 12:42:26.466936 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application May 23 12:42:26.467657 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70a7cd4d-0cc1-42ae-afb9-7be280dba27d None admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:42:26.467732 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 524/1046] 10.222.0.30 () {56 vars in 1096 bytes} [Mon May 23 12:42:26 2022] GET /identity/v3/roles/admin => generated 83 bytes in 174 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) May 23 12:42:26.468749 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a03cd82-d80f-431b-ad1a-40d9c34c75a7 None admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:42:26.482982 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 523/1047] 10.222.0.30 () {56 vars in 1148 bytes} [Mon May 23 12:42:26 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 163 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) May 23 12:42:26.484004 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70a7cd4d-0cc1-42ae-afb9-7be280dba27d None admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:42:26.580129 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a03cd82-d80f-431b-ad1a-40d9c34c75a7 None admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:42:26.600455 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70a7cd4d-0cc1-42ae-afb9-7be280dba27d None admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:42:26.615603 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-abc480c1-f063-49f4-96b8-b3ea8869172a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:42:26.616349 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abc480c1-f063-49f4-96b8-b3ea8869172a None admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:42:26.616636 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abc480c1-f063-49f4-96b8-b3ea8869172a None admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:42:26.616913 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abc480c1-f063-49f4-96b8-b3ea8869172a None admin] PATH_INFO: `/v3/roles` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:42:26.617650 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-abc480c1-f063-49f4-96b8-b3ea8869172a None admin] RBAC: Adding query filter params (name=admin) {{(pid=92057) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} May 23 12:42:26.617983 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-abc480c1-f063-49f4-96b8-b3ea8869172a None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:42:26.619386 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-abc480c1-f063-49f4-96b8-b3ea8869172a None admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:42:26.628229 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 525/1048] 10.222.0.30 () {56 vars in 1099 bytes} [Mon May 23 12:42:26 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 160 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) May 23 12:42:26.634333 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-abc480c1-f063-49f4-96b8-b3ea8869172a None admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:42:26.641386 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ecf0b44-1023-4921-afba-65011b3100c0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:42:26.643099 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ecf0b44-1023-4921-afba-65011b3100c0 None admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:42:26.643561 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ecf0b44-1023-4921-afba-65011b3100c0 None admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:42:26.643989 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ecf0b44-1023-4921-afba-65011b3100c0 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:42:26.644966 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ecf0b44-1023-4921-afba-65011b3100c0 None admin] RBAC: Adding query filter params (role.id=940396c67e6744c784e95e01fe84ad45, scope.project.id=9c389ae65aaa4542bb5a5c4fd0bacf5b, user.id=cbe9931d94d44cc7ace0e83a7e5015fc) {{(pid=92056) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} May 23 12:42:26.645429 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ecf0b44-1023-4921-afba-65011b3100c0 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:42:26.647051 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ecf0b44-1023-4921-afba-65011b3100c0 None admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:42:26.653989 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 524/1049] 10.222.0.30 () {56 vars in 1374 bytes} [Mon May 23 12:42:26 2022] GET /identity/v3/role_assignments?user.id=cbe9931d94d44cc7ace0e83a7e5015fc&scope.project.id=9c389ae65aaa4542bb5a5c4fd0bacf5b&role.id=940396c67e6744c784e95e01fe84ad45 => generated 259 bytes in 170 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) May 23 12:42:26.688849 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=100 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:42:26.762445 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-abc480c1-f063-49f4-96b8-b3ea8869172a None admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:42:26.801992 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f95f810-333d-432c-b621-1aab511e2510 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:42:26.802891 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f95f810-333d-432c-b621-1aab511e2510 None admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:42:26.803175 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f95f810-333d-432c-b621-1aab511e2510 None admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:42:26.803450 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f95f810-333d-432c-b621-1aab511e2510 None admin] PATH_INFO: `/v3/users/5eb771149c44430c8ee4cd2cd1e23890` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:42:26.819308 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f95f810-333d-432c-b621-1aab511e2510 None admin] RBAC: Authorizing `identity:get_user(user_id=5eb771149c44430c8ee4cd2cd1e23890)` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:42:26.820797 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f95f810-333d-432c-b621-1aab511e2510 None admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:42:26.822813 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 526/1050] 10.222.0.30 () {56 vars in 1177 bytes} [Mon May 23 12:42:26 2022] GET /identity/v3/users/5eb771149c44430c8ee4cd2cd1e23890 => generated 285 bytes in 189 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) May 23 12:42:26.828523 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ecf0b44-1023-4921-afba-65011b3100c0 None admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:42:26.847842 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=100 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:42:26.934030 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ecf0b44-1023-4921-afba-65011b3100c0 None admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:42:26.967311 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c70d8a9b-e066-4ea7-8d37-246d88e700de None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:42:26.967869 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c70d8a9b-e066-4ea7-8d37-246d88e700de None admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:42:26.968150 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c70d8a9b-e066-4ea7-8d37-246d88e700de None admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:42:26.968429 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c70d8a9b-e066-4ea7-8d37-246d88e700de None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:42:27.036062 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c70d8a9b-e066-4ea7-8d37-246d88e700de None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:42:27.037524 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c70d8a9b-e066-4ea7-8d37-246d88e700de None admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:42:27.053207 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=100 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:42:27.142716 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 525/1051] 10.222.0.30 () {58 vars in 1282 bytes} [Mon May 23 12:42:26 2022] GET /identity/v3/auth/tokens => generated 1959 bytes in 310 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) May 23 12:42:27.143937 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f95f810-333d-432c-b621-1aab511e2510 None admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:42:27.237241 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=100 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:42:27.246385 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f95f810-333d-432c-b621-1aab511e2510 None admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:42:27.280734 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb7d7e81-d4a7-4962-8777-feea78dd0e8b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:42:27.281587 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb7d7e81-d4a7-4962-8777-feea78dd0e8b None admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:42:27.281867 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb7d7e81-d4a7-4962-8777-feea78dd0e8b None admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:42:27.282166 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb7d7e81-d4a7-4962-8777-feea78dd0e8b None admin] PATH_INFO: `/v3/projects/c258ee56e55945e7b3182d1d20762347` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:42:27.283339 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb7d7e81-d4a7-4962-8777-feea78dd0e8b None admin] RBAC: Authorizing `identity:get_project(project_id=c258ee56e55945e7b3182d1d20762347)` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:42:27.285455 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb7d7e81-d4a7-4962-8777-feea78dd0e8b None admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:42:27.287337 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 527/1052] 10.222.0.30 () {56 vars in 1186 bytes} [Mon May 23 12:42:27 2022] GET /identity/v3/projects/c258ee56e55945e7b3182d1d20762347 => generated 311 bytes in 144 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) May 23 12:42:27.293109 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c70d8a9b-e066-4ea7-8d37-246d88e700de None admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:42:27.394124 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c70d8a9b-e066-4ea7-8d37-246d88e700de None admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:42:27.409567 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=100 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:42:27.430249 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ddb4dba-55f9-476c-99af-b0db42f1797a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:42:27.430907 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ddb4dba-55f9-476c-99af-b0db42f1797a None admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:42:27.431170 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ddb4dba-55f9-476c-99af-b0db42f1797a None admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:42:27.431445 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ddb4dba-55f9-476c-99af-b0db42f1797a None admin] PATH_INFO: `/v3/role_assignments` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:42:27.432269 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ddb4dba-55f9-476c-99af-b0db42f1797a None admin] RBAC: Adding query filter params (role.id=bc3c3c9d654940cbaa5177a6be23d1bc, scope.project.id=c258ee56e55945e7b3182d1d20762347, user.id=5eb771149c44430c8ee4cd2cd1e23890) {{(pid=92056) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} May 23 12:42:27.432579 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ddb4dba-55f9-476c-99af-b0db42f1797a None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:42:27.434026 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ddb4dba-55f9-476c-99af-b0db42f1797a None admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:42:27.439666 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 526/1053] 10.222.0.30 () {56 vars in 1374 bytes} [Mon May 23 12:42:27 2022] GET /identity/v3/role_assignments?user.id=5eb771149c44430c8ee4cd2cd1e23890&scope.project.id=c258ee56e55945e7b3182d1d20762347&role.id=bc3c3c9d654940cbaa5177a6be23d1bc => generated 259 bytes in 147 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) May 23 12:42:27.819916 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53a6c994-b929-48b8-ad7e-9166cc1c1b19 None None] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:42:27.820421 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53a6c994-b929-48b8-ad7e-9166cc1c1b19 None None] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:42:27.820514 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53a6c994-b929-48b8-ad7e-9166cc1c1b19 None None] PATH_INFO: `/` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:42:27.822003 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 528/1054] 10.222.0.30 () {54 vars in 926 bytes} [Mon May 23 12:42:27 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) May 23 12:42:27.826966 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-befec708-1e5e-4bdd-a32d-5f5aafb58a23 None None] REQUEST_METHOD: `POST` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:42:27.827259 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-befec708-1e5e-4bdd-a32d-5f5aafb58a23 None None] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:42:27.827532 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-befec708-1e5e-4bdd-a32d-5f5aafb58a23 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:42:27.845111 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.auth.core [None req-befec708-1e5e-4bdd-a32d-5f5aafb58a23 None None] MFA Rules not processed for user `0f7119f8fa5c4445977671140ded91a4`. Rule list: `[]` (Enabled: `True`). {{(pid=92056) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 12:42:27.846515 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.fernet_utils [None req-befec708-1e5e-4bdd-a32d-5f5aafb58a23 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92056) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 12:42:27.920582 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 527/1055] 10.222.0.30 () {58 vars in 1023 bytes} [Mon May 23 12:42:27 2022] POST /identity/v3/auth/tokens => generated 1959 bytes in 94 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) May 23 12:42:27.946612 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53a6c994-b929-48b8-ad7e-9166cc1c1b19 None None] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:42:28.058896 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53a6c994-b929-48b8-ad7e-9166cc1c1b19 None None] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:42:28.094795 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-812875e5-ed2f-484b-9ba9-a1cfbc352784 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:42:28.095582 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-812875e5-ed2f-484b-9ba9-a1cfbc352784 None admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:42:28.095909 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-812875e5-ed2f-484b-9ba9-a1cfbc352784 None admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:42:28.097228 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-812875e5-ed2f-484b-9ba9-a1cfbc352784 None admin] PATH_INFO: `/v3/services/image` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:42:28.097898 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-812875e5-ed2f-484b-9ba9-a1cfbc352784 None admin] RBAC: Authorizing `identity:get_service(service_id=image)` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:42:28.099285 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-812875e5-ed2f-484b-9ba9-a1cfbc352784 None admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:42:28.110817 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application [None req-812875e5-ed2f-484b-9ba9-a1cfbc352784 None admin] Could not find service: image.: keystone.exception.ServiceNotFound: Could not find service: image. May 23 12:42:28.110817 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application Traceback (most recent call last): May 23 12:42:28.110817 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service May 23 12:42:28.110817 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) May 23 12:42:28.110817 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service May 23 12:42:28.110817 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() May 23 12:42:28.110817 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service May 23 12:42:28.110817 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) May 23 12:42:28.110817 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: image. May 23 12:42:28.110817 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application May 23 12:42:28.110817 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: May 23 12:42:28.110817 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application May 23 12:42:28.110817 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application Traceback (most recent call last): May 23 12:42:28.110817 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request May 23 12:42:28.110817 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application rv = self.dispatch_request() May 23 12:42:28.110817 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request May 23 12:42:28.110817 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) May 23 12:42:28.112052 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper May 23 12:42:28.112052 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) May 23 12:42:28.112052 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view May 23 12:42:28.112052 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) May 23 12:42:28.112052 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request May 23 12:42:28.112052 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) May 23 12:42:28.112052 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get May 23 12:42:28.112052 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self._get_service(service_id) May 23 12:42:28.112052 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service May 23 12:42:28.112052 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) May 23 12:42:28.112052 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped May 23 12:42:28.112052 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) May 23 12:42:28.112052 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun May 23 12:42:28.112052 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) May 23 12:42:28.112052 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func May 23 12:42:28.112052 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self.get_or_create( May 23 12:42:28.118482 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create May 23 12:42:28.118482 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application with Lock( May 23 12:42:28.118482 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ May 23 12:42:28.118482 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self._enter() May 23 12:42:28.118482 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter May 23 12:42:28.118482 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) May 23 12:42:28.118482 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create May 23 12:42:28.118482 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self.creator() May 23 12:42:28.118482 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value May 23 12:42:28.118482 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application created_value = creator( May 23 12:42:28.118482 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service May 23 12:42:28.118482 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) May 23 12:42:28.118482 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: image. May 23 12:42:28.118482 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application May 23 12:42:28.118482 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 529/1056] 10.222.0.30 () {56 vars in 1105 bytes} [Mon May 23 12:42:27 2022] GET /identity/v3/services/image => generated 86 bytes in 168 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) May 23 12:42:28.119395 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-befec708-1e5e-4bdd-a32d-5f5aafb58a23 None None] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:42:28.228743 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-befec708-1e5e-4bdd-a32d-5f5aafb58a23 None None] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:42:28.238262 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=100 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:42:28.263071 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7436754f-d40c-4bd2-a52f-8112a8f9805d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:42:28.263777 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7436754f-d40c-4bd2-a52f-8112a8f9805d None admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:42:28.264057 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7436754f-d40c-4bd2-a52f-8112a8f9805d None admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:42:28.264335 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7436754f-d40c-4bd2-a52f-8112a8f9805d None admin] PATH_INFO: `/v3/services` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:42:28.265034 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7436754f-d40c-4bd2-a52f-8112a8f9805d None admin] RBAC: Adding query filter params (name=image) {{(pid=92056) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} May 23 12:42:28.265342 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7436754f-d40c-4bd2-a52f-8112a8f9805d None admin] RBAC: Authorizing `identity:list_services()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:42:28.266761 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7436754f-d40c-4bd2-a52f-8112a8f9805d None admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:42:28.277167 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 528/1057] 10.222.0.30 () {56 vars in 1108 bytes} [Mon May 23 12:42:28 2022] GET /identity/v3/services?name=image => generated 122 bytes in 154 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) May 23 12:42:28.278409 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-812875e5-ed2f-484b-9ba9-a1cfbc352784 None admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:42:28.304667 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=100 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:42:28.380032 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-812875e5-ed2f-484b-9ba9-a1cfbc352784 None admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:42:28.413725 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f31c5f64-e792-4775-886f-32ff2a4ca444 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:42:28.414414 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f31c5f64-e792-4775-886f-32ff2a4ca444 None admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:42:28.414698 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f31c5f64-e792-4775-886f-32ff2a4ca444 None admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:42:28.414970 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f31c5f64-e792-4775-886f-32ff2a4ca444 None admin] PATH_INFO: `/v3/services` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:42:28.415660 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f31c5f64-e792-4775-886f-32ff2a4ca444 None admin] RBAC: Adding query filter params (type=image) {{(pid=92057) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} May 23 12:42:28.415966 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f31c5f64-e792-4775-886f-32ff2a4ca444 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:42:28.417343 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f31c5f64-e792-4775-886f-32ff2a4ca444 None admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:42:28.427167 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 530/1058] 10.222.0.30 () {56 vars in 1108 bytes} [Mon May 23 12:42:28 2022] GET /identity/v3/services?type=image => generated 350 bytes in 144 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) May 23 12:42:28.428014 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7436754f-d40c-4bd2-a52f-8112a8f9805d None admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:42:28.536391 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7436754f-d40c-4bd2-a52f-8112a8f9805d None admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:42:28.571008 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d4ed4da-9f2a-4687-a0ad-8c945a94119a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:42:28.571665 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d4ed4da-9f2a-4687-a0ad-8c945a94119a None admin] REQUEST_METHOD: `POST` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:42:28.571946 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d4ed4da-9f2a-4687-a0ad-8c945a94119a None admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:42:28.572219 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d4ed4da-9f2a-4687-a0ad-8c945a94119a None admin] PATH_INFO: `/v3/endpoints` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:42:28.572965 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d4ed4da-9f2a-4687-a0ad-8c945a94119a None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:42:28.574371 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d4ed4da-9f2a-4687-a0ad-8c945a94119a None admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:42:28.649272 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 529/1059] 10.222.0.30 () {60 vars in 1144 bytes} [Mon May 23 12:42:28 2022] POST /identity/v3/endpoints => generated 327 bytes in 222 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) May 23 12:42:28.896863 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06398fbc-c0b8-4c56-8af4-1f3e3b762144 None None] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:42:28.897399 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06398fbc-c0b8-4c56-8af4-1f3e3b762144 None None] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:42:28.897487 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06398fbc-c0b8-4c56-8af4-1f3e3b762144 None None] PATH_INFO: `/` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:42:28.898991 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 531/1060] 10.222.0.30 () {54 vars in 926 bytes} [Mon May 23 12:42:28 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) May 23 12:42:28.904390 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ccbcbaa-c38f-499f-9bbd-c0838721b514 None None] REQUEST_METHOD: `POST` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:42:28.904690 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ccbcbaa-c38f-499f-9bbd-c0838721b514 None None] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:42:28.904971 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ccbcbaa-c38f-499f-9bbd-c0838721b514 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:42:28.921693 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.auth.core [None req-3ccbcbaa-c38f-499f-9bbd-c0838721b514 None None] MFA Rules not processed for user `0f7119f8fa5c4445977671140ded91a4`. Rule list: `[]` (Enabled: `True`). {{(pid=92056) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 12:42:28.922913 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.fernet_utils [None req-3ccbcbaa-c38f-499f-9bbd-c0838721b514 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92056) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 12:42:28.947872 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:42:28.998391 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 530/1061] 10.222.0.30 () {58 vars in 1023 bytes} [Mon May 23 12:42:28 2022] POST /identity/v3/auth/tokens => generated 2108 bytes in 95 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) May 23 12:42:29.021078 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06398fbc-c0b8-4c56-8af4-1f3e3b762144 None None] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:42:29.116613 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=100 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:42:29.129448 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06398fbc-c0b8-4c56-8af4-1f3e3b762144 None None] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:42:29.166813 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-443f8604-2762-4caa-b1bc-c65831a91d80 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:42:29.167536 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-443f8604-2762-4caa-b1bc-c65831a91d80 None admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:42:29.167817 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-443f8604-2762-4caa-b1bc-c65831a91d80 None admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:42:29.168090 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-443f8604-2762-4caa-b1bc-c65831a91d80 None admin] PATH_INFO: `/v3/roles/ResellerAdmin` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:42:29.179245 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-443f8604-2762-4caa-b1bc-c65831a91d80 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} May 23 12:42:29.187665 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application [None req-443f8604-2762-4caa-b1bc-c65831a91d80 None admin] Could not find role: ResellerAdmin.: keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. May 23 12:42:29.187665 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application Traceback (most recent call last): May 23 12:42:29.187665 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request May 23 12:42:29.187665 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application rv = self.dispatch_request() May 23 12:42:29.187665 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request May 23 12:42:29.187665 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) May 23 12:42:29.187665 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper May 23 12:42:29.187665 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) May 23 12:42:29.187665 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view May 23 12:42:29.187665 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) May 23 12:42:29.187665 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request May 23 12:42:29.187665 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) May 23 12:42:29.187665 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get May 23 12:42:29.187665 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self._get_role(role_id) May 23 12:42:29.187665 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role May 23 12:42:29.187665 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') May 23 12:42:29.188710 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call May 23 12:42:29.188710 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( May 23 12:42:29.188710 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data May 23 12:42:29.188710 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) May 23 12:42:29.188710 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped May 23 12:42:29.188710 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) May 23 12:42:29.188710 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun May 23 12:42:29.188710 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) May 23 12:42:29.188710 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func May 23 12:42:29.188710 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self.get_or_create( May 23 12:42:29.188710 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create May 23 12:42:29.188710 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application with Lock( May 23 12:42:29.188710 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ May 23 12:42:29.188710 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self._enter() May 23 12:42:29.188710 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter May 23 12:42:29.188710 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) May 23 12:42:29.189664 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create May 23 12:42:29.189664 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self.creator() May 23 12:42:29.189664 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value May 23 12:42:29.189664 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application created_value = creator( May 23 12:42:29.189664 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role May 23 12:42:29.189664 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) May 23 12:42:29.189664 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role May 23 12:42:29.189664 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() May 23 12:42:29.189664 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role May 23 12:42:29.189664 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) May 23 12:42:29.189664 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. May 23 12:42:29.189664 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application May 23 12:42:29.189664 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 532/1062] 10.222.0.30 () {56 vars in 1120 bytes} [Mon May 23 12:42:29 2022] GET /identity/v3/roles/ResellerAdmin => generated 91 bytes in 169 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) May 23 12:42:29.196890 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ccbcbaa-c38f-499f-9bbd-c0838721b514 None None] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:42:29.299551 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ccbcbaa-c38f-499f-9bbd-c0838721b514 None None] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:42:29.334014 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a7c6ef6-85f5-45d5-8739-90b78595bdd4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:42:29.334688 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a7c6ef6-85f5-45d5-8739-90b78595bdd4 None admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:42:29.334974 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a7c6ef6-85f5-45d5-8739-90b78595bdd4 None admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:42:29.335252 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a7c6ef6-85f5-45d5-8739-90b78595bdd4 None admin] PATH_INFO: `/v3/roles` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:42:29.336002 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a7c6ef6-85f5-45d5-8739-90b78595bdd4 None admin] RBAC: Adding query filter params (name=ResellerAdmin) {{(pid=92056) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} May 23 12:42:29.336313 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a7c6ef6-85f5-45d5-8739-90b78595bdd4 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:42:29.337690 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a7c6ef6-85f5-45d5-8739-90b78595bdd4 None admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:42:29.346854 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 531/1063] 10.222.0.30 () {56 vars in 1115 bytes} [Mon May 23 12:42:29 2022] GET /identity/v3/roles?name=ResellerAdmin => generated 338 bytes in 150 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) May 23 12:42:29.352457 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-443f8604-2762-4caa-b1bc-c65831a91d80 None admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:42:29.459070 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-443f8604-2762-4caa-b1bc-c65831a91d80 None admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:42:29.494642 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c970a34b-ce34-4c82-9ec2-7cfb1b66cf94 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:42:29.495439 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c970a34b-ce34-4c82-9ec2-7cfb1b66cf94 None admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:42:29.495726 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c970a34b-ce34-4c82-9ec2-7cfb1b66cf94 None admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:42:29.496002 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c970a34b-ce34-4c82-9ec2-7cfb1b66cf94 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:42:29.507588 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c970a34b-ce34-4c82-9ec2-7cfb1b66cf94 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:42:29.509065 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c970a34b-ce34-4c82-9ec2-7cfb1b66cf94 None admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:42:29.517240 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=100 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:42:29.527360 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application [None req-c970a34b-ce34-4c82-9ec2-7cfb1b66cf94 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. May 23 12:42:29.527360 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application Traceback (most recent call last): May 23 12:42:29.527360 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request May 23 12:42:29.527360 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application rv = self.dispatch_request() May 23 12:42:29.527360 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request May 23 12:42:29.527360 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) May 23 12:42:29.527360 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper May 23 12:42:29.527360 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) May 23 12:42:29.527360 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view May 23 12:42:29.527360 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) May 23 12:42:29.527360 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request May 23 12:42:29.527360 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) May 23 12:42:29.527360 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get May 23 12:42:29.527360 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self._get_domain(domain_id) May 23 12:42:29.527360 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain May 23 12:42:29.533007 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) May 23 12:42:29.533007 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped May 23 12:42:29.533007 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) May 23 12:42:29.533007 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun May 23 12:42:29.533007 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) May 23 12:42:29.533007 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func May 23 12:42:29.533007 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self.get_or_create( May 23 12:42:29.533007 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create May 23 12:42:29.533007 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application with Lock( May 23 12:42:29.533007 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ May 23 12:42:29.533007 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self._enter() May 23 12:42:29.533007 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter May 23 12:42:29.533007 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) May 23 12:42:29.533007 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create May 23 12:42:29.533007 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self.creator() May 23 12:42:29.533007 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value May 23 12:42:29.533007 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application created_value = creator( May 23 12:42:29.534779 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain May 23 12:42:29.534779 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) May 23 12:42:29.534779 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. May 23 12:42:29.534779 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application May 23 12:42:29.534779 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 533/1064] 10.222.0.30 () {56 vars in 1108 bytes} [Mon May 23 12:42:29 2022] GET /identity/v3/domains/Default => generated 87 bytes in 177 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) May 23 12:42:29.537943 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a7c6ef6-85f5-45d5-8739-90b78595bdd4 None admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:42:29.561737 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=100 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:42:29.630246 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38d28839-e2a2-4947-8ad1-f450caf80e21 None None] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:42:29.630883 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38d28839-e2a2-4947-8ad1-f450caf80e21 None None] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:42:29.631402 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38d28839-e2a2-4947-8ad1-f450caf80e21 None None] PATH_INFO: `/` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:42:29.633150 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 534/1065] 10.222.0.30 () {54 vars in 926 bytes} [Mon May 23 12:42:29 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) May 23 12:42:29.638359 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10fbd3e0-9663-4f66-9c0f-eee48e553b11 None None] REQUEST_METHOD: `POST` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:42:29.638649 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10fbd3e0-9663-4f66-9c0f-eee48e553b11 None None] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:42:29.638926 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10fbd3e0-9663-4f66-9c0f-eee48e553b11 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:42:29.648354 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a7c6ef6-85f5-45d5-8739-90b78595bdd4 None admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:42:29.657110 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.auth.core [None req-10fbd3e0-9663-4f66-9c0f-eee48e553b11 None None] MFA Rules not processed for user `0f7119f8fa5c4445977671140ded91a4`. Rule list: `[]` (Enabled: `True`). {{(pid=92057) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 12:42:29.658513 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.fernet_utils [None req-10fbd3e0-9663-4f66-9c0f-eee48e553b11 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92057) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 12:42:29.688083 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e43742e2-0486-4387-b026-a1f4e4a076f9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:42:29.690469 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e43742e2-0486-4387-b026-a1f4e4a076f9 None admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:42:29.692656 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e43742e2-0486-4387-b026-a1f4e4a076f9 None admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:42:29.692656 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e43742e2-0486-4387-b026-a1f4e4a076f9 None admin] PATH_INFO: `/v3/domains` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:42:29.692656 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e43742e2-0486-4387-b026-a1f4e4a076f9 None admin] RBAC: Adding query filter params (name=Default) {{(pid=92056) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} May 23 12:42:29.692656 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e43742e2-0486-4387-b026-a1f4e4a076f9 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:42:29.694352 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e43742e2-0486-4387-b026-a1f4e4a076f9 None admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:42:29.709518 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 532/1066] 10.222.0.30 () {56 vars in 1109 bytes} [Mon May 23 12:42:29 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 173 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) May 23 12:42:29.715016 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e43742e2-0486-4387-b026-a1f4e4a076f9 None admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:42:29.735264 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 535/1067] 10.222.0.30 () {58 vars in 1023 bytes} [Mon May 23 12:42:29 2022] POST /identity/v3/auth/tokens => generated 2108 bytes in 97 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) May 23 12:42:29.757729 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10fbd3e0-9663-4f66-9c0f-eee48e553b11 None None] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:42:29.799867 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=100 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:42:29.827520 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e43742e2-0486-4387-b026-a1f4e4a076f9 None admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:42:29.867688 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf35573f-39d2-401c-a395-a7e789315286 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:42:29.868483 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf35573f-39d2-401c-a395-a7e789315286 None admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:42:29.868767 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf35573f-39d2-401c-a395-a7e789315286 None admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:42:29.869040 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf35573f-39d2-401c-a395-a7e789315286 None admin] PATH_INFO: `/v3/users/nova` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:42:29.872103 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10fbd3e0-9663-4f66-9c0f-eee48e553b11 None None] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:42:29.876592 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf35573f-39d2-401c-a395-a7e789315286 None admin] RBAC: Authorizing `identity:get_user(user_id=nova)` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:42:29.878720 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf35573f-39d2-401c-a395-a7e789315286 None admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:42:29.895846 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application [None req-bf35573f-39d2-401c-a395-a7e789315286 None admin] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova. May 23 12:42:29.895846 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application Traceback (most recent call last): May 23 12:42:29.895846 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request May 23 12:42:29.895846 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application rv = self.dispatch_request() May 23 12:42:29.895846 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request May 23 12:42:29.895846 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) May 23 12:42:29.895846 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper May 23 12:42:29.895846 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) May 23 12:42:29.895846 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view May 23 12:42:29.895846 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) May 23 12:42:29.895846 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request May 23 12:42:29.895846 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) May 23 12:42:29.895846 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get May 23 12:42:29.895846 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self._get_user(user_id) May 23 12:42:29.895846 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user May 23 12:42:29.895846 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) May 23 12:42:29.896892 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped May 23 12:42:29.896892 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) May 23 12:42:29.896892 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper May 23 12:42:29.896892 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) May 23 12:42:29.896892 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper May 23 12:42:29.896892 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) May 23 12:42:29.896892 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun May 23 12:42:29.896892 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) May 23 12:42:29.896892 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func May 23 12:42:29.896892 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self.get_or_create( May 23 12:42:29.896892 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create May 23 12:42:29.896892 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application with Lock( May 23 12:42:29.896892 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ May 23 12:42:29.896892 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self._enter() May 23 12:42:29.896892 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter May 23 12:42:29.896892 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) May 23 12:42:29.896892 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create May 23 12:42:29.896892 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self.creator() May 23 12:42:29.897927 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value May 23 12:42:29.897927 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application created_value = creator( May 23 12:42:29.897927 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user May 23 12:42:29.897927 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) May 23 12:42:29.897927 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user May 23 12:42:29.897927 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) May 23 12:42:29.897927 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user May 23 12:42:29.897927 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) May 23 12:42:29.897927 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: nova. May 23 12:42:29.897927 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application May 23 12:42:29.902174 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 533/1068] 10.222.0.30 () {56 vars in 1093 bytes} [Mon May 23 12:42:29 2022] GET /identity/v3/users/nova => generated 82 bytes in 187 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) May 23 12:42:29.902880 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf35573f-39d2-401c-a395-a7e789315286 None admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:42:29.909753 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=100 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:42:29.916198 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64a8df31-d5d8-41a0-a905-edcc5b4ce196 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:42:29.917061 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64a8df31-d5d8-41a0-a905-edcc5b4ce196 None admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:42:29.922367 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64a8df31-d5d8-41a0-a905-edcc5b4ce196 None admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:42:29.922367 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64a8df31-d5d8-41a0-a905-edcc5b4ce196 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:42:29.928323 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-64a8df31-d5d8-41a0-a905-edcc5b4ce196 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} May 23 12:42:29.936562 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application [None req-64a8df31-d5d8-41a0-a905-edcc5b4ce196 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. May 23 12:42:29.936562 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application Traceback (most recent call last): May 23 12:42:29.936562 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request May 23 12:42:29.936562 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application rv = self.dispatch_request() May 23 12:42:29.936562 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request May 23 12:42:29.936562 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) May 23 12:42:29.936562 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper May 23 12:42:29.936562 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) May 23 12:42:29.936562 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view May 23 12:42:29.936562 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) May 23 12:42:29.936562 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request May 23 12:42:29.936562 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) May 23 12:42:29.936562 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get May 23 12:42:29.936562 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self._get_role(role_id) May 23 12:42:29.936562 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role May 23 12:42:29.936562 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') May 23 12:42:29.937840 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call May 23 12:42:29.937840 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( May 23 12:42:29.937840 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data May 23 12:42:29.937840 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) May 23 12:42:29.937840 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped May 23 12:42:29.937840 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) May 23 12:42:29.937840 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun May 23 12:42:29.937840 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) May 23 12:42:29.937840 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func May 23 12:42:29.937840 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self.get_or_create( May 23 12:42:29.937840 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create May 23 12:42:29.937840 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application with Lock( May 23 12:42:29.937840 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ May 23 12:42:29.937840 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self._enter() May 23 12:42:29.937840 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter May 23 12:42:29.937840 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) May 23 12:42:29.939012 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create May 23 12:42:29.939012 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self.creator() May 23 12:42:29.939012 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value May 23 12:42:29.939012 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application created_value = creator( May 23 12:42:29.939012 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role May 23 12:42:29.939012 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) May 23 12:42:29.939012 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role May 23 12:42:29.939012 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() May 23 12:42:29.939012 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role May 23 12:42:29.939012 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) May 23 12:42:29.939012 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. May 23 12:42:29.939012 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application May 23 12:42:29.946992 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 536/1069] 10.222.0.30 () {56 vars in 1096 bytes} [Mon May 23 12:42:29 2022] GET /identity/v3/roles/admin => generated 83 bytes in 190 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) May 23 12:42:29.947753 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64a8df31-d5d8-41a0-a905-edcc5b4ce196 None admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:42:30.017554 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf35573f-39d2-401c-a395-a7e789315286 None admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:42:30.054197 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eee4aa30-d969-4687-8130-2e66c5fb87b8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:42:30.054860 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eee4aa30-d969-4687-8130-2e66c5fb87b8 None admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:42:30.055138 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eee4aa30-d969-4687-8130-2e66c5fb87b8 None admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:42:30.055264 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64a8df31-d5d8-41a0-a905-edcc5b4ce196 None admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:42:30.055434 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eee4aa30-d969-4687-8130-2e66c5fb87b8 None admin] PATH_INFO: `/v3/users` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:42:30.056233 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eee4aa30-d969-4687-8130-2e66c5fb87b8 None admin] RBAC: Adding query filter params (domain_id=default, name=nova) {{(pid=92056) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} May 23 12:42:30.056597 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eee4aa30-d969-4687-8130-2e66c5fb87b8 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:42:30.058120 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eee4aa30-d969-4687-8130-2e66c5fb87b8 None admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:42:30.061519 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=100 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:42:30.077351 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 534/1070] 10.222.0.30 () {56 vars in 1133 bytes} [Mon May 23 12:42:29 2022] GET /identity/v3/users?domain_id=default&name=nova => generated 368 bytes in 175 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) May 23 12:42:30.083888 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eee4aa30-d969-4687-8130-2e66c5fb87b8 None admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:42:30.097831 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05d66182-4d37-40c2-b297-ed9f7067a841 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:42:30.098647 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05d66182-4d37-40c2-b297-ed9f7067a841 None admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:42:30.098943 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05d66182-4d37-40c2-b297-ed9f7067a841 None admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:42:30.099248 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05d66182-4d37-40c2-b297-ed9f7067a841 None admin] PATH_INFO: `/v3/roles` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:42:30.100672 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05d66182-4d37-40c2-b297-ed9f7067a841 None admin] RBAC: Adding query filter params (name=admin) {{(pid=92057) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} May 23 12:42:30.100993 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05d66182-4d37-40c2-b297-ed9f7067a841 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:42:30.102945 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05d66182-4d37-40c2-b297-ed9f7067a841 None admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:42:30.114035 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 537/1071] 10.222.0.30 () {56 vars in 1099 bytes} [Mon May 23 12:42:29 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 167 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) May 23 12:42:30.119154 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05d66182-4d37-40c2-b297-ed9f7067a841 None admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:42:30.174052 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=100 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:42:30.198638 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eee4aa30-d969-4687-8130-2e66c5fb87b8 None admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:42:30.228102 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05d66182-4d37-40c2-b297-ed9f7067a841 None admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:42:30.238031 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa5dadb7-abf0-4722-bf33-07e0c35dbcf2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:42:30.238831 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa5dadb7-abf0-4722-bf33-07e0c35dbcf2 None admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:42:30.239113 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa5dadb7-abf0-4722-bf33-07e0c35dbcf2 None admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:42:30.239384 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa5dadb7-abf0-4722-bf33-07e0c35dbcf2 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:42:30.254087 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa5dadb7-abf0-4722-bf33-07e0c35dbcf2 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:42:30.255962 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa5dadb7-abf0-4722-bf33-07e0c35dbcf2 None admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:42:30.264791 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94101224-dc62-411d-8818-db61e2aaf341 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:42:30.266093 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94101224-dc62-411d-8818-db61e2aaf341 None admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:42:30.266380 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94101224-dc62-411d-8818-db61e2aaf341 None admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:42:30.266806 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94101224-dc62-411d-8818-db61e2aaf341 None admin] PATH_INFO: `/v3/users/5eb771149c44430c8ee4cd2cd1e23890` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:42:30.268312 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94101224-dc62-411d-8818-db61e2aaf341 None admin] RBAC: Authorizing `identity:get_user(user_id=5eb771149c44430c8ee4cd2cd1e23890)` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:42:30.269913 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94101224-dc62-411d-8818-db61e2aaf341 None admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:42:30.271765 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 538/1072] 10.222.0.30 () {56 vars in 1177 bytes} [Mon May 23 12:42:30 2022] GET /identity/v3/users/5eb771149c44430c8ee4cd2cd1e23890 => generated 285 bytes in 153 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) May 23 12:42:30.275460 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application [None req-fa5dadb7-abf0-4722-bf33-07e0c35dbcf2 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. May 23 12:42:30.275460 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application Traceback (most recent call last): May 23 12:42:30.275460 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request May 23 12:42:30.275460 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application rv = self.dispatch_request() May 23 12:42:30.275460 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request May 23 12:42:30.275460 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) May 23 12:42:30.275460 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper May 23 12:42:30.275460 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) May 23 12:42:30.275460 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view May 23 12:42:30.275460 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) May 23 12:42:30.275460 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request May 23 12:42:30.275460 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) May 23 12:42:30.275460 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get May 23 12:42:30.275460 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self._get_domain(domain_id) May 23 12:42:30.275460 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain May 23 12:42:30.276591 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) May 23 12:42:30.276591 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped May 23 12:42:30.276591 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) May 23 12:42:30.276591 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun May 23 12:42:30.276591 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) May 23 12:42:30.276591 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func May 23 12:42:30.276591 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self.get_or_create( May 23 12:42:30.276591 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create May 23 12:42:30.276591 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application with Lock( May 23 12:42:30.276591 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ May 23 12:42:30.276591 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self._enter() May 23 12:42:30.276591 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter May 23 12:42:30.276591 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) May 23 12:42:30.276591 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create May 23 12:42:30.276591 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self.creator() May 23 12:42:30.276591 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value May 23 12:42:30.276591 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application created_value = creator( May 23 12:42:30.277834 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain May 23 12:42:30.277834 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) May 23 12:42:30.277834 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. May 23 12:42:30.277834 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application May 23 12:42:30.277834 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 535/1073] 10.222.0.30 () {56 vars in 1108 bytes} [Mon May 23 12:42:30 2022] GET /identity/v3/domains/Default => generated 87 bytes in 193 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) May 23 12:42:30.278334 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94101224-dc62-411d-8818-db61e2aaf341 None admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:42:30.283094 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa5dadb7-abf0-4722-bf33-07e0c35dbcf2 None admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:42:30.333395 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=100 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:42:30.391345 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94101224-dc62-411d-8818-db61e2aaf341 None admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:42:30.391798 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa5dadb7-abf0-4722-bf33-07e0c35dbcf2 None admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:42:30.415835 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=100 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:42:30.432132 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5348d72-a1bb-45f1-840c-37cd14a2529c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:42:30.432801 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5348d72-a1bb-45f1-840c-37cd14a2529c None admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:42:30.433125 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5348d72-a1bb-45f1-840c-37cd14a2529c None admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:42:30.433289 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09a7ef70-eb8d-4dc8-ad1d-7b7c73ae5af7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:42:30.433477 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5348d72-a1bb-45f1-840c-37cd14a2529c None admin] PATH_INFO: `/v3/domains` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:42:30.434169 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09a7ef70-eb8d-4dc8-ad1d-7b7c73ae5af7 None admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:42:30.434346 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5348d72-a1bb-45f1-840c-37cd14a2529c None admin] RBAC: Adding query filter params (name=Default) {{(pid=92056) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} May 23 12:42:30.434447 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09a7ef70-eb8d-4dc8-ad1d-7b7c73ae5af7 None admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:42:30.434708 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5348d72-a1bb-45f1-840c-37cd14a2529c None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:42:30.434806 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09a7ef70-eb8d-4dc8-ad1d-7b7c73ae5af7 None admin] PATH_INFO: `/v3/projects/c258ee56e55945e7b3182d1d20762347` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:42:30.436088 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09a7ef70-eb8d-4dc8-ad1d-7b7c73ae5af7 None admin] RBAC: Authorizing `identity:get_project(project_id=c258ee56e55945e7b3182d1d20762347)` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:42:30.436401 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5348d72-a1bb-45f1-840c-37cd14a2529c None admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:42:30.438472 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09a7ef70-eb8d-4dc8-ad1d-7b7c73ae5af7 None admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:42:30.441516 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 539/1074] 10.222.0.30 () {56 vars in 1186 bytes} [Mon May 23 12:42:30 2022] GET /identity/v3/projects/c258ee56e55945e7b3182d1d20762347 => generated 311 bytes in 164 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) May 23 12:42:30.447856 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09a7ef70-eb8d-4dc8-ad1d-7b7c73ae5af7 None admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:42:30.453614 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 536/1075] 10.222.0.30 () {56 vars in 1109 bytes} [Mon May 23 12:42:30 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 167 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) May 23 12:42:30.455075 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5348d72-a1bb-45f1-840c-37cd14a2529c None admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:42:30.561268 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5348d72-a1bb-45f1-840c-37cd14a2529c None admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:42:30.566734 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09a7ef70-eb8d-4dc8-ad1d-7b7c73ae5af7 None admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:42:30.569632 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=100 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:42:30.598911 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2e7a51c-95a3-4107-bad7-efcd29052c43 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:42:30.599783 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2e7a51c-95a3-4107-bad7-efcd29052c43 None admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:42:30.600134 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2e7a51c-95a3-4107-bad7-efcd29052c43 None admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:42:30.600472 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2e7a51c-95a3-4107-bad7-efcd29052c43 None admin] PATH_INFO: `/v3/projects/service` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:42:30.604168 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a01b371-988d-400f-84ee-dc8df3d5b833 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:42:30.605062 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a01b371-988d-400f-84ee-dc8df3d5b833 None admin] REQUEST_METHOD: `PUT` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:42:30.605409 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a01b371-988d-400f-84ee-dc8df3d5b833 None admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:42:30.605694 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a01b371-988d-400f-84ee-dc8df3d5b833 None admin] PATH_INFO: `/v3/projects/c258ee56e55945e7b3182d1d20762347/users/5eb771149c44430c8ee4cd2cd1e23890/roles/bc3c3c9d654940cbaa5177a6be23d1bc` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:42:30.607552 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2e7a51c-95a3-4107-bad7-efcd29052c43 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:42:30.607753 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a01b371-988d-400f-84ee-dc8df3d5b833 None admin] RBAC: Authorizing `identity:create_grant(project_id=c258ee56e55945e7b3182d1d20762347, user_id=5eb771149c44430c8ee4cd2cd1e23890, role_id=bc3c3c9d654940cbaa5177a6be23d1bc)` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:42:30.609127 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2e7a51c-95a3-4107-bad7-efcd29052c43 None admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:42:30.609232 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a01b371-988d-400f-84ee-dc8df3d5b833 None admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:42:30.624812 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application [None req-a2e7a51c-95a3-4107-bad7-efcd29052c43 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. May 23 12:42:30.624812 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application Traceback (most recent call last): May 23 12:42:30.624812 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request May 23 12:42:30.624812 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application rv = self.dispatch_request() May 23 12:42:30.624812 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request May 23 12:42:30.624812 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) May 23 12:42:30.624812 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper May 23 12:42:30.624812 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) May 23 12:42:30.624812 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view May 23 12:42:30.624812 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) May 23 12:42:30.624812 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request May 23 12:42:30.624812 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) May 23 12:42:30.624812 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get May 23 12:42:30.624812 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self._get_project(project_id) May 23 12:42:30.624812 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project May 23 12:42:30.626146 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) May 23 12:42:30.626146 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped May 23 12:42:30.626146 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) May 23 12:42:30.626146 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun May 23 12:42:30.626146 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) May 23 12:42:30.626146 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func May 23 12:42:30.626146 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self.get_or_create( May 23 12:42:30.626146 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create May 23 12:42:30.626146 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application with Lock( May 23 12:42:30.626146 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ May 23 12:42:30.626146 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self._enter() May 23 12:42:30.626146 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter May 23 12:42:30.626146 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) May 23 12:42:30.626146 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create May 23 12:42:30.626146 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self.creator() May 23 12:42:30.626146 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value May 23 12:42:30.626146 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application created_value = creator( May 23 12:42:30.627401 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project May 23 12:42:30.627401 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) May 23 12:42:30.627401 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project May 23 12:42:30.627401 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() May 23 12:42:30.627401 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project May 23 12:42:30.627401 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) May 23 12:42:30.627401 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. May 23 12:42:30.627401 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application May 23 12:42:30.627401 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 537/1076] 10.222.0.30 () {56 vars in 1111 bytes} [Mon May 23 12:42:30 2022] GET /identity/v3/projects/service => generated 88 bytes in 171 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) May 23 12:42:30.632262 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2e7a51c-95a3-4107-bad7-efcd29052c43 None admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:42:30.688656 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 540/1077] 10.222.0.30 () {58 vars in 1439 bytes} [Mon May 23 12:42:30 2022] PUT /identity/v3/projects/c258ee56e55945e7b3182d1d20762347/users/5eb771149c44430c8ee4cd2cd1e23890/roles/bc3c3c9d654940cbaa5177a6be23d1bc => generated 0 bytes in 241 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) May 23 12:42:30.751676 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2e7a51c-95a3-4107-bad7-efcd29052c43 None admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:42:30.794847 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7273533-e2ae-4525-9fb5-f011c9d9271c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:42:30.795530 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7273533-e2ae-4525-9fb5-f011c9d9271c None admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:42:30.795812 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7273533-e2ae-4525-9fb5-f011c9d9271c None admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:42:30.796091 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7273533-e2ae-4525-9fb5-f011c9d9271c None admin] PATH_INFO: `/v3/projects` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:42:30.796904 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7273533-e2ae-4525-9fb5-f011c9d9271c None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=92056) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} May 23 12:42:30.797215 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7273533-e2ae-4525-9fb5-f011c9d9271c None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:42:30.798784 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7273533-e2ae-4525-9fb5-f011c9d9271c None admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:42:30.817859 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 538/1078] 10.222.0.30 () {56 vars in 1148 bytes} [Mon May 23 12:42:30 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 180 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) May 23 12:42:30.819125 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a01b371-988d-400f-84ee-dc8df3d5b833 None admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:42:30.875426 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:42:30.931790 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a01b371-988d-400f-84ee-dc8df3d5b833 None admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:42:30.940190 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27437e86-4a37-4547-b9cf-72d3dda42e56 None None] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:42:30.940600 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27437e86-4a37-4547-b9cf-72d3dda42e56 None None] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:42:30.940980 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27437e86-4a37-4547-b9cf-72d3dda42e56 None None] PATH_INFO: `/` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:42:30.942541 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 539/1079] 10.222.0.30 () {54 vars in 926 bytes} [Mon May 23 12:42:30 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) May 23 12:42:30.947475 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e4edd8d-7fa3-45c4-8a2c-c7ba31ef4bfa None None] REQUEST_METHOD: `POST` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:42:30.947763 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e4edd8d-7fa3-45c4-8a2c-c7ba31ef4bfa None None] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:42:30.948037 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e4edd8d-7fa3-45c4-8a2c-c7ba31ef4bfa None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:42:30.965782 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.auth.core [None req-1e4edd8d-7fa3-45c4-8a2c-c7ba31ef4bfa None None] MFA Rules not processed for user `0f7119f8fa5c4445977671140ded91a4`. Rule list: `[]` (Enabled: `True`). {{(pid=92056) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 12:42:30.966990 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.fernet_utils [None req-1e4edd8d-7fa3-45c4-8a2c-c7ba31ef4bfa None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92056) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 12:42:30.971399 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aef39842-65c0-4ff0-abb0-78bbaced3f7b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:42:30.972397 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aef39842-65c0-4ff0-abb0-78bbaced3f7b None admin] REQUEST_METHOD: `PUT` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:42:30.972788 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aef39842-65c0-4ff0-abb0-78bbaced3f7b None admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:42:30.973155 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aef39842-65c0-4ff0-abb0-78bbaced3f7b None admin] PATH_INFO: `/v3/projects/9c389ae65aaa4542bb5a5c4fd0bacf5b/users/cbe9931d94d44cc7ace0e83a7e5015fc/roles/940396c67e6744c784e95e01fe84ad45` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:42:30.975527 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aef39842-65c0-4ff0-abb0-78bbaced3f7b None admin] RBAC: Authorizing `identity:create_grant(project_id=9c389ae65aaa4542bb5a5c4fd0bacf5b, user_id=cbe9931d94d44cc7ace0e83a7e5015fc, role_id=940396c67e6744c784e95e01fe84ad45)` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:42:30.977061 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aef39842-65c0-4ff0-abb0-78bbaced3f7b None admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:42:30.989542 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=100 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:42:31.035558 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 540/1080] 10.222.0.30 () {58 vars in 1023 bytes} [Mon May 23 12:42:30 2022] POST /identity/v3/auth/tokens => generated 2108 bytes in 89 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) May 23 12:42:31.057796 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e4edd8d-7fa3-45c4-8a2c-c7ba31ef4bfa None None] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:42:31.077466 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 541/1081] 10.222.0.30 () {58 vars in 1439 bytes} [Mon May 23 12:42:30 2022] PUT /identity/v3/projects/9c389ae65aaa4542bb5a5c4fd0bacf5b/users/cbe9931d94d44cc7ace0e83a7e5015fc/roles/940396c67e6744c784e95e01fe84ad45 => generated 0 bytes in 259 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) May 23 12:42:31.169326 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e4edd8d-7fa3-45c4-8a2c-c7ba31ef4bfa None None] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:42:31.207427 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d88b4b6-dbe6-43f4-acf3-11c88fcb27e2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:42:31.208153 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d88b4b6-dbe6-43f4-acf3-11c88fcb27e2 None admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:42:31.208438 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d88b4b6-dbe6-43f4-acf3-11c88fcb27e2 None admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:42:31.208713 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d88b4b6-dbe6-43f4-acf3-11c88fcb27e2 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:42:31.236861 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=100 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:42:31.284681 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d88b4b6-dbe6-43f4-acf3-11c88fcb27e2 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:42:31.286196 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d88b4b6-dbe6-43f4-acf3-11c88fcb27e2 None admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:42:31.388548 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 541/1082] 10.222.0.30 () {58 vars in 1282 bytes} [Mon May 23 12:42:31 2022] GET /identity/v3/auth/tokens => generated 2108 bytes in 331 msecs (HTTP/1.1 200) 6 headers in 359 bytes (2 switches on core 0) May 23 12:42:31.394364 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aef39842-65c0-4ff0-abb0-78bbaced3f7b None admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:42:31.500243 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aef39842-65c0-4ff0-abb0-78bbaced3f7b None admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:42:31.509142 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:42:31.534977 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e64d5649-556a-424e-85cf-c69f3a6f6fc8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:42:31.535778 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e64d5649-556a-424e-85cf-c69f3a6f6fc8 None admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:42:31.536068 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e64d5649-556a-424e-85cf-c69f3a6f6fc8 None admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:42:31.536350 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e64d5649-556a-424e-85cf-c69f3a6f6fc8 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:42:31.547867 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e64d5649-556a-424e-85cf-c69f3a6f6fc8 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:42:31.550217 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e64d5649-556a-424e-85cf-c69f3a6f6fc8 None admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:42:31.569253 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application [None req-e64d5649-556a-424e-85cf-c69f3a6f6fc8 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. May 23 12:42:31.569253 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application Traceback (most recent call last): May 23 12:42:31.569253 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request May 23 12:42:31.569253 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application rv = self.dispatch_request() May 23 12:42:31.569253 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request May 23 12:42:31.569253 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) May 23 12:42:31.569253 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper May 23 12:42:31.569253 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) May 23 12:42:31.569253 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view May 23 12:42:31.569253 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) May 23 12:42:31.569253 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request May 23 12:42:31.569253 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) May 23 12:42:31.569253 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get May 23 12:42:31.569253 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self._get_domain(domain_id) May 23 12:42:31.569253 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain May 23 12:42:31.570483 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) May 23 12:42:31.570483 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped May 23 12:42:31.570483 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) May 23 12:42:31.570483 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun May 23 12:42:31.570483 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) May 23 12:42:31.570483 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func May 23 12:42:31.570483 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self.get_or_create( May 23 12:42:31.570483 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create May 23 12:42:31.570483 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application with Lock( May 23 12:42:31.570483 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ May 23 12:42:31.570483 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self._enter() May 23 12:42:31.570483 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter May 23 12:42:31.570483 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) May 23 12:42:31.570483 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create May 23 12:42:31.570483 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self.creator() May 23 12:42:31.570483 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value May 23 12:42:31.570483 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application created_value = creator( May 23 12:42:31.571655 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain May 23 12:42:31.571655 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) May 23 12:42:31.571655 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. May 23 12:42:31.571655 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application May 23 12:42:31.573948 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 542/1083] 10.222.0.30 () {56 vars in 1108 bytes} [Mon May 23 12:42:31 2022] GET /identity/v3/domains/Default => generated 87 bytes in 177 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) May 23 12:42:31.576562 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d88b4b6-dbe6-43f4-acf3-11c88fcb27e2 None admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:42:31.634068 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=100 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:42:31.681270 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d88b4b6-dbe6-43f4-acf3-11c88fcb27e2 None admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:42:31.718104 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4691893-eef9-42df-9ee8-4f058fc866b3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:42:31.718857 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4691893-eef9-42df-9ee8-4f058fc866b3 None admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:42:31.719184 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4691893-eef9-42df-9ee8-4f058fc866b3 None admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:42:31.719502 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4691893-eef9-42df-9ee8-4f058fc866b3 None admin] PATH_INFO: `/v3/domains` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:42:31.720301 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c4691893-eef9-42df-9ee8-4f058fc866b3 None admin] RBAC: Adding query filter params (name=Default) {{(pid=92056) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} May 23 12:42:31.721263 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c4691893-eef9-42df-9ee8-4f058fc866b3 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:42:31.722826 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c4691893-eef9-42df-9ee8-4f058fc866b3 None admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:42:31.741974 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 542/1084] 10.222.0.30 () {56 vars in 1109 bytes} [Mon May 23 12:42:31 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 158 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) May 23 12:42:32.930924 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f235f92-7e03-4374-87de-8a10ca4cb92c None None] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:42:32.931499 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f235f92-7e03-4374-87de-8a10ca4cb92c None None] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:42:32.931499 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f235f92-7e03-4374-87de-8a10ca4cb92c None None] PATH_INFO: `/` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:42:32.932837 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 543/1085] 10.222.0.30 () {54 vars in 926 bytes} [Mon May 23 12:42:32 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) May 23 12:42:32.938252 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95791b0b-b18a-4f91-83a0-bc68674e7f32 None None] REQUEST_METHOD: `POST` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:42:32.938667 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95791b0b-b18a-4f91-83a0-bc68674e7f32 None None] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:42:32.938809 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95791b0b-b18a-4f91-83a0-bc68674e7f32 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:42:32.956878 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.auth.core [None req-95791b0b-b18a-4f91-83a0-bc68674e7f32 None None] MFA Rules not processed for user `0f7119f8fa5c4445977671140ded91a4`. Rule list: `[]` (Enabled: `True`). {{(pid=92056) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 12:42:32.958010 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.fernet_utils [None req-95791b0b-b18a-4f91-83a0-bc68674e7f32 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92056) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 12:42:33.025648 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 543/1086] 10.222.0.30 () {58 vars in 1023 bytes} [Mon May 23 12:42:32 2022] POST /identity/v3/auth/tokens => generated 2108 bytes in 88 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) May 23 12:42:33.048938 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f235f92-7e03-4374-87de-8a10ca4cb92c None None] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:42:33.160054 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f235f92-7e03-4374-87de-8a10ca4cb92c None None] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:42:33.196643 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-369a3f3e-929d-427c-b7f4-43b4fc465fe9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:42:33.197428 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-369a3f3e-929d-427c-b7f4-43b4fc465fe9 None admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:42:33.197712 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-369a3f3e-929d-427c-b7f4-43b4fc465fe9 None admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:42:33.198025 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-369a3f3e-929d-427c-b7f4-43b4fc465fe9 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:42:33.209118 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-369a3f3e-929d-427c-b7f4-43b4fc465fe9 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} May 23 12:42:33.216636 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application [None req-369a3f3e-929d-427c-b7f4-43b4fc465fe9 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. May 23 12:42:33.216636 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application Traceback (most recent call last): May 23 12:42:33.216636 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request May 23 12:42:33.216636 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application rv = self.dispatch_request() May 23 12:42:33.216636 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request May 23 12:42:33.216636 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) May 23 12:42:33.216636 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper May 23 12:42:33.216636 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) May 23 12:42:33.216636 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view May 23 12:42:33.216636 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) May 23 12:42:33.216636 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request May 23 12:42:33.216636 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) May 23 12:42:33.216636 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get May 23 12:42:33.216636 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self._get_role(role_id) May 23 12:42:33.216636 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role May 23 12:42:33.216636 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') May 23 12:42:33.217635 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call May 23 12:42:33.217635 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( May 23 12:42:33.217635 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data May 23 12:42:33.217635 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) May 23 12:42:33.217635 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped May 23 12:42:33.217635 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) May 23 12:42:33.217635 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun May 23 12:42:33.217635 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) May 23 12:42:33.217635 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func May 23 12:42:33.217635 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self.get_or_create( May 23 12:42:33.217635 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create May 23 12:42:33.217635 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application with Lock( May 23 12:42:33.217635 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ May 23 12:42:33.217635 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self._enter() May 23 12:42:33.217635 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter May 23 12:42:33.217635 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) May 23 12:42:33.219014 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create May 23 12:42:33.219014 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self.creator() May 23 12:42:33.219014 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value May 23 12:42:33.219014 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application created_value = creator( May 23 12:42:33.219014 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role May 23 12:42:33.219014 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) May 23 12:42:33.219014 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role May 23 12:42:33.219014 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() May 23 12:42:33.219014 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role May 23 12:42:33.219014 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) May 23 12:42:33.219014 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. May 23 12:42:33.219014 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application May 23 12:42:33.219014 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 544/1087] 10.222.0.30 () {56 vars in 1096 bytes} [Mon May 23 12:42:33 2022] GET /identity/v3/roles/admin => generated 83 bytes in 170 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) May 23 12:42:33.223873 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95791b0b-b18a-4f91-83a0-bc68674e7f32 None None] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:42:33.271971 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=100 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:42:33.317185 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd0f1f27-0a77-407f-9c16-41d45bde3097 None None] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:42:33.317460 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd0f1f27-0a77-407f-9c16-41d45bde3097 None None] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:42:33.317733 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd0f1f27-0a77-407f-9c16-41d45bde3097 None None] PATH_INFO: `/` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:42:33.319166 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 545/1088] 10.222.0.30 () {54 vars in 926 bytes} [Mon May 23 12:42:33 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) May 23 12:42:33.324186 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e77bd902-5726-446a-bbc2-128dd6de7314 None None] REQUEST_METHOD: `POST` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:42:33.324477 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e77bd902-5726-446a-bbc2-128dd6de7314 None None] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:42:33.324754 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e77bd902-5726-446a-bbc2-128dd6de7314 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:42:33.326714 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95791b0b-b18a-4f91-83a0-bc68674e7f32 None None] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:42:33.340948 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.auth.core [None req-e77bd902-5726-446a-bbc2-128dd6de7314 None None] MFA Rules not processed for user `0f7119f8fa5c4445977671140ded91a4`. Rule list: `[]` (Enabled: `True`). {{(pid=92057) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 12:42:33.342161 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.fernet_utils [None req-e77bd902-5726-446a-bbc2-128dd6de7314 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92057) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 12:42:33.355873 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=100 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:42:33.360947 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6e4f1f0-2167-4a58-a1ee-7b3d9b4ba165 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:42:33.362216 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6e4f1f0-2167-4a58-a1ee-7b3d9b4ba165 None admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:42:33.366785 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6e4f1f0-2167-4a58-a1ee-7b3d9b4ba165 None admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:42:33.366785 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6e4f1f0-2167-4a58-a1ee-7b3d9b4ba165 None admin] PATH_INFO: `/v3/roles` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:42:33.366785 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6e4f1f0-2167-4a58-a1ee-7b3d9b4ba165 None admin] RBAC: Adding query filter params (name=admin) {{(pid=92056) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} May 23 12:42:33.366785 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6e4f1f0-2167-4a58-a1ee-7b3d9b4ba165 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:42:33.366785 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6e4f1f0-2167-4a58-a1ee-7b3d9b4ba165 None admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:42:33.373409 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 544/1089] 10.222.0.30 () {56 vars in 1099 bytes} [Mon May 23 12:42:33 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 150 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) May 23 12:42:33.379671 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6e4f1f0-2167-4a58-a1ee-7b3d9b4ba165 None admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:42:33.411704 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 546/1090] 10.222.0.30 () {58 vars in 1023 bytes} [Mon May 23 12:42:33 2022] POST /identity/v3/auth/tokens => generated 2108 bytes in 88 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) May 23 12:42:33.434890 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e77bd902-5726-446a-bbc2-128dd6de7314 None None] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:42:33.478664 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6e4f1f0-2167-4a58-a1ee-7b3d9b4ba165 None admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:42:33.487947 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=100 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:42:33.512751 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e3727c7-8e91-4230-bcda-815365f8d3fb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:42:33.513621 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e3727c7-8e91-4230-bcda-815365f8d3fb None admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:42:33.513925 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e3727c7-8e91-4230-bcda-815365f8d3fb None admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:42:33.514213 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e3727c7-8e91-4230-bcda-815365f8d3fb None admin] PATH_INFO: `/v3/users/5eb771149c44430c8ee4cd2cd1e23890` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:42:33.515421 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e3727c7-8e91-4230-bcda-815365f8d3fb None admin] RBAC: Authorizing `identity:get_user(user_id=5eb771149c44430c8ee4cd2cd1e23890)` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:42:33.516875 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e3727c7-8e91-4230-bcda-815365f8d3fb None admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:42:33.518586 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 545/1091] 10.222.0.30 () {56 vars in 1177 bytes} [Mon May 23 12:42:33 2022] GET /identity/v3/users/5eb771149c44430c8ee4cd2cd1e23890 => generated 285 bytes in 139 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) May 23 12:42:33.524102 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e3727c7-8e91-4230-bcda-815365f8d3fb None admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:42:33.534860 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e77bd902-5726-446a-bbc2-128dd6de7314 None None] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:42:33.569396 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57a906c6-a3c1-40b6-9f41-49a90190f8b8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:42:33.570226 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57a906c6-a3c1-40b6-9f41-49a90190f8b8 None admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:42:33.570507 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57a906c6-a3c1-40b6-9f41-49a90190f8b8 None admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:42:33.570781 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57a906c6-a3c1-40b6-9f41-49a90190f8b8 None admin] PATH_INFO: `/v3/roles/ResellerAdmin` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:42:33.581392 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57a906c6-a3c1-40b6-9f41-49a90190f8b8 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} May 23 12:42:33.589804 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application [None req-57a906c6-a3c1-40b6-9f41-49a90190f8b8 None admin] Could not find role: ResellerAdmin.: keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. May 23 12:42:33.589804 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application Traceback (most recent call last): May 23 12:42:33.589804 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request May 23 12:42:33.589804 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application rv = self.dispatch_request() May 23 12:42:33.589804 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request May 23 12:42:33.589804 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) May 23 12:42:33.589804 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper May 23 12:42:33.589804 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) May 23 12:42:33.589804 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view May 23 12:42:33.589804 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) May 23 12:42:33.589804 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request May 23 12:42:33.589804 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) May 23 12:42:33.589804 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get May 23 12:42:33.589804 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self._get_role(role_id) May 23 12:42:33.589804 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role May 23 12:42:33.589804 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') May 23 12:42:33.590950 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call May 23 12:42:33.590950 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( May 23 12:42:33.590950 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data May 23 12:42:33.590950 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) May 23 12:42:33.590950 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped May 23 12:42:33.590950 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) May 23 12:42:33.590950 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun May 23 12:42:33.590950 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) May 23 12:42:33.590950 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func May 23 12:42:33.590950 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self.get_or_create( May 23 12:42:33.590950 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create May 23 12:42:33.590950 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application with Lock( May 23 12:42:33.590950 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ May 23 12:42:33.590950 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self._enter() May 23 12:42:33.590950 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter May 23 12:42:33.590950 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) May 23 12:42:33.591853 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create May 23 12:42:33.591853 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self.creator() May 23 12:42:33.591853 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value May 23 12:42:33.591853 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application created_value = creator( May 23 12:42:33.591853 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role May 23 12:42:33.591853 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) May 23 12:42:33.591853 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role May 23 12:42:33.591853 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() May 23 12:42:33.591853 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role May 23 12:42:33.591853 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) May 23 12:42:33.591853 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. May 23 12:42:33.591853 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application May 23 12:42:33.591853 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 547/1092] 10.222.0.30 () {56 vars in 1120 bytes} [Mon May 23 12:42:33 2022] GET /identity/v3/roles/ResellerAdmin => generated 91 bytes in 157 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) May 23 12:42:33.597564 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57a906c6-a3c1-40b6-9f41-49a90190f8b8 None admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:42:33.610722 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=100 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:42:33.627744 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e3727c7-8e91-4230-bcda-815365f8d3fb None admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:42:33.661357 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-007741a3-daef-4b60-902f-2c6376b7237f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:42:33.662031 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-007741a3-daef-4b60-902f-2c6376b7237f None admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:42:33.662315 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-007741a3-daef-4b60-902f-2c6376b7237f None admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:42:33.662593 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-007741a3-daef-4b60-902f-2c6376b7237f None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:42:33.705564 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57a906c6-a3c1-40b6-9f41-49a90190f8b8 None admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:42:33.707631 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=100 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:42:33.729401 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-007741a3-daef-4b60-902f-2c6376b7237f None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:42:33.730806 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-007741a3-daef-4b60-902f-2c6376b7237f None admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:42:33.739361 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10b47583-df71-4c77-8070-aabfdfdd1765 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:42:33.740029 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10b47583-df71-4c77-8070-aabfdfdd1765 None admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:42:33.740315 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10b47583-df71-4c77-8070-aabfdfdd1765 None admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:42:33.740590 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10b47583-df71-4c77-8070-aabfdfdd1765 None admin] PATH_INFO: `/v3/roles` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:42:33.741331 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10b47583-df71-4c77-8070-aabfdfdd1765 None admin] RBAC: Adding query filter params (name=ResellerAdmin) {{(pid=92057) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} May 23 12:42:33.741638 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10b47583-df71-4c77-8070-aabfdfdd1765 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:42:33.743043 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10b47583-df71-4c77-8070-aabfdfdd1765 None admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:42:33.751525 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 548/1093] 10.222.0.30 () {56 vars in 1115 bytes} [Mon May 23 12:42:33 2022] GET /identity/v3/roles?name=ResellerAdmin => generated 338 bytes in 154 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) May 23 12:42:33.757630 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10b47583-df71-4c77-8070-aabfdfdd1765 None admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:42:33.827555 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=100 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:42:33.831379 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 546/1094] 10.222.0.30 () {58 vars in 1282 bytes} [Mon May 23 12:42:33 2022] GET /identity/v3/auth/tokens => generated 2108 bytes in 308 msecs (HTTP/1.1 200) 6 headers in 359 bytes (2 switches on core 0) May 23 12:42:33.837160 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-007741a3-daef-4b60-902f-2c6376b7237f None admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:42:33.858889 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10b47583-df71-4c77-8070-aabfdfdd1765 None admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:42:33.895746 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f00fc39-54c8-4822-bcf6-374f97ea29cb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:42:33.896528 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f00fc39-54c8-4822-bcf6-374f97ea29cb None admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:42:33.896813 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f00fc39-54c8-4822-bcf6-374f97ea29cb None admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:42:33.897088 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f00fc39-54c8-4822-bcf6-374f97ea29cb None admin] PATH_INFO: `/v3/domains/Default` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:42:33.907638 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f00fc39-54c8-4822-bcf6-374f97ea29cb None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:42:33.909136 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f00fc39-54c8-4822-bcf6-374f97ea29cb None admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:42:33.920558 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=100 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:42:33.926529 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application [None req-4f00fc39-54c8-4822-bcf6-374f97ea29cb None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. May 23 12:42:33.926529 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application Traceback (most recent call last): May 23 12:42:33.926529 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request May 23 12:42:33.926529 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application rv = self.dispatch_request() May 23 12:42:33.926529 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request May 23 12:42:33.926529 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) May 23 12:42:33.926529 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper May 23 12:42:33.926529 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) May 23 12:42:33.926529 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view May 23 12:42:33.926529 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) May 23 12:42:33.926529 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request May 23 12:42:33.926529 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) May 23 12:42:33.926529 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get May 23 12:42:33.926529 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self._get_domain(domain_id) May 23 12:42:33.926529 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain May 23 12:42:33.927518 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) May 23 12:42:33.927518 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped May 23 12:42:33.927518 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) May 23 12:42:33.927518 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun May 23 12:42:33.927518 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) May 23 12:42:33.927518 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func May 23 12:42:33.927518 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self.get_or_create( May 23 12:42:33.927518 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create May 23 12:42:33.927518 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application with Lock( May 23 12:42:33.927518 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ May 23 12:42:33.927518 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self._enter() May 23 12:42:33.927518 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter May 23 12:42:33.927518 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) May 23 12:42:33.927518 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create May 23 12:42:33.927518 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self.creator() May 23 12:42:33.927518 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value May 23 12:42:33.927518 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application created_value = creator( May 23 12:42:33.928479 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain May 23 12:42:33.928479 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) May 23 12:42:33.928479 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. May 23 12:42:33.928479 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application May 23 12:42:33.928479 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 549/1095] 10.222.0.30 () {56 vars in 1108 bytes} [Mon May 23 12:42:33 2022] GET /identity/v3/domains/Default => generated 87 bytes in 170 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) May 23 12:42:33.933306 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f00fc39-54c8-4822-bcf6-374f97ea29cb None admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:42:33.937753 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-007741a3-daef-4b60-902f-2c6376b7237f None admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:42:33.973386 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93fd4c6e-f289-4992-a039-9106478f177e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:42:33.974420 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93fd4c6e-f289-4992-a039-9106478f177e None admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:42:33.974699 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93fd4c6e-f289-4992-a039-9106478f177e None admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:42:33.974972 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93fd4c6e-f289-4992-a039-9106478f177e None admin] PATH_INFO: `/v3/projects/c258ee56e55945e7b3182d1d20762347` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:42:33.976190 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93fd4c6e-f289-4992-a039-9106478f177e None admin] RBAC: Authorizing `identity:get_project(project_id=c258ee56e55945e7b3182d1d20762347)` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:42:33.977636 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93fd4c6e-f289-4992-a039-9106478f177e None admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:42:33.979387 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 547/1096] 10.222.0.30 () {56 vars in 1186 bytes} [Mon May 23 12:42:33 2022] GET /identity/v3/projects/c258ee56e55945e7b3182d1d20762347 => generated 311 bytes in 143 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) May 23 12:42:33.985950 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93fd4c6e-f289-4992-a039-9106478f177e None admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:42:34.037944 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f00fc39-54c8-4822-bcf6-374f97ea29cb None admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:42:34.062444 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=100 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:42:34.074302 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c96a1e1a-8ee9-4efa-bb3c-74a1e608bdad None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:42:34.074962 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c96a1e1a-8ee9-4efa-bb3c-74a1e608bdad None admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:42:34.075257 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c96a1e1a-8ee9-4efa-bb3c-74a1e608bdad None admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:42:34.075583 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c96a1e1a-8ee9-4efa-bb3c-74a1e608bdad None admin] PATH_INFO: `/v3/domains` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:42:34.076298 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c96a1e1a-8ee9-4efa-bb3c-74a1e608bdad None admin] RBAC: Adding query filter params (name=Default) {{(pid=92057) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} May 23 12:42:34.076610 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c96a1e1a-8ee9-4efa-bb3c-74a1e608bdad None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:42:34.078000 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c96a1e1a-8ee9-4efa-bb3c-74a1e608bdad None admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:42:34.088804 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93fd4c6e-f289-4992-a039-9106478f177e None admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:42:34.089549 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 550/1097] 10.222.0.30 () {56 vars in 1109 bytes} [Mon May 23 12:42:33 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 157 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) May 23 12:42:34.095099 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c96a1e1a-8ee9-4efa-bb3c-74a1e608bdad None admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:42:34.125055 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b98c52f2-9ec3-4850-91e9-bd6ce9642170 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:42:34.125696 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b98c52f2-9ec3-4850-91e9-bd6ce9642170 None admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:42:34.126009 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b98c52f2-9ec3-4850-91e9-bd6ce9642170 None admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:42:34.126283 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b98c52f2-9ec3-4850-91e9-bd6ce9642170 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:42:34.127744 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b98c52f2-9ec3-4850-91e9-bd6ce9642170 None admin] RBAC: Adding query filter params (role.id=bc3c3c9d654940cbaa5177a6be23d1bc, scope.project.id=c258ee56e55945e7b3182d1d20762347, user.id=5eb771149c44430c8ee4cd2cd1e23890) {{(pid=92056) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} May 23 12:42:34.128052 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b98c52f2-9ec3-4850-91e9-bd6ce9642170 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:42:34.129473 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b98c52f2-9ec3-4850-91e9-bd6ce9642170 None admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:42:34.135631 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 548/1098] 10.222.0.30 () {56 vars in 1374 bytes} [Mon May 23 12:42:33 2022] GET /identity/v3/role_assignments?user.id=5eb771149c44430c8ee4cd2cd1e23890&scope.project.id=c258ee56e55945e7b3182d1d20762347&role.id=bc3c3c9d654940cbaa5177a6be23d1bc => generated 608 bytes in 150 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) May 23 12:42:34.198808 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c96a1e1a-8ee9-4efa-bb3c-74a1e608bdad None admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:42:34.235282 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f82d462d-a290-4b44-b2af-23055268edc5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:42:34.236723 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f82d462d-a290-4b44-b2af-23055268edc5 None admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:42:34.237060 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f82d462d-a290-4b44-b2af-23055268edc5 None admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:42:34.237340 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f82d462d-a290-4b44-b2af-23055268edc5 None admin] PATH_INFO: `/v3/users/nova` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:42:34.245293 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f82d462d-a290-4b44-b2af-23055268edc5 None admin] RBAC: Authorizing `identity:get_user(user_id=nova)` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:42:34.246814 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f82d462d-a290-4b44-b2af-23055268edc5 None admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:42:34.262894 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application [None req-f82d462d-a290-4b44-b2af-23055268edc5 None admin] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova. May 23 12:42:34.262894 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application Traceback (most recent call last): May 23 12:42:34.262894 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request May 23 12:42:34.262894 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application rv = self.dispatch_request() May 23 12:42:34.262894 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request May 23 12:42:34.262894 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) May 23 12:42:34.262894 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper May 23 12:42:34.262894 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) May 23 12:42:34.262894 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view May 23 12:42:34.262894 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) May 23 12:42:34.262894 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request May 23 12:42:34.262894 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) May 23 12:42:34.262894 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get May 23 12:42:34.262894 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self._get_user(user_id) May 23 12:42:34.262894 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user May 23 12:42:34.262894 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) May 23 12:42:34.264112 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped May 23 12:42:34.264112 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) May 23 12:42:34.264112 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper May 23 12:42:34.264112 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) May 23 12:42:34.264112 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper May 23 12:42:34.264112 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) May 23 12:42:34.264112 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun May 23 12:42:34.264112 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) May 23 12:42:34.264112 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func May 23 12:42:34.264112 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self.get_or_create( May 23 12:42:34.264112 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create May 23 12:42:34.264112 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application with Lock( May 23 12:42:34.264112 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ May 23 12:42:34.264112 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self._enter() May 23 12:42:34.264112 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter May 23 12:42:34.264112 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) May 23 12:42:34.264112 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create May 23 12:42:34.264112 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self.creator() May 23 12:42:34.265638 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value May 23 12:42:34.265638 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application created_value = creator( May 23 12:42:34.265638 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user May 23 12:42:34.265638 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) May 23 12:42:34.265638 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user May 23 12:42:34.265638 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) May 23 12:42:34.265638 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user May 23 12:42:34.265638 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) May 23 12:42:34.265638 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: nova. May 23 12:42:34.265638 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application May 23 12:42:34.265638 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 551/1099] 10.222.0.30 () {56 vars in 1093 bytes} [Mon May 23 12:42:34 2022] GET /identity/v3/users/nova => generated 82 bytes in 169 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) May 23 12:42:34.270299 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b98c52f2-9ec3-4850-91e9-bd6ce9642170 None admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:42:34.281993 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=100 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:42:34.374978 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b98c52f2-9ec3-4850-91e9-bd6ce9642170 None admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:42:34.409714 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60546c7d-dfb8-4b22-a06f-46ac39d626f8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:42:34.410402 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60546c7d-dfb8-4b22-a06f-46ac39d626f8 None admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:42:34.410681 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60546c7d-dfb8-4b22-a06f-46ac39d626f8 None admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:42:34.410958 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60546c7d-dfb8-4b22-a06f-46ac39d626f8 None admin] PATH_INFO: `/v3/users` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:42:34.411780 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60546c7d-dfb8-4b22-a06f-46ac39d626f8 None admin] RBAC: Adding query filter params (domain_id=default, name=nova) {{(pid=92056) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} May 23 12:42:34.412085 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60546c7d-dfb8-4b22-a06f-46ac39d626f8 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:42:34.413529 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60546c7d-dfb8-4b22-a06f-46ac39d626f8 None admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:42:34.431308 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 549/1100] 10.222.0.30 () {56 vars in 1133 bytes} [Mon May 23 12:42:34 2022] GET /identity/v3/users?domain_id=default&name=nova => generated 368 bytes in 162 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) May 23 12:42:34.437064 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f82d462d-a290-4b44-b2af-23055268edc5 None admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:42:34.484863 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=100 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:42:34.539718 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f82d462d-a290-4b44-b2af-23055268edc5 None admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:42:34.574301 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24515058-fcc2-4083-9a2a-69e1f05b057d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:42:34.575011 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24515058-fcc2-4083-9a2a-69e1f05b057d None admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:42:34.575338 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24515058-fcc2-4083-9a2a-69e1f05b057d None admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:42:34.575658 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24515058-fcc2-4083-9a2a-69e1f05b057d None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:42:34.642846 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24515058-fcc2-4083-9a2a-69e1f05b057d None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:42:34.644534 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24515058-fcc2-4083-9a2a-69e1f05b057d None admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:42:34.688630 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=100 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:42:34.746529 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 552/1101] 10.222.0.30 () {58 vars in 1282 bytes} [Mon May 23 12:42:34 2022] GET /identity/v3/auth/tokens => generated 2108 bytes in 310 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) May 23 12:42:34.753391 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60546c7d-dfb8-4b22-a06f-46ac39d626f8 None admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:42:34.820683 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=100 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:42:34.854590 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60546c7d-dfb8-4b22-a06f-46ac39d626f8 None admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:42:34.889550 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24d69500-2492-4c13-86f7-602d5be789dc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:42:34.890384 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24d69500-2492-4c13-86f7-602d5be789dc None admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:42:34.890653 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24d69500-2492-4c13-86f7-602d5be789dc None admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:42:34.890924 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24d69500-2492-4c13-86f7-602d5be789dc None admin] PATH_INFO: `/v3/domains/Default` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:42:34.901520 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24d69500-2492-4c13-86f7-602d5be789dc None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:42:34.902983 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24d69500-2492-4c13-86f7-602d5be789dc None admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:42:34.920056 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application [None req-24d69500-2492-4c13-86f7-602d5be789dc None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. May 23 12:42:34.920056 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application Traceback (most recent call last): May 23 12:42:34.920056 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request May 23 12:42:34.920056 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application rv = self.dispatch_request() May 23 12:42:34.920056 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request May 23 12:42:34.920056 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) May 23 12:42:34.920056 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper May 23 12:42:34.920056 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) May 23 12:42:34.920056 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view May 23 12:42:34.920056 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) May 23 12:42:34.920056 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request May 23 12:42:34.920056 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) May 23 12:42:34.920056 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get May 23 12:42:34.920056 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self._get_domain(domain_id) May 23 12:42:34.920056 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain May 23 12:42:34.920959 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) May 23 12:42:34.920959 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped May 23 12:42:34.920959 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) May 23 12:42:34.920959 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun May 23 12:42:34.920959 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) May 23 12:42:34.920959 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func May 23 12:42:34.920959 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self.get_or_create( May 23 12:42:34.920959 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create May 23 12:42:34.920959 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application with Lock( May 23 12:42:34.920959 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ May 23 12:42:34.920959 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self._enter() May 23 12:42:34.920959 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter May 23 12:42:34.920959 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) May 23 12:42:34.920959 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create May 23 12:42:34.920959 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self.creator() May 23 12:42:34.920959 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value May 23 12:42:34.920959 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application created_value = creator( May 23 12:42:34.921933 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain May 23 12:42:34.921933 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) May 23 12:42:34.921933 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. May 23 12:42:34.921933 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application May 23 12:42:34.921933 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 550/1102] 10.222.0.30 () {56 vars in 1108 bytes} [Mon May 23 12:42:34 2022] GET /identity/v3/domains/Default => generated 87 bytes in 170 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) May 23 12:42:34.927447 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24515058-fcc2-4083-9a2a-69e1f05b057d None admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:42:35.025212 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24515058-fcc2-4083-9a2a-69e1f05b057d None admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:42:35.058825 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4228cb0e-a576-4040-bdc1-80b3ac6d76c8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:42:35.059479 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4228cb0e-a576-4040-bdc1-80b3ac6d76c8 None admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:42:35.059767 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4228cb0e-a576-4040-bdc1-80b3ac6d76c8 None admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:42:35.060047 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4228cb0e-a576-4040-bdc1-80b3ac6d76c8 None admin] PATH_INFO: `/v3/domains` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:42:35.063536 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4228cb0e-a576-4040-bdc1-80b3ac6d76c8 None admin] RBAC: Adding query filter params (name=Default) {{(pid=92057) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} May 23 12:42:35.063536 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4228cb0e-a576-4040-bdc1-80b3ac6d76c8 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:42:35.063536 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4228cb0e-a576-4040-bdc1-80b3ac6d76c8 None admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:42:35.074272 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 553/1103] 10.222.0.30 () {56 vars in 1109 bytes} [Mon May 23 12:42:34 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 147 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) May 23 12:42:35.079761 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24d69500-2492-4c13-86f7-602d5be789dc None admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:42:35.178117 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24d69500-2492-4c13-86f7-602d5be789dc None admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:42:35.195318 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=100 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:42:35.213631 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ebd5a426-b900-426a-92ef-05122c500464 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:42:35.214554 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebd5a426-b900-426a-92ef-05122c500464 None admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:42:35.214837 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebd5a426-b900-426a-92ef-05122c500464 None admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:42:35.215110 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebd5a426-b900-426a-92ef-05122c500464 None admin] PATH_INFO: `/v3/projects/service` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:42:35.224024 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ebd5a426-b900-426a-92ef-05122c500464 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:42:35.225461 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ebd5a426-b900-426a-92ef-05122c500464 None admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:42:35.244398 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application [None req-ebd5a426-b900-426a-92ef-05122c500464 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. May 23 12:42:35.244398 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application Traceback (most recent call last): May 23 12:42:35.244398 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request May 23 12:42:35.244398 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application rv = self.dispatch_request() May 23 12:42:35.244398 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request May 23 12:42:35.244398 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) May 23 12:42:35.244398 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper May 23 12:42:35.244398 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) May 23 12:42:35.244398 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view May 23 12:42:35.244398 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) May 23 12:42:35.244398 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request May 23 12:42:35.244398 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) May 23 12:42:35.244398 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get May 23 12:42:35.244398 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self._get_project(project_id) May 23 12:42:35.244398 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project May 23 12:42:35.245414 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) May 23 12:42:35.245414 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped May 23 12:42:35.245414 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) May 23 12:42:35.245414 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun May 23 12:42:35.245414 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) May 23 12:42:35.245414 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func May 23 12:42:35.245414 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self.get_or_create( May 23 12:42:35.245414 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create May 23 12:42:35.245414 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application with Lock( May 23 12:42:35.245414 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ May 23 12:42:35.245414 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self._enter() May 23 12:42:35.245414 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter May 23 12:42:35.245414 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) May 23 12:42:35.245414 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create May 23 12:42:35.245414 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self.creator() May 23 12:42:35.245414 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value May 23 12:42:35.245414 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application created_value = creator( May 23 12:42:35.246376 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project May 23 12:42:35.246376 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) May 23 12:42:35.246376 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project May 23 12:42:35.246376 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() May 23 12:42:35.246376 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project May 23 12:42:35.246376 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) May 23 12:42:35.246376 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. May 23 12:42:35.246376 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application May 23 12:42:35.246376 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 551/1104] 10.222.0.30 () {56 vars in 1111 bytes} [Mon May 23 12:42:35 2022] GET /identity/v3/projects/service => generated 88 bytes in 166 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) May 23 12:42:35.252116 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4228cb0e-a576-4040-bdc1-80b3ac6d76c8 None admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:42:35.256439 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=100 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:42:35.356089 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4228cb0e-a576-4040-bdc1-80b3ac6d76c8 None admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:42:35.388883 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-617ac2c5-1802-4122-8393-ea02c964dd3b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:42:35.389539 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-617ac2c5-1802-4122-8393-ea02c964dd3b None admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:42:35.389820 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-617ac2c5-1802-4122-8393-ea02c964dd3b None admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:42:35.390121 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-617ac2c5-1802-4122-8393-ea02c964dd3b None admin] PATH_INFO: `/v3/projects` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:42:35.390896 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-617ac2c5-1802-4122-8393-ea02c964dd3b None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=92057) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} May 23 12:42:35.391202 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-617ac2c5-1802-4122-8393-ea02c964dd3b None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:42:35.393559 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-617ac2c5-1802-4122-8393-ea02c964dd3b None admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:42:35.405528 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 554/1105] 10.222.0.30 () {56 vars in 1148 bytes} [Mon May 23 12:42:35 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 155 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) May 23 12:42:35.411682 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ebd5a426-b900-426a-92ef-05122c500464 None admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:42:35.511801 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ebd5a426-b900-426a-92ef-05122c500464 None admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:42:35.546759 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-946c75b0-c817-4ecd-a0ae-0fd5edca0299 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:42:35.547415 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-946c75b0-c817-4ecd-a0ae-0fd5edca0299 None admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:42:35.547708 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-946c75b0-c817-4ecd-a0ae-0fd5edca0299 None admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:42:35.547989 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-946c75b0-c817-4ecd-a0ae-0fd5edca0299 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:42:35.549368 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-946c75b0-c817-4ecd-a0ae-0fd5edca0299 None admin] RBAC: Adding query filter params (role.id=940396c67e6744c784e95e01fe84ad45, scope.project.id=9c389ae65aaa4542bb5a5c4fd0bacf5b, user.id=cbe9931d94d44cc7ace0e83a7e5015fc) {{(pid=92056) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} May 23 12:42:35.549682 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-946c75b0-c817-4ecd-a0ae-0fd5edca0299 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:42:35.551153 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-946c75b0-c817-4ecd-a0ae-0fd5edca0299 None admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:42:35.557263 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 552/1106] 10.222.0.30 () {56 vars in 1374 bytes} [Mon May 23 12:42:35 2022] GET /identity/v3/role_assignments?user.id=cbe9931d94d44cc7ace0e83a7e5015fc&scope.project.id=9c389ae65aaa4542bb5a5c4fd0bacf5b&role.id=940396c67e6744c784e95e01fe84ad45 => generated 608 bytes in 146 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) May 23 12:42:36.391134 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c035604-6ed1-4fc0-aff6-b48c71b98631 None None] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:42:36.391648 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c035604-6ed1-4fc0-aff6-b48c71b98631 None None] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:42:36.391757 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c035604-6ed1-4fc0-aff6-b48c71b98631 None None] PATH_INFO: `/` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:42:36.393249 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 555/1107] 10.222.0.30 () {54 vars in 926 bytes} [Mon May 23 12:42:36 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) May 23 12:42:36.398702 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9446bba7-4f38-498e-94cc-86a85630c3d5 None None] REQUEST_METHOD: `POST` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:42:36.399168 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9446bba7-4f38-498e-94cc-86a85630c3d5 None None] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:42:36.399262 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9446bba7-4f38-498e-94cc-86a85630c3d5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:42:36.417536 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.auth.core [None req-9446bba7-4f38-498e-94cc-86a85630c3d5 None None] MFA Rules not processed for user `0f7119f8fa5c4445977671140ded91a4`. Rule list: `[]` (Enabled: `True`). {{(pid=92056) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 12:42:36.418785 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.fernet_utils [None req-9446bba7-4f38-498e-94cc-86a85630c3d5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92056) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 12:42:36.454252 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=100 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:42:36.483707 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 553/1108] 10.222.0.30 () {58 vars in 1023 bytes} [Mon May 23 12:42:36 2022] POST /identity/v3/auth/tokens => generated 2108 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) May 23 12:42:36.509208 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c035604-6ed1-4fc0-aff6-b48c71b98631 None None] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:42:36.571049 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=100 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:42:36.608236 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c035604-6ed1-4fc0-aff6-b48c71b98631 None None] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:42:36.641744 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bffe5184-e459-4c0b-8557-d3ab5ed51dc2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:42:36.642702 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bffe5184-e459-4c0b-8557-d3ab5ed51dc2 None admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:42:36.642985 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bffe5184-e459-4c0b-8557-d3ab5ed51dc2 None admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:42:36.643257 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bffe5184-e459-4c0b-8557-d3ab5ed51dc2 None admin] PATH_INFO: `/v3/domains/default` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:42:36.644399 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bffe5184-e459-4c0b-8557-d3ab5ed51dc2 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:42:36.645831 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bffe5184-e459-4c0b-8557-d3ab5ed51dc2 None admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:42:36.647450 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 556/1109] 10.222.0.30 () {56 vars in 1108 bytes} [Mon May 23 12:42:36 2022] GET /identity/v3/domains/default => generated 199 bytes in 139 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) May 23 12:42:36.653371 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9446bba7-4f38-498e-94cc-86a85630c3d5 None None] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:42:36.749898 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9446bba7-4f38-498e-94cc-86a85630c3d5 None None] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:42:36.782196 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1df82471-7415-4ccf-93d4-8d75237d28b9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:42:36.783433 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1df82471-7415-4ccf-93d4-8d75237d28b9 None admin] REQUEST_METHOD: `POST` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:42:36.783742 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1df82471-7415-4ccf-93d4-8d75237d28b9 None admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:42:36.784019 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1df82471-7415-4ccf-93d4-8d75237d28b9 None admin] PATH_INFO: `/v3/users` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:42:36.784713 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1df82471-7415-4ccf-93d4-8d75237d28b9 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:42:36.786156 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1df82471-7415-4ccf-93d4-8d75237d28b9 None admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:42:36.875059 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 554/1110] 10.222.0.30 () {60 vars in 1132 bytes} [Mon May 23 12:42:36 2022] POST /identity/v3/users => generated 286 bytes in 222 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) May 23 12:42:39.114392 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff291fcf-2055-470a-80e2-5e706c8abd7e None None] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:42:39.114908 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff291fcf-2055-470a-80e2-5e706c8abd7e None None] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:42:39.115002 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff291fcf-2055-470a-80e2-5e706c8abd7e None None] PATH_INFO: `/` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:42:39.116314 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 557/1111] 10.222.0.30 () {54 vars in 926 bytes} [Mon May 23 12:42:39 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) May 23 12:42:39.121590 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efd18f4f-0437-47f0-bfde-599d643c2ffb None None] REQUEST_METHOD: `POST` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:42:39.122034 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efd18f4f-0437-47f0-bfde-599d643c2ffb None None] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:42:39.122172 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efd18f4f-0437-47f0-bfde-599d643c2ffb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:42:39.140240 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.auth.core [None req-efd18f4f-0437-47f0-bfde-599d643c2ffb None None] MFA Rules not processed for user `0f7119f8fa5c4445977671140ded91a4`. Rule list: `[]` (Enabled: `True`). {{(pid=92056) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 12:42:39.141410 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.fernet_utils [None req-efd18f4f-0437-47f0-bfde-599d643c2ffb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92056) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 12:42:39.165344 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:42:39.225782 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 555/1112] 10.222.0.30 () {58 vars in 1023 bytes} [Mon May 23 12:42:39 2022] POST /identity/v3/auth/tokens => generated 2108 bytes in 105 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) May 23 12:42:39.250130 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff291fcf-2055-470a-80e2-5e706c8abd7e None None] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:42:39.350853 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff291fcf-2055-470a-80e2-5e706c8abd7e None None] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:42:39.381956 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=100 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:42:39.386696 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3cfc361-0e6f-4645-9a72-d9b4e5c3250a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:42:39.387548 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3cfc361-0e6f-4645-9a72-d9b4e5c3250a None admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:42:39.387834 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3cfc361-0e6f-4645-9a72-d9b4e5c3250a None admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:42:39.388110 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3cfc361-0e6f-4645-9a72-d9b4e5c3250a None admin] PATH_INFO: `/v3/roles/e3b90948645643e888620ab1e5c774bf` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:42:39.389577 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3cfc361-0e6f-4645-9a72-d9b4e5c3250a None admin] RBAC: Authorizing `identity:get_role(role_id=e3b90948645643e888620ab1e5c774bf)` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:42:39.390999 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3cfc361-0e6f-4645-9a72-d9b4e5c3250a None admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:42:39.392658 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 558/1113] 10.222.0.30 () {56 vars in 1177 bytes} [Mon May 23 12:42:39 2022] GET /identity/v3/roles/e3b90948645643e888620ab1e5c774bf => generated 223 bytes in 143 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) May 23 12:42:39.398322 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-efd18f4f-0437-47f0-bfde-599d643c2ffb None None] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:42:39.496411 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-efd18f4f-0437-47f0-bfde-599d643c2ffb None None] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:42:39.530153 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-328a350d-986f-4e21-85b0-bf95f0d9b9e6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:42:39.531014 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-328a350d-986f-4e21-85b0-bf95f0d9b9e6 None admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:42:39.531298 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-328a350d-986f-4e21-85b0-bf95f0d9b9e6 None admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:42:39.531574 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-328a350d-986f-4e21-85b0-bf95f0d9b9e6 None admin] PATH_INFO: `/v3/users/45375a5ae9f648e0abc2cbf1aeba83d7` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:42:39.544747 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-328a350d-986f-4e21-85b0-bf95f0d9b9e6 None admin] RBAC: Authorizing `identity:get_user(user_id=45375a5ae9f648e0abc2cbf1aeba83d7)` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:42:39.546212 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-328a350d-986f-4e21-85b0-bf95f0d9b9e6 None admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:42:39.547859 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 556/1114] 10.222.0.30 () {56 vars in 1177 bytes} [Mon May 23 12:42:39 2022] GET /identity/v3/users/45375a5ae9f648e0abc2cbf1aeba83d7 => generated 286 bytes in 150 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) May 23 12:42:39.554689 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3cfc361-0e6f-4645-9a72-d9b4e5c3250a None admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:42:39.653773 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3cfc361-0e6f-4645-9a72-d9b4e5c3250a None admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:42:39.689014 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0fec595c-7619-4a2b-9845-34a27552f70e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:42:39.689668 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fec595c-7619-4a2b-9845-34a27552f70e None admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:42:39.690077 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fec595c-7619-4a2b-9845-34a27552f70e None admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:42:39.690351 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fec595c-7619-4a2b-9845-34a27552f70e None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:42:39.755528 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=100 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:42:39.757958 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0fec595c-7619-4a2b-9845-34a27552f70e None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:42:39.759595 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0fec595c-7619-4a2b-9845-34a27552f70e None admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:42:39.856091 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 559/1115] 10.222.0.30 () {58 vars in 1282 bytes} [Mon May 23 12:42:39 2022] GET /identity/v3/auth/tokens => generated 2108 bytes in 302 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) May 23 12:42:39.862818 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-328a350d-986f-4e21-85b0-bf95f0d9b9e6 None admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:42:39.880416 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=100 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:42:39.959865 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-328a350d-986f-4e21-85b0-bf95f0d9b9e6 None admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:42:39.994395 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb790e95-4379-46e2-8be7-d8567704cea0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:42:39.995248 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb790e95-4379-46e2-8be7-d8567704cea0 None admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:42:39.995528 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb790e95-4379-46e2-8be7-d8567704cea0 None admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:42:39.995797 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb790e95-4379-46e2-8be7-d8567704cea0 None admin] PATH_INFO: `/v3/projects/c258ee56e55945e7b3182d1d20762347` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:42:39.997020 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb790e95-4379-46e2-8be7-d8567704cea0 None admin] RBAC: Authorizing `identity:get_project(project_id=c258ee56e55945e7b3182d1d20762347)` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:42:39.998520 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb790e95-4379-46e2-8be7-d8567704cea0 None admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:42:40.000255 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 557/1116] 10.222.0.30 () {56 vars in 1186 bytes} [Mon May 23 12:42:39 2022] GET /identity/v3/projects/c258ee56e55945e7b3182d1d20762347 => generated 311 bytes in 138 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) May 23 12:42:40.006415 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0fec595c-7619-4a2b-9845-34a27552f70e None admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:42:40.100523 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=100 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:42:40.103358 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0fec595c-7619-4a2b-9845-34a27552f70e None admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:42:40.135254 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fad1a584-6538-460e-84ef-a44028e9afd4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:42:40.135896 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fad1a584-6538-460e-84ef-a44028e9afd4 None admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:42:40.136176 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fad1a584-6538-460e-84ef-a44028e9afd4 None admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:42:40.136447 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fad1a584-6538-460e-84ef-a44028e9afd4 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:42:40.137258 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fad1a584-6538-460e-84ef-a44028e9afd4 None admin] RBAC: Adding query filter params (role.id=e3b90948645643e888620ab1e5c774bf, scope.project.id=c258ee56e55945e7b3182d1d20762347, user.id=45375a5ae9f648e0abc2cbf1aeba83d7) {{(pid=92057) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} May 23 12:42:40.137567 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fad1a584-6538-460e-84ef-a44028e9afd4 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:42:40.139035 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fad1a584-6538-460e-84ef-a44028e9afd4 None admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:42:40.144887 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 560/1117] 10.222.0.30 () {56 vars in 1374 bytes} [Mon May 23 12:42:40 2022] GET /identity/v3/role_assignments?user.id=45375a5ae9f648e0abc2cbf1aeba83d7&scope.project.id=c258ee56e55945e7b3182d1d20762347&role.id=e3b90948645643e888620ab1e5c774bf => generated 259 bytes in 139 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) May 23 12:42:42.299104 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1e7d686-624e-42fb-92c1-4a53030f4cb3 None None] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:42:42.299662 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1e7d686-624e-42fb-92c1-4a53030f4cb3 None None] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:42:42.299662 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1e7d686-624e-42fb-92c1-4a53030f4cb3 None None] PATH_INFO: `/` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:42:42.301002 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 558/1118] 10.222.0.30 () {54 vars in 926 bytes} [Mon May 23 12:42:42 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) May 23 12:42:42.306148 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-046c7861-27a5-4ab8-8c09-967292cf2cf0 None None] REQUEST_METHOD: `POST` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:42:42.306541 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-046c7861-27a5-4ab8-8c09-967292cf2cf0 None None] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:42:42.306702 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-046c7861-27a5-4ab8-8c09-967292cf2cf0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:42:42.323795 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.auth.core [None req-046c7861-27a5-4ab8-8c09-967292cf2cf0 None None] MFA Rules not processed for user `0f7119f8fa5c4445977671140ded91a4`. Rule list: `[]` (Enabled: `True`). {{(pid=92057) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 12:42:42.325075 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.fernet_utils [None req-046c7861-27a5-4ab8-8c09-967292cf2cf0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92057) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 12:42:42.390228 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 561/1119] 10.222.0.30 () {58 vars in 1023 bytes} [Mon May 23 12:42:42 2022] POST /identity/v3/auth/tokens => generated 2108 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) May 23 12:42:42.412839 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1e7d686-624e-42fb-92c1-4a53030f4cb3 None None] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:42:42.496046 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=100 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:42:42.512332 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1e7d686-624e-42fb-92c1-4a53030f4cb3 None None] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:42:42.545766 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01c4d1d1-b81d-4ea0-852f-22a8814d1bbd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:42:42.546633 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01c4d1d1-b81d-4ea0-852f-22a8814d1bbd None admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:42:42.546917 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01c4d1d1-b81d-4ea0-852f-22a8814d1bbd None admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:42:42.547197 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01c4d1d1-b81d-4ea0-852f-22a8814d1bbd None admin] PATH_INFO: `/v3/roles/e3b90948645643e888620ab1e5c774bf` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:42:42.549209 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01c4d1d1-b81d-4ea0-852f-22a8814d1bbd None admin] RBAC: Authorizing `identity:get_role(role_id=e3b90948645643e888620ab1e5c774bf)` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:42:42.550561 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01c4d1d1-b81d-4ea0-852f-22a8814d1bbd None admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:42:42.551922 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 559/1120] 10.222.0.30 () {56 vars in 1177 bytes} [Mon May 23 12:42:42 2022] GET /identity/v3/roles/e3b90948645643e888620ab1e5c774bf => generated 223 bytes in 140 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) May 23 12:42:42.557840 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-046c7861-27a5-4ab8-8c09-967292cf2cf0 None None] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:42:42.651296 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=100 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:42:42.658388 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-046c7861-27a5-4ab8-8c09-967292cf2cf0 None None] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:42:42.692133 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9bfa7c1-a317-4bac-bbc3-9b90a7229ecd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:42:42.692996 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9bfa7c1-a317-4bac-bbc3-9b90a7229ecd None admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:42:42.693276 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9bfa7c1-a317-4bac-bbc3-9b90a7229ecd None admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:42:42.693547 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9bfa7c1-a317-4bac-bbc3-9b90a7229ecd None admin] PATH_INFO: `/v3/users/45375a5ae9f648e0abc2cbf1aeba83d7` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:42:42.694749 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9bfa7c1-a317-4bac-bbc3-9b90a7229ecd None admin] RBAC: Authorizing `identity:get_user(user_id=45375a5ae9f648e0abc2cbf1aeba83d7)` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:42:42.696177 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9bfa7c1-a317-4bac-bbc3-9b90a7229ecd None admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:42:42.697789 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 562/1121] 10.222.0.30 () {56 vars in 1177 bytes} [Mon May 23 12:42:42 2022] GET /identity/v3/users/45375a5ae9f648e0abc2cbf1aeba83d7 => generated 286 bytes in 140 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) May 23 12:42:42.703690 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01c4d1d1-b81d-4ea0-852f-22a8814d1bbd None admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:42:42.801469 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01c4d1d1-b81d-4ea0-852f-22a8814d1bbd None admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:42:42.833838 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-519c7149-1196-45b0-92fc-3f931cbd8bc0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:42:42.834703 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-519c7149-1196-45b0-92fc-3f931cbd8bc0 None admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:42:42.834984 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-519c7149-1196-45b0-92fc-3f931cbd8bc0 None admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:42:42.835256 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-519c7149-1196-45b0-92fc-3f931cbd8bc0 None admin] PATH_INFO: `/v3/projects/c258ee56e55945e7b3182d1d20762347` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:42:42.837032 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-519c7149-1196-45b0-92fc-3f931cbd8bc0 None admin] RBAC: Authorizing `identity:get_project(project_id=c258ee56e55945e7b3182d1d20762347)` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:42:42.839688 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-519c7149-1196-45b0-92fc-3f931cbd8bc0 None admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:42:42.842878 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 560/1122] 10.222.0.30 () {56 vars in 1186 bytes} [Mon May 23 12:42:42 2022] GET /identity/v3/projects/c258ee56e55945e7b3182d1d20762347 => generated 311 bytes in 140 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) May 23 12:42:42.850018 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9bfa7c1-a317-4bac-bbc3-9b90a7229ecd None admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:42:42.950582 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9bfa7c1-a317-4bac-bbc3-9b90a7229ecd None admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:42:42.984663 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-210bce01-00fb-45ae-a7dc-4571d20cdc84 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:42:42.985529 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-210bce01-00fb-45ae-a7dc-4571d20cdc84 None admin] REQUEST_METHOD: `PUT` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:42:42.985909 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-210bce01-00fb-45ae-a7dc-4571d20cdc84 None admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:42:42.986665 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-210bce01-00fb-45ae-a7dc-4571d20cdc84 None admin] PATH_INFO: `/v3/projects/c258ee56e55945e7b3182d1d20762347/users/45375a5ae9f648e0abc2cbf1aeba83d7/roles/e3b90948645643e888620ab1e5c774bf` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:42:42.988831 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-210bce01-00fb-45ae-a7dc-4571d20cdc84 None admin] RBAC: Authorizing `identity:create_grant(project_id=c258ee56e55945e7b3182d1d20762347, user_id=45375a5ae9f648e0abc2cbf1aeba83d7, role_id=e3b90948645643e888620ab1e5c774bf)` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:42:42.990302 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-210bce01-00fb-45ae-a7dc-4571d20cdc84 None admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:42:43.058568 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 563/1123] 10.222.0.30 () {58 vars in 1439 bytes} [Mon May 23 12:42:42 2022] PUT /identity/v3/projects/c258ee56e55945e7b3182d1d20762347/users/45375a5ae9f648e0abc2cbf1aeba83d7/roles/e3b90948645643e888620ab1e5c774bf => generated 0 bytes in 209 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) May 23 12:42:45.345720 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6034d2c4-62cf-438b-8197-821ea8e38af0 None None] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:42:45.346283 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6034d2c4-62cf-438b-8197-821ea8e38af0 None None] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:42:45.346397 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6034d2c4-62cf-438b-8197-821ea8e38af0 None None] PATH_INFO: `/` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:42:45.347684 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 561/1124] 10.222.0.30 () {54 vars in 926 bytes} [Mon May 23 12:42:45 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) May 23 12:42:45.353308 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e97da236-ab59-4353-b575-035f87fd5e55 None None] REQUEST_METHOD: `POST` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:42:45.353703 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e97da236-ab59-4353-b575-035f87fd5e55 None None] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:42:45.353954 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e97da236-ab59-4353-b575-035f87fd5e55 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:42:45.371854 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.auth.core [None req-e97da236-ab59-4353-b575-035f87fd5e55 None None] MFA Rules not processed for user `0f7119f8fa5c4445977671140ded91a4`. Rule list: `[]` (Enabled: `True`). {{(pid=92057) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 12:42:45.373189 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.fernet_utils [None req-e97da236-ab59-4353-b575-035f87fd5e55 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92057) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 12:42:45.390794 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:42:45.442226 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 564/1125] 10.222.0.30 () {58 vars in 1023 bytes} [Mon May 23 12:42:45 2022] POST /identity/v3/auth/tokens => generated 2108 bytes in 90 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) May 23 12:42:45.466904 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6034d2c4-62cf-438b-8197-821ea8e38af0 None None] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:42:45.488607 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=100 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:42:45.568195 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6034d2c4-62cf-438b-8197-821ea8e38af0 None None] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:42:45.601623 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79525efc-6833-40cf-8ca7-7f5ce89ab3e5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:42:45.602422 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79525efc-6833-40cf-8ca7-7f5ce89ab3e5 None admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:42:45.602703 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79525efc-6833-40cf-8ca7-7f5ce89ab3e5 None admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:42:45.602974 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79525efc-6833-40cf-8ca7-7f5ce89ab3e5 None admin] PATH_INFO: `/v3/roles/e3b90948645643e888620ab1e5c774bf` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:42:45.604414 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79525efc-6833-40cf-8ca7-7f5ce89ab3e5 None admin] RBAC: Authorizing `identity:get_role(role_id=e3b90948645643e888620ab1e5c774bf)` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:42:45.605771 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79525efc-6833-40cf-8ca7-7f5ce89ab3e5 None admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:42:45.607160 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 562/1126] 10.222.0.30 () {56 vars in 1177 bytes} [Mon May 23 12:42:45 2022] GET /identity/v3/roles/e3b90948645643e888620ab1e5c774bf => generated 223 bytes in 141 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) May 23 12:42:45.613632 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e97da236-ab59-4353-b575-035f87fd5e55 None None] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:42:45.709655 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e97da236-ab59-4353-b575-035f87fd5e55 None None] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:42:45.743553 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-500068d2-be85-41bd-af97-d650227a09a9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:42:45.744463 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-500068d2-be85-41bd-af97-d650227a09a9 None admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:42:45.744765 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-500068d2-be85-41bd-af97-d650227a09a9 None admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:42:45.745062 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-500068d2-be85-41bd-af97-d650227a09a9 None admin] PATH_INFO: `/v3/users/45375a5ae9f648e0abc2cbf1aeba83d7` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:42:45.746344 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-500068d2-be85-41bd-af97-d650227a09a9 None admin] RBAC: Authorizing `identity:get_user(user_id=45375a5ae9f648e0abc2cbf1aeba83d7)` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:42:45.747806 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-500068d2-be85-41bd-af97-d650227a09a9 None admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:42:45.749604 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 565/1127] 10.222.0.30 () {56 vars in 1177 bytes} [Mon May 23 12:42:45 2022] GET /identity/v3/users/45375a5ae9f648e0abc2cbf1aeba83d7 => generated 286 bytes in 136 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) May 23 12:42:45.755446 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79525efc-6833-40cf-8ca7-7f5ce89ab3e5 None admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:42:45.842364 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=100 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:42:45.854989 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79525efc-6833-40cf-8ca7-7f5ce89ab3e5 None admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:42:45.888985 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03d75f38-c28c-4f86-9d1e-67a029f374f0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:42:45.889614 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03d75f38-c28c-4f86-9d1e-67a029f374f0 None admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:42:45.889910 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03d75f38-c28c-4f86-9d1e-67a029f374f0 None admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:42:45.890187 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03d75f38-c28c-4f86-9d1e-67a029f374f0 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:42:45.956599 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03d75f38-c28c-4f86-9d1e-67a029f374f0 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:42:45.958230 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03d75f38-c28c-4f86-9d1e-67a029f374f0 None admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:42:46.043330 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=100 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:42:46.055993 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 563/1128] 10.222.0.30 () {58 vars in 1282 bytes} [Mon May 23 12:42:45 2022] GET /identity/v3/auth/tokens => generated 2108 bytes in 301 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) May 23 12:42:46.061937 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-500068d2-be85-41bd-af97-d650227a09a9 None admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:42:46.083394 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=100 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:42:46.159095 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-500068d2-be85-41bd-af97-d650227a09a9 None admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:42:46.191563 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b16e1b73-49fe-4432-8ac9-82e0f2d37a85 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:42:46.192428 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b16e1b73-49fe-4432-8ac9-82e0f2d37a85 None admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:42:46.192709 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b16e1b73-49fe-4432-8ac9-82e0f2d37a85 None admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:42:46.192979 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b16e1b73-49fe-4432-8ac9-82e0f2d37a85 None admin] PATH_INFO: `/v3/projects/c258ee56e55945e7b3182d1d20762347` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:42:46.194153 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b16e1b73-49fe-4432-8ac9-82e0f2d37a85 None admin] RBAC: Authorizing `identity:get_project(project_id=c258ee56e55945e7b3182d1d20762347)` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:42:46.195584 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b16e1b73-49fe-4432-8ac9-82e0f2d37a85 None admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:42:46.197249 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 566/1129] 10.222.0.30 () {56 vars in 1186 bytes} [Mon May 23 12:42:46 2022] GET /identity/v3/projects/c258ee56e55945e7b3182d1d20762347 => generated 311 bytes in 136 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) May 23 12:42:46.203302 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03d75f38-c28c-4f86-9d1e-67a029f374f0 None admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:42:46.299744 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03d75f38-c28c-4f86-9d1e-67a029f374f0 None admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:42:46.332298 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0839b0d2-a875-44d2-93bc-fa8f4d326133 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:42:46.332930 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0839b0d2-a875-44d2-93bc-fa8f4d326133 None admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:42:46.333208 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0839b0d2-a875-44d2-93bc-fa8f4d326133 None admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:42:46.333481 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0839b0d2-a875-44d2-93bc-fa8f4d326133 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:42:46.334319 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0839b0d2-a875-44d2-93bc-fa8f4d326133 None admin] RBAC: Adding query filter params (role.id=e3b90948645643e888620ab1e5c774bf, scope.project.id=c258ee56e55945e7b3182d1d20762347, user.id=45375a5ae9f648e0abc2cbf1aeba83d7) {{(pid=92056) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} May 23 12:42:46.334628 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0839b0d2-a875-44d2-93bc-fa8f4d326133 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:42:46.336039 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0839b0d2-a875-44d2-93bc-fa8f4d326133 None admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:42:46.341727 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 564/1130] 10.222.0.30 () {56 vars in 1374 bytes} [Mon May 23 12:42:46 2022] GET /identity/v3/role_assignments?user.id=45375a5ae9f648e0abc2cbf1aeba83d7&scope.project.id=c258ee56e55945e7b3182d1d20762347&role.id=e3b90948645643e888620ab1e5c774bf => generated 608 bytes in 139 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) May 23 12:42:48.553102 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1d8d750-04b2-4ef4-8b64-603639325a3f None None] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:42:48.553661 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1d8d750-04b2-4ef4-8b64-603639325a3f None None] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:42:48.553661 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1d8d750-04b2-4ef4-8b64-603639325a3f None None] PATH_INFO: `/` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:42:48.555064 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 567/1131] 10.222.0.30 () {54 vars in 926 bytes} [Mon May 23 12:42:48 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) May 23 12:42:48.560387 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34259a78-f393-422d-9645-e3f968c6a324 None None] REQUEST_METHOD: `POST` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:42:48.560791 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34259a78-f393-422d-9645-e3f968c6a324 None None] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:42:48.560949 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34259a78-f393-422d-9645-e3f968c6a324 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:42:48.578608 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.auth.core [None req-34259a78-f393-422d-9645-e3f968c6a324 None None] MFA Rules not processed for user `0f7119f8fa5c4445977671140ded91a4`. Rule list: `[]` (Enabled: `True`). {{(pid=92056) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 12:42:48.580831 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.fernet_utils [None req-34259a78-f393-422d-9645-e3f968c6a324 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92056) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 12:42:48.628359 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=100 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:42:48.649271 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 565/1132] 10.222.0.30 () {58 vars in 1023 bytes} [Mon May 23 12:42:48 2022] POST /identity/v3/auth/tokens => generated 2108 bytes in 90 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) May 23 12:42:48.673386 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1d8d750-04b2-4ef4-8b64-603639325a3f None None] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:42:48.759900 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=100 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:42:48.772636 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1d8d750-04b2-4ef4-8b64-603639325a3f None None] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:42:48.807382 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a743af42-b68a-475d-a402-93155bfa865b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:42:48.808100 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a743af42-b68a-475d-a402-93155bfa865b None admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:42:48.808385 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a743af42-b68a-475d-a402-93155bfa865b None admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:42:48.808662 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a743af42-b68a-475d-a402-93155bfa865b None admin] PATH_INFO: `/v3/domains/default` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:42:48.809929 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a743af42-b68a-475d-a402-93155bfa865b None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:42:48.811450 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a743af42-b68a-475d-a402-93155bfa865b None admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:42:48.813180 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 568/1133] 10.222.0.30 () {56 vars in 1108 bytes} [Mon May 23 12:42:48 2022] GET /identity/v3/domains/default => generated 199 bytes in 140 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) May 23 12:42:48.819751 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34259a78-f393-422d-9645-e3f968c6a324 None None] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:42:48.919218 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34259a78-f393-422d-9645-e3f968c6a324 None None] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:42:48.952494 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc7c2aeb-f67a-4433-bf3d-222682dd9034 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:42:48.953134 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc7c2aeb-f67a-4433-bf3d-222682dd9034 None admin] REQUEST_METHOD: `POST` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:42:48.953413 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc7c2aeb-f67a-4433-bf3d-222682dd9034 None admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:42:48.953685 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc7c2aeb-f67a-4433-bf3d-222682dd9034 None admin] PATH_INFO: `/v3/projects` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:42:48.955367 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc7c2aeb-f67a-4433-bf3d-222682dd9034 None admin] RBAC: Authorizing `identity:create_project()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:42:48.956803 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc7c2aeb-f67a-4433-bf3d-222682dd9034 None admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:42:49.024023 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 566/1134] 10.222.0.30 () {60 vars in 1141 bytes} [Mon May 23 12:42:48 2022] POST /identity/v3/projects => generated 311 bytes in 205 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) May 23 12:42:51.238591 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a11923a-9de9-46fc-b14d-843ede709757 None None] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:42:51.239118 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a11923a-9de9-46fc-b14d-843ede709757 None None] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:42:51.239118 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a11923a-9de9-46fc-b14d-843ede709757 None None] PATH_INFO: `/` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:42:51.240487 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 569/1135] 10.222.0.30 () {54 vars in 926 bytes} [Mon May 23 12:42:51 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) May 23 12:42:51.245621 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76fcb7e1-b2d5-40e2-b759-ec36d5fbb09a None None] REQUEST_METHOD: `POST` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:42:51.246084 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76fcb7e1-b2d5-40e2-b759-ec36d5fbb09a None None] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:42:51.246202 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76fcb7e1-b2d5-40e2-b759-ec36d5fbb09a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:42:51.262750 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.auth.core [None req-76fcb7e1-b2d5-40e2-b759-ec36d5fbb09a None None] MFA Rules not processed for user `0f7119f8fa5c4445977671140ded91a4`. Rule list: `[]` (Enabled: `True`). {{(pid=92056) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 12:42:51.263909 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.fernet_utils [None req-76fcb7e1-b2d5-40e2-b759-ec36d5fbb09a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92056) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 12:42:51.290612 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:42:51.329817 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 567/1136] 10.222.0.30 () {58 vars in 1023 bytes} [Mon May 23 12:42:51 2022] POST /identity/v3/auth/tokens => generated 2108 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) May 23 12:42:51.355267 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a11923a-9de9-46fc-b14d-843ede709757 None None] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:42:51.452194 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a11923a-9de9-46fc-b14d-843ede709757 None None] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:42:51.485759 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92cc3b92-8511-4291-9778-992d3653e6e9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:42:51.486556 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92cc3b92-8511-4291-9778-992d3653e6e9 None admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:42:51.486837 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92cc3b92-8511-4291-9778-992d3653e6e9 None admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:42:51.487110 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92cc3b92-8511-4291-9778-992d3653e6e9 None admin] PATH_INFO: `/v3/domains/default` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:42:51.488259 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92cc3b92-8511-4291-9778-992d3653e6e9 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:42:51.489742 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92cc3b92-8511-4291-9778-992d3653e6e9 None admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:42:51.491319 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 570/1137] 10.222.0.30 () {56 vars in 1108 bytes} [Mon May 23 12:42:51 2022] GET /identity/v3/domains/default => generated 199 bytes in 137 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) May 23 12:42:51.496873 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76fcb7e1-b2d5-40e2-b759-ec36d5fbb09a None None] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:42:51.591986 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76fcb7e1-b2d5-40e2-b759-ec36d5fbb09a None None] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:42:51.624984 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54d97818-0fc9-4c98-a891-a369d7208011 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:42:51.625631 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54d97818-0fc9-4c98-a891-a369d7208011 None admin] REQUEST_METHOD: `POST` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:42:51.626570 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54d97818-0fc9-4c98-a891-a369d7208011 None admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:42:51.626978 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54d97818-0fc9-4c98-a891-a369d7208011 None admin] PATH_INFO: `/v3/users` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:42:51.627685 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54d97818-0fc9-4c98-a891-a369d7208011 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:42:51.629102 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54d97818-0fc9-4c98-a891-a369d7208011 None admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:42:51.713679 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 568/1138] 10.222.0.30 () {60 vars in 1132 bytes} [Mon May 23 12:42:51 2022] POST /identity/v3/users => generated 286 bytes in 217 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) May 23 12:42:53.976620 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfa560bc-4eed-42f2-ba51-d1c53bd3d1e3 None None] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:42:53.977135 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfa560bc-4eed-42f2-ba51-d1c53bd3d1e3 None None] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:42:53.977135 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfa560bc-4eed-42f2-ba51-d1c53bd3d1e3 None None] PATH_INFO: `/` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:42:53.978497 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 571/1139] 10.222.0.30 () {54 vars in 926 bytes} [Mon May 23 12:42:53 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) May 23 12:42:53.983883 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27909718-56b2-482c-95d5-b70165272668 None None] REQUEST_METHOD: `POST` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:42:53.984274 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27909718-56b2-482c-95d5-b70165272668 None None] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:42:53.984432 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27909718-56b2-482c-95d5-b70165272668 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:42:54.001583 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.auth.core [None req-27909718-56b2-482c-95d5-b70165272668 None None] MFA Rules not processed for user `0f7119f8fa5c4445977671140ded91a4`. Rule list: `[]` (Enabled: `True`). {{(pid=92056) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 12:42:54.002780 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.fernet_utils [None req-27909718-56b2-482c-95d5-b70165272668 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92056) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 12:42:54.016255 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:42:54.068213 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 569/1140] 10.222.0.30 () {58 vars in 1023 bytes} [Mon May 23 12:42:53 2022] POST /identity/v3/auth/tokens => generated 2108 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) May 23 12:42:54.091662 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dfa560bc-4eed-42f2-ba51-d1c53bd3d1e3 None None] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:42:54.117018 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=100 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:42:54.191719 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dfa560bc-4eed-42f2-ba51-d1c53bd3d1e3 None None] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:42:54.224551 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-580168c9-ec86-452e-8431-47768947cb20 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:42:54.225319 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-580168c9-ec86-452e-8431-47768947cb20 None admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:42:54.225639 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-580168c9-ec86-452e-8431-47768947cb20 None admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:42:54.225978 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-580168c9-ec86-452e-8431-47768947cb20 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:42:54.235867 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-580168c9-ec86-452e-8431-47768947cb20 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} May 23 12:42:54.243389 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application [None req-580168c9-ec86-452e-8431-47768947cb20 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. May 23 12:42:54.243389 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application Traceback (most recent call last): May 23 12:42:54.243389 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request May 23 12:42:54.243389 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application rv = self.dispatch_request() May 23 12:42:54.243389 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request May 23 12:42:54.243389 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) May 23 12:42:54.243389 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper May 23 12:42:54.243389 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) May 23 12:42:54.243389 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view May 23 12:42:54.243389 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) May 23 12:42:54.243389 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request May 23 12:42:54.243389 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) May 23 12:42:54.243389 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get May 23 12:42:54.243389 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self._get_role(role_id) May 23 12:42:54.243389 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role May 23 12:42:54.243389 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') May 23 12:42:54.244501 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call May 23 12:42:54.244501 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( May 23 12:42:54.244501 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data May 23 12:42:54.244501 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) May 23 12:42:54.244501 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped May 23 12:42:54.244501 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) May 23 12:42:54.244501 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun May 23 12:42:54.244501 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) May 23 12:42:54.244501 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func May 23 12:42:54.244501 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self.get_or_create( May 23 12:42:54.244501 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create May 23 12:42:54.244501 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application with Lock( May 23 12:42:54.244501 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ May 23 12:42:54.244501 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self._enter() May 23 12:42:54.244501 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter May 23 12:42:54.244501 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) May 23 12:42:54.245574 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create May 23 12:42:54.245574 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self.creator() May 23 12:42:54.245574 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value May 23 12:42:54.245574 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application created_value = creator( May 23 12:42:54.245574 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role May 23 12:42:54.245574 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) May 23 12:42:54.245574 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role May 23 12:42:54.245574 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() May 23 12:42:54.245574 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role May 23 12:42:54.245574 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) May 23 12:42:54.245574 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. May 23 12:42:54.245574 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application May 23 12:42:54.245574 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 572/1141] 10.222.0.30 () {56 vars in 1096 bytes} [Mon May 23 12:42:54 2022] GET /identity/v3/roles/admin => generated 83 bytes in 153 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) May 23 12:42:54.250636 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27909718-56b2-482c-95d5-b70165272668 None None] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:42:54.345391 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27909718-56b2-482c-95d5-b70165272668 None None] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:42:54.377921 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0164d2c-2e7c-465b-90c4-86545fd3776a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:42:54.378562 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0164d2c-2e7c-465b-90c4-86545fd3776a None admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:42:54.378841 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0164d2c-2e7c-465b-90c4-86545fd3776a None admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:42:54.379113 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0164d2c-2e7c-465b-90c4-86545fd3776a None admin] PATH_INFO: `/v3/roles` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:42:54.379849 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0164d2c-2e7c-465b-90c4-86545fd3776a None admin] RBAC: Adding query filter params (name=admin) {{(pid=92056) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} May 23 12:42:54.380154 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0164d2c-2e7c-465b-90c4-86545fd3776a None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:42:54.381523 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0164d2c-2e7c-465b-90c4-86545fd3776a None admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:42:54.389187 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 570/1142] 10.222.0.30 () {56 vars in 1099 bytes} [Mon May 23 12:42:54 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 139 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) May 23 12:42:54.395854 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-580168c9-ec86-452e-8431-47768947cb20 None admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:42:54.480168 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=100 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:42:54.495528 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-580168c9-ec86-452e-8431-47768947cb20 None admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:42:54.528166 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d83b4ed-bb82-4936-ab5a-9fdd58625772 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:42:54.529019 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d83b4ed-bb82-4936-ab5a-9fdd58625772 None admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:42:54.529303 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d83b4ed-bb82-4936-ab5a-9fdd58625772 None admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:42:54.529579 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d83b4ed-bb82-4936-ab5a-9fdd58625772 None admin] PATH_INFO: `/v3/users/fd19c9a870d24fd58f81be06718d032e` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:42:54.543493 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d83b4ed-bb82-4936-ab5a-9fdd58625772 None admin] RBAC: Authorizing `identity:get_user(user_id=fd19c9a870d24fd58f81be06718d032e)` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:42:54.544943 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d83b4ed-bb82-4936-ab5a-9fdd58625772 None admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:42:54.546608 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 573/1143] 10.222.0.30 () {56 vars in 1177 bytes} [Mon May 23 12:42:54 2022] GET /identity/v3/users/fd19c9a870d24fd58f81be06718d032e => generated 286 bytes in 151 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) May 23 12:42:54.552414 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0164d2c-2e7c-465b-90c4-86545fd3776a None admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:42:54.568393 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=100 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:42:54.651490 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0164d2c-2e7c-465b-90c4-86545fd3776a None admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:42:54.685586 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a60f5c9a-1da8-4d2c-98a2-a55819bc443d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:42:54.686262 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a60f5c9a-1da8-4d2c-98a2-a55819bc443d None admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:42:54.686544 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a60f5c9a-1da8-4d2c-98a2-a55819bc443d None admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:42:54.686818 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a60f5c9a-1da8-4d2c-98a2-a55819bc443d None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:42:54.754234 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a60f5c9a-1da8-4d2c-98a2-a55819bc443d None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:42:54.755652 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a60f5c9a-1da8-4d2c-98a2-a55819bc443d None admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:42:54.769126 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=100 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:42:54.855141 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 571/1144] 10.222.0.30 () {58 vars in 1282 bytes} [Mon May 23 12:42:54 2022] GET /identity/v3/auth/tokens => generated 2108 bytes in 303 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) May 23 12:42:54.861271 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d83b4ed-bb82-4936-ab5a-9fdd58625772 None admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:42:54.960824 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d83b4ed-bb82-4936-ab5a-9fdd58625772 None admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:42:54.994442 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d76164bf-86c3-4802-aabb-ed0e6b6fb640 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:42:54.995290 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d76164bf-86c3-4802-aabb-ed0e6b6fb640 None admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:42:54.995571 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d76164bf-86c3-4802-aabb-ed0e6b6fb640 None admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:42:54.995843 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d76164bf-86c3-4802-aabb-ed0e6b6fb640 None admin] PATH_INFO: `/v3/projects/2cfa8fc1578b4c099c1ce415a0266e74` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:42:54.996996 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d76164bf-86c3-4802-aabb-ed0e6b6fb640 None admin] RBAC: Authorizing `identity:get_project(project_id=2cfa8fc1578b4c099c1ce415a0266e74)` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:42:54.998471 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d76164bf-86c3-4802-aabb-ed0e6b6fb640 None admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:42:55.000141 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 574/1145] 10.222.0.30 () {56 vars in 1186 bytes} [Mon May 23 12:42:54 2022] GET /identity/v3/projects/2cfa8fc1578b4c099c1ce415a0266e74 => generated 311 bytes in 139 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) May 23 12:42:55.006439 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a60f5c9a-1da8-4d2c-98a2-a55819bc443d None admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:42:55.104007 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a60f5c9a-1da8-4d2c-98a2-a55819bc443d None admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:42:55.116998 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=100 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:42:55.136804 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3fe66c82-c394-4ed5-86f1-806532681b85 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:42:55.137489 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fe66c82-c394-4ed5-86f1-806532681b85 None admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:42:55.137774 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fe66c82-c394-4ed5-86f1-806532681b85 None admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:42:55.138077 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fe66c82-c394-4ed5-86f1-806532681b85 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:42:55.138897 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3fe66c82-c394-4ed5-86f1-806532681b85 None admin] RBAC: Adding query filter params (role.id=bc3c3c9d654940cbaa5177a6be23d1bc, scope.project.id=2cfa8fc1578b4c099c1ce415a0266e74, user.id=fd19c9a870d24fd58f81be06718d032e) {{(pid=92056) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} May 23 12:42:55.139205 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3fe66c82-c394-4ed5-86f1-806532681b85 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:42:55.140618 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3fe66c82-c394-4ed5-86f1-806532681b85 None admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:42:55.146444 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 572/1146] 10.222.0.30 () {56 vars in 1374 bytes} [Mon May 23 12:42:55 2022] GET /identity/v3/role_assignments?user.id=fd19c9a870d24fd58f81be06718d032e&scope.project.id=2cfa8fc1578b4c099c1ce415a0266e74&role.id=bc3c3c9d654940cbaa5177a6be23d1bc => generated 259 bytes in 141 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) May 23 12:42:57.290214 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2a7982e-4a94-4652-9e57-bfea25df9fa1 None None] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:42:57.290781 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2a7982e-4a94-4652-9e57-bfea25df9fa1 None None] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:42:57.290781 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2a7982e-4a94-4652-9e57-bfea25df9fa1 None None] PATH_INFO: `/` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:42:57.292112 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 575/1147] 10.222.0.30 () {54 vars in 926 bytes} [Mon May 23 12:42:57 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) May 23 12:42:57.297203 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af9e4a5b-a26d-482f-ad66-154e67e9216a None None] REQUEST_METHOD: `POST` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:42:57.297579 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af9e4a5b-a26d-482f-ad66-154e67e9216a None None] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:42:57.297759 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af9e4a5b-a26d-482f-ad66-154e67e9216a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:42:57.315429 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.auth.core [None req-af9e4a5b-a26d-482f-ad66-154e67e9216a None None] MFA Rules not processed for user `0f7119f8fa5c4445977671140ded91a4`. Rule list: `[]` (Enabled: `True`). {{(pid=92056) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 12:42:57.316601 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.fernet_utils [None req-af9e4a5b-a26d-482f-ad66-154e67e9216a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92056) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 12:42:57.380914 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 573/1148] 10.222.0.30 () {58 vars in 1023 bytes} [Mon May 23 12:42:57 2022] POST /identity/v3/auth/tokens => generated 2108 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) May 23 12:42:57.403206 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2a7982e-4a94-4652-9e57-bfea25df9fa1 None None] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:42:57.415149 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=100 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:42:57.502368 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2a7982e-4a94-4652-9e57-bfea25df9fa1 None None] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:42:57.535049 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b497c241-dddb-4f46-8519-1da233cd597a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:42:57.535818 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b497c241-dddb-4f46-8519-1da233cd597a None admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:42:57.536097 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b497c241-dddb-4f46-8519-1da233cd597a None admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:42:57.536369 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b497c241-dddb-4f46-8519-1da233cd597a None admin] PATH_INFO: `/v3/roles/admin` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:42:57.546096 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b497c241-dddb-4f46-8519-1da233cd597a None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} May 23 12:42:57.554087 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application [None req-b497c241-dddb-4f46-8519-1da233cd597a None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. May 23 12:42:57.554087 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application Traceback (most recent call last): May 23 12:42:57.554087 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request May 23 12:42:57.554087 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application rv = self.dispatch_request() May 23 12:42:57.554087 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request May 23 12:42:57.554087 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) May 23 12:42:57.554087 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper May 23 12:42:57.554087 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) May 23 12:42:57.554087 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view May 23 12:42:57.554087 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) May 23 12:42:57.554087 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request May 23 12:42:57.554087 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) May 23 12:42:57.554087 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get May 23 12:42:57.554087 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self._get_role(role_id) May 23 12:42:57.554087 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role May 23 12:42:57.554087 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') May 23 12:42:57.555196 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call May 23 12:42:57.555196 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( May 23 12:42:57.555196 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data May 23 12:42:57.555196 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) May 23 12:42:57.555196 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped May 23 12:42:57.555196 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) May 23 12:42:57.555196 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun May 23 12:42:57.555196 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) May 23 12:42:57.555196 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func May 23 12:42:57.555196 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self.get_or_create( May 23 12:42:57.555196 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create May 23 12:42:57.555196 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application with Lock( May 23 12:42:57.555196 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ May 23 12:42:57.555196 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self._enter() May 23 12:42:57.555196 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter May 23 12:42:57.555196 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) May 23 12:42:57.556243 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create May 23 12:42:57.556243 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self.creator() May 23 12:42:57.556243 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value May 23 12:42:57.556243 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application created_value = creator( May 23 12:42:57.556243 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role May 23 12:42:57.556243 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) May 23 12:42:57.556243 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role May 23 12:42:57.556243 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() May 23 12:42:57.556243 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role May 23 12:42:57.556243 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) May 23 12:42:57.556243 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. May 23 12:42:57.556243 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application May 23 12:42:57.556243 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 576/1149] 10.222.0.30 () {56 vars in 1096 bytes} [Mon May 23 12:42:57 2022] GET /identity/v3/roles/admin => generated 83 bytes in 153 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) May 23 12:42:57.561273 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af9e4a5b-a26d-482f-ad66-154e67e9216a None None] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:42:57.656688 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af9e4a5b-a26d-482f-ad66-154e67e9216a None None] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:42:57.664463 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=100 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:42:57.691458 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38f27a3c-ae0f-4726-9d44-985fbd97bd58 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:42:57.691458 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38f27a3c-ae0f-4726-9d44-985fbd97bd58 None admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:42:57.691458 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38f27a3c-ae0f-4726-9d44-985fbd97bd58 None admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:42:57.691871 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38f27a3c-ae0f-4726-9d44-985fbd97bd58 None admin] PATH_INFO: `/v3/roles` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:42:57.692632 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38f27a3c-ae0f-4726-9d44-985fbd97bd58 None admin] RBAC: Adding query filter params (name=admin) {{(pid=92056) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} May 23 12:42:57.693211 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38f27a3c-ae0f-4726-9d44-985fbd97bd58 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:42:57.694759 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38f27a3c-ae0f-4726-9d44-985fbd97bd58 None admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:42:57.702928 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 574/1150] 10.222.0.30 () {56 vars in 1099 bytes} [Mon May 23 12:42:57 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 142 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) May 23 12:42:57.709053 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b497c241-dddb-4f46-8519-1da233cd597a None admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:42:57.778018 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=100 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:42:57.808883 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b497c241-dddb-4f46-8519-1da233cd597a None admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:42:57.840612 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba5d116b-4c5f-47fe-be00-e0a55a0b0765 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:42:57.841505 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba5d116b-4c5f-47fe-be00-e0a55a0b0765 None admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:42:57.841821 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba5d116b-4c5f-47fe-be00-e0a55a0b0765 None admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:42:57.842120 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba5d116b-4c5f-47fe-be00-e0a55a0b0765 None admin] PATH_INFO: `/v3/users/fd19c9a870d24fd58f81be06718d032e` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:42:57.843286 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba5d116b-4c5f-47fe-be00-e0a55a0b0765 None admin] RBAC: Authorizing `identity:get_user(user_id=fd19c9a870d24fd58f81be06718d032e)` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:42:57.844715 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba5d116b-4c5f-47fe-be00-e0a55a0b0765 None admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:42:57.847051 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 577/1151] 10.222.0.30 () {56 vars in 1177 bytes} [Mon May 23 12:42:57 2022] GET /identity/v3/users/fd19c9a870d24fd58f81be06718d032e => generated 286 bytes in 138 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) May 23 12:42:57.852701 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38f27a3c-ae0f-4726-9d44-985fbd97bd58 None admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:42:57.952935 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38f27a3c-ae0f-4726-9d44-985fbd97bd58 None admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:42:57.985464 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0fcf55b5-d06a-4d8e-9ba8-bbc8f4157f00 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:42:57.987265 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fcf55b5-d06a-4d8e-9ba8-bbc8f4157f00 None admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:42:57.987543 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fcf55b5-d06a-4d8e-9ba8-bbc8f4157f00 None admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:42:57.987812 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fcf55b5-d06a-4d8e-9ba8-bbc8f4157f00 None admin] PATH_INFO: `/v3/projects/2cfa8fc1578b4c099c1ce415a0266e74` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:42:57.988988 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0fcf55b5-d06a-4d8e-9ba8-bbc8f4157f00 None admin] RBAC: Authorizing `identity:get_project(project_id=2cfa8fc1578b4c099c1ce415a0266e74)` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:42:57.990452 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0fcf55b5-d06a-4d8e-9ba8-bbc8f4157f00 None admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:42:57.992123 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 575/1152] 10.222.0.30 () {56 vars in 1186 bytes} [Mon May 23 12:42:57 2022] GET /identity/v3/projects/2cfa8fc1578b4c099c1ce415a0266e74 => generated 311 bytes in 140 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) May 23 12:42:57.998703 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba5d116b-4c5f-47fe-be00-e0a55a0b0765 None admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:42:58.100750 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba5d116b-4c5f-47fe-be00-e0a55a0b0765 None admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:42:58.134457 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2af82132-9deb-4969-ace9-c1dffaa250e6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:42:58.135322 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2af82132-9deb-4969-ace9-c1dffaa250e6 None admin] REQUEST_METHOD: `PUT` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:42:58.135607 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2af82132-9deb-4969-ace9-c1dffaa250e6 None admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:42:58.135884 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2af82132-9deb-4969-ace9-c1dffaa250e6 None admin] PATH_INFO: `/v3/projects/2cfa8fc1578b4c099c1ce415a0266e74/users/fd19c9a870d24fd58f81be06718d032e/roles/bc3c3c9d654940cbaa5177a6be23d1bc` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:42:58.137821 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2af82132-9deb-4969-ace9-c1dffaa250e6 None admin] RBAC: Authorizing `identity:create_grant(project_id=2cfa8fc1578b4c099c1ce415a0266e74, user_id=fd19c9a870d24fd58f81be06718d032e, role_id=bc3c3c9d654940cbaa5177a6be23d1bc)` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:42:58.139284 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2af82132-9deb-4969-ace9-c1dffaa250e6 None admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:42:58.188906 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:42:58.195165 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 578/1153] 10.222.0.30 () {58 vars in 1439 bytes} [Mon May 23 12:42:57 2022] PUT /identity/v3/projects/2cfa8fc1578b4c099c1ce415a0266e74/users/fd19c9a870d24fd58f81be06718d032e/roles/bc3c3c9d654940cbaa5177a6be23d1bc => generated 0 bytes in 198 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) May 23 12:43:00.387334 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd1e517a-6483-4431-b315-cbb8fde17457 None None] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:43:00.387919 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd1e517a-6483-4431-b315-cbb8fde17457 None None] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:43:00.387919 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd1e517a-6483-4431-b315-cbb8fde17457 None None] PATH_INFO: `/` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:43:00.389234 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 576/1154] 10.222.0.30 () {54 vars in 926 bytes} [Mon May 23 12:43:00 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) May 23 12:43:00.394913 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fea93cd7-0f72-4ff8-b9bb-e167dc1b71b2 None None] REQUEST_METHOD: `POST` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:43:00.395350 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fea93cd7-0f72-4ff8-b9bb-e167dc1b71b2 None None] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:43:00.395452 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fea93cd7-0f72-4ff8-b9bb-e167dc1b71b2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:43:00.412774 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.auth.core [None req-fea93cd7-0f72-4ff8-b9bb-e167dc1b71b2 None None] MFA Rules not processed for user `0f7119f8fa5c4445977671140ded91a4`. Rule list: `[]` (Enabled: `True`). {{(pid=92057) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 12:43:00.413988 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.fernet_utils [None req-fea93cd7-0f72-4ff8-b9bb-e167dc1b71b2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92057) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 12:43:00.478705 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 579/1155] 10.222.0.30 () {58 vars in 1023 bytes} [Mon May 23 12:43:00 2022] POST /identity/v3/auth/tokens => generated 2108 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) May 23 12:43:00.502056 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd1e517a-6483-4431-b315-cbb8fde17457 None None] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:43:00.538188 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=100 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:43:00.601849 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd1e517a-6483-4431-b315-cbb8fde17457 None None] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:43:00.634795 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f587f3c5-3f5d-46c6-83df-3a468c82982d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:43:00.635645 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f587f3c5-3f5d-46c6-83df-3a468c82982d None admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:43:00.635924 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f587f3c5-3f5d-46c6-83df-3a468c82982d None admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:43:00.636195 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f587f3c5-3f5d-46c6-83df-3a468c82982d None admin] PATH_INFO: `/v3/roles/admin` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:43:00.647032 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f587f3c5-3f5d-46c6-83df-3a468c82982d None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} May 23 12:43:00.654680 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application [None req-f587f3c5-3f5d-46c6-83df-3a468c82982d None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. May 23 12:43:00.654680 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application Traceback (most recent call last): May 23 12:43:00.654680 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request May 23 12:43:00.654680 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application rv = self.dispatch_request() May 23 12:43:00.654680 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request May 23 12:43:00.654680 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) May 23 12:43:00.654680 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper May 23 12:43:00.654680 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) May 23 12:43:00.654680 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view May 23 12:43:00.654680 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) May 23 12:43:00.654680 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request May 23 12:43:00.654680 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) May 23 12:43:00.654680 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get May 23 12:43:00.654680 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self._get_role(role_id) May 23 12:43:00.654680 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role May 23 12:43:00.654680 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') May 23 12:43:00.655873 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call May 23 12:43:00.655873 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( May 23 12:43:00.655873 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data May 23 12:43:00.655873 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) May 23 12:43:00.655873 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped May 23 12:43:00.655873 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) May 23 12:43:00.655873 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun May 23 12:43:00.655873 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) May 23 12:43:00.655873 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func May 23 12:43:00.655873 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self.get_or_create( May 23 12:43:00.655873 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create May 23 12:43:00.655873 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application with Lock( May 23 12:43:00.655873 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ May 23 12:43:00.655873 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self._enter() May 23 12:43:00.655873 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter May 23 12:43:00.655873 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) May 23 12:43:00.657146 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create May 23 12:43:00.657146 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self.creator() May 23 12:43:00.657146 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value May 23 12:43:00.657146 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application created_value = creator( May 23 12:43:00.657146 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role May 23 12:43:00.657146 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) May 23 12:43:00.657146 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role May 23 12:43:00.657146 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() May 23 12:43:00.657146 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role May 23 12:43:00.657146 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) May 23 12:43:00.657146 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. May 23 12:43:00.657146 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application May 23 12:43:00.657146 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 577/1156] 10.222.0.30 () {56 vars in 1096 bytes} [Mon May 23 12:43:00 2022] GET /identity/v3/roles/admin => generated 83 bytes in 154 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) May 23 12:43:00.661749 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fea93cd7-0f72-4ff8-b9bb-e167dc1b71b2 None None] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:43:00.756936 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fea93cd7-0f72-4ff8-b9bb-e167dc1b71b2 None None] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:43:00.782796 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=100 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:43:00.790405 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2caeb76a-43e2-4342-9fdf-f1cc55cb5ccc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:43:00.791076 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2caeb76a-43e2-4342-9fdf-f1cc55cb5ccc None admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:43:00.791362 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2caeb76a-43e2-4342-9fdf-f1cc55cb5ccc None admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:43:00.791640 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2caeb76a-43e2-4342-9fdf-f1cc55cb5ccc None admin] PATH_INFO: `/v3/roles` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:43:00.792383 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2caeb76a-43e2-4342-9fdf-f1cc55cb5ccc None admin] RBAC: Adding query filter params (name=admin) {{(pid=92057) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} May 23 12:43:00.792690 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2caeb76a-43e2-4342-9fdf-f1cc55cb5ccc None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:43:00.794136 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2caeb76a-43e2-4342-9fdf-f1cc55cb5ccc None admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:43:00.802277 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 580/1157] 10.222.0.30 () {56 vars in 1099 bytes} [Mon May 23 12:43:00 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 141 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) May 23 12:43:00.808183 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f587f3c5-3f5d-46c6-83df-3a468c82982d None admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:43:00.906291 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=100 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:43:00.912314 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f587f3c5-3f5d-46c6-83df-3a468c82982d None admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:43:00.946838 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10c416c5-4e10-44d3-b47a-2e100dccc4e9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:43:00.947697 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10c416c5-4e10-44d3-b47a-2e100dccc4e9 None admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:43:00.947983 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10c416c5-4e10-44d3-b47a-2e100dccc4e9 None admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:43:00.948262 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10c416c5-4e10-44d3-b47a-2e100dccc4e9 None admin] PATH_INFO: `/v3/users/fd19c9a870d24fd58f81be06718d032e` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:43:00.949448 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10c416c5-4e10-44d3-b47a-2e100dccc4e9 None admin] RBAC: Authorizing `identity:get_user(user_id=fd19c9a870d24fd58f81be06718d032e)` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:43:00.950907 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10c416c5-4e10-44d3-b47a-2e100dccc4e9 None admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:43:00.952543 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 578/1158] 10.222.0.30 () {56 vars in 1177 bytes} [Mon May 23 12:43:00 2022] GET /identity/v3/users/fd19c9a870d24fd58f81be06718d032e => generated 286 bytes in 145 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) May 23 12:43:00.959095 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2caeb76a-43e2-4342-9fdf-f1cc55cb5ccc None admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:43:01.057154 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2caeb76a-43e2-4342-9fdf-f1cc55cb5ccc None admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:43:01.090706 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4bd3676-628f-4ae2-9715-7896d2a4c0c7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:43:01.091343 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4bd3676-628f-4ae2-9715-7896d2a4c0c7 None admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:43:01.091631 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4bd3676-628f-4ae2-9715-7896d2a4c0c7 None admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:43:01.091911 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4bd3676-628f-4ae2-9715-7896d2a4c0c7 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:43:01.148485 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=100 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:43:01.159463 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f4bd3676-628f-4ae2-9715-7896d2a4c0c7 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:43:01.160871 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f4bd3676-628f-4ae2-9715-7896d2a4c0c7 None admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:43:01.255597 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 581/1159] 10.222.0.30 () {58 vars in 1282 bytes} [Mon May 23 12:43:00 2022] GET /identity/v3/auth/tokens => generated 2108 bytes in 297 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) May 23 12:43:01.261474 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10c416c5-4e10-44d3-b47a-2e100dccc4e9 None admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:43:01.357884 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10c416c5-4e10-44d3-b47a-2e100dccc4e9 None admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:43:01.390715 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18ea75a8-6481-4fea-98e3-35a1def6befc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:43:01.391563 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18ea75a8-6481-4fea-98e3-35a1def6befc None admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:43:01.391840 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18ea75a8-6481-4fea-98e3-35a1def6befc None admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:43:01.392108 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18ea75a8-6481-4fea-98e3-35a1def6befc None admin] PATH_INFO: `/v3/projects/2cfa8fc1578b4c099c1ce415a0266e74` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:43:01.393900 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18ea75a8-6481-4fea-98e3-35a1def6befc None admin] RBAC: Authorizing `identity:get_project(project_id=2cfa8fc1578b4c099c1ce415a0266e74)` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:43:01.395317 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18ea75a8-6481-4fea-98e3-35a1def6befc None admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:43:01.396967 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 579/1160] 10.222.0.30 () {56 vars in 1186 bytes} [Mon May 23 12:43:01 2022] GET /identity/v3/projects/2cfa8fc1578b4c099c1ce415a0266e74 => generated 311 bytes in 136 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) May 23 12:43:01.403304 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4bd3676-628f-4ae2-9715-7896d2a4c0c7 None admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:43:01.490260 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=100 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:43:01.499326 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4bd3676-628f-4ae2-9715-7896d2a4c0c7 None admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:43:01.534115 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11e5dffc-2d74-4d7b-aceb-ee3459d6ea42 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:43:01.534755 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11e5dffc-2d74-4d7b-aceb-ee3459d6ea42 None admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:43:01.535035 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11e5dffc-2d74-4d7b-aceb-ee3459d6ea42 None admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:43:01.535305 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11e5dffc-2d74-4d7b-aceb-ee3459d6ea42 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:43:01.536118 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11e5dffc-2d74-4d7b-aceb-ee3459d6ea42 None admin] RBAC: Adding query filter params (role.id=bc3c3c9d654940cbaa5177a6be23d1bc, scope.project.id=2cfa8fc1578b4c099c1ce415a0266e74, user.id=fd19c9a870d24fd58f81be06718d032e) {{(pid=92057) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} May 23 12:43:01.536427 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11e5dffc-2d74-4d7b-aceb-ee3459d6ea42 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:43:01.537895 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11e5dffc-2d74-4d7b-aceb-ee3459d6ea42 None admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:43:01.543526 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 582/1161] 10.222.0.30 () {56 vars in 1374 bytes} [Mon May 23 12:43:01 2022] GET /identity/v3/role_assignments?user.id=fd19c9a870d24fd58f81be06718d032e&scope.project.id=2cfa8fc1578b4c099c1ce415a0266e74&role.id=bc3c3c9d654940cbaa5177a6be23d1bc => generated 608 bytes in 141 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) May 23 12:43:03.746409 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82ff40aa-03a0-4226-b999-8331a67403ae None None] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:43:03.746915 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82ff40aa-03a0-4226-b999-8331a67403ae None None] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:43:03.747006 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82ff40aa-03a0-4226-b999-8331a67403ae None None] PATH_INFO: `/` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:43:03.748342 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 580/1162] 10.222.0.30 () {54 vars in 926 bytes} [Mon May 23 12:43:03 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) May 23 12:43:03.753593 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e0df78b-a118-4dd3-93a2-d593aa5f89fa None None] REQUEST_METHOD: `POST` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:43:03.754005 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e0df78b-a118-4dd3-93a2-d593aa5f89fa None None] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:43:03.754176 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e0df78b-a118-4dd3-93a2-d593aa5f89fa None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:43:03.771102 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.auth.core [None req-9e0df78b-a118-4dd3-93a2-d593aa5f89fa None None] MFA Rules not processed for user `0f7119f8fa5c4445977671140ded91a4`. Rule list: `[]` (Enabled: `True`). {{(pid=92057) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 12:43:03.772255 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.fernet_utils [None req-9e0df78b-a118-4dd3-93a2-d593aa5f89fa None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92057) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 12:43:03.837618 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 583/1163] 10.222.0.30 () {58 vars in 1023 bytes} [Mon May 23 12:43:03 2022] POST /identity/v3/auth/tokens => generated 2108 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) May 23 12:43:03.860014 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82ff40aa-03a0-4226-b999-8331a67403ae None None] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:43:03.891554 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=100 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:43:03.964885 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82ff40aa-03a0-4226-b999-8331a67403ae None None] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:43:04.009679 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05f698f5-1dc7-4eab-b8e2-8de380ad6644 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:43:04.010358 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05f698f5-1dc7-4eab-b8e2-8de380ad6644 None admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:43:04.010645 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05f698f5-1dc7-4eab-b8e2-8de380ad6644 None admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:43:04.010924 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05f698f5-1dc7-4eab-b8e2-8de380ad6644 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:43:04.078310 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05f698f5-1dc7-4eab-b8e2-8de380ad6644 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:43:04.081367 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05f698f5-1dc7-4eab-b8e2-8de380ad6644 None admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:43:04.111189 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=100 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:43:04.180277 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 581/1164] 10.222.0.30 () {58 vars in 1282 bytes} [Mon May 23 12:43:03 2022] GET /identity/v3/auth/tokens => generated 2108 bytes in 321 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) May 23 12:43:04.186974 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e0df78b-a118-4dd3-93a2-d593aa5f89fa None None] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:43:04.269884 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=100 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:43:04.285999 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e0df78b-a118-4dd3-93a2-d593aa5f89fa None None] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:43:04.319555 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01770d2a-719a-4557-8091-bc72f904380f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:43:04.320347 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01770d2a-719a-4557-8091-bc72f904380f None admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:43:04.320634 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01770d2a-719a-4557-8091-bc72f904380f None admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:43:04.320915 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01770d2a-719a-4557-8091-bc72f904380f None admin] PATH_INFO: `/v3/domains/swift_test` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:43:04.327502 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01770d2a-719a-4557-8091-bc72f904380f None admin] RBAC: Authorizing `identity:get_domain(domain_id=swift_test)` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:43:04.328955 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01770d2a-719a-4557-8091-bc72f904380f None admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:43:04.343452 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application [None req-01770d2a-719a-4557-8091-bc72f904380f None admin] Could not find domain: swift_test.: keystone.exception.DomainNotFound: Could not find domain: swift_test. May 23 12:43:04.343452 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application Traceback (most recent call last): May 23 12:43:04.343452 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 712, in get_domain May 23 12:43:04.343452 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application project = self.driver.get_project(domain_id) May 23 12:43:04.343452 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project May 23 12:43:04.343452 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() May 23 12:43:04.343452 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project May 23 12:43:04.343452 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) May 23 12:43:04.343452 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: swift_test. May 23 12:43:04.343452 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application May 23 12:43:04.343452 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: May 23 12:43:04.343452 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application May 23 12:43:04.343452 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application Traceback (most recent call last): May 23 12:43:04.343452 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request May 23 12:43:04.343452 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application rv = self.dispatch_request() May 23 12:43:04.343452 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request May 23 12:43:04.343452 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) May 23 12:43:04.344721 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper May 23 12:43:04.344721 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) May 23 12:43:04.344721 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view May 23 12:43:04.344721 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) May 23 12:43:04.344721 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request May 23 12:43:04.344721 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) May 23 12:43:04.344721 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get May 23 12:43:04.344721 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self._get_domain(domain_id) May 23 12:43:04.344721 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain May 23 12:43:04.344721 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) May 23 12:43:04.344721 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped May 23 12:43:04.344721 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) May 23 12:43:04.344721 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun May 23 12:43:04.344721 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) May 23 12:43:04.344721 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func May 23 12:43:04.344721 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self.get_or_create( May 23 12:43:04.346014 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create May 23 12:43:04.346014 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application with Lock( May 23 12:43:04.346014 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ May 23 12:43:04.346014 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self._enter() May 23 12:43:04.346014 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter May 23 12:43:04.346014 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) May 23 12:43:04.346014 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create May 23 12:43:04.346014 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self.creator() May 23 12:43:04.346014 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value May 23 12:43:04.346014 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application created_value = creator( May 23 12:43:04.346014 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 718, in get_domain May 23 12:43:04.346014 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) May 23 12:43:04.346014 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: swift_test. May 23 12:43:04.346014 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application May 23 12:43:04.346014 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 584/1165] 10.222.0.30 () {56 vars in 1117 bytes} [Mon May 23 12:43:04 2022] GET /identity/v3/domains/swift_test => generated 90 bytes in 158 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) May 23 12:43:04.350698 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05f698f5-1dc7-4eab-b8e2-8de380ad6644 None admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:43:04.447171 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05f698f5-1dc7-4eab-b8e2-8de380ad6644 None admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:43:04.475700 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=100 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:43:04.480539 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-183fd27b-8aaa-4270-8967-0fb34fb1f9dc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:43:04.488516 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-183fd27b-8aaa-4270-8967-0fb34fb1f9dc None admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:43:04.488516 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-183fd27b-8aaa-4270-8967-0fb34fb1f9dc None admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:43:04.488516 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-183fd27b-8aaa-4270-8967-0fb34fb1f9dc None admin] PATH_INFO: `/v3/domains` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:43:04.488516 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-183fd27b-8aaa-4270-8967-0fb34fb1f9dc None admin] RBAC: Adding query filter params (name=swift_test) {{(pid=92056) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} May 23 12:43:04.488516 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-183fd27b-8aaa-4270-8967-0fb34fb1f9dc None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:43:04.488516 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-183fd27b-8aaa-4270-8967-0fb34fb1f9dc None admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:43:04.492408 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 582/1166] 10.222.0.30 () {56 vars in 1115 bytes} [Mon May 23 12:43:04 2022] GET /identity/v3/domains?name=swift_test => generated 125 bytes in 142 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) May 23 12:43:06.722601 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae85500d-c3a3-4989-9768-23849ca9bb10 None None] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:43:06.723041 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae85500d-c3a3-4989-9768-23849ca9bb10 None None] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:43:06.723164 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae85500d-c3a3-4989-9768-23849ca9bb10 None None] PATH_INFO: `/` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:43:06.724546 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 585/1167] 10.222.0.30 () {54 vars in 926 bytes} [Mon May 23 12:43:06 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) May 23 12:43:06.729618 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d48ba20a-af4e-48e4-99b8-a5ff00e94853 None None] REQUEST_METHOD: `POST` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:43:06.730071 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d48ba20a-af4e-48e4-99b8-a5ff00e94853 None None] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:43:06.730197 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d48ba20a-af4e-48e4-99b8-a5ff00e94853 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:43:06.747111 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.auth.core [None req-d48ba20a-af4e-48e4-99b8-a5ff00e94853 None None] MFA Rules not processed for user `0f7119f8fa5c4445977671140ded91a4`. Rule list: `[]` (Enabled: `True`). {{(pid=92056) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 12:43:06.748282 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.fernet_utils [None req-d48ba20a-af4e-48e4-99b8-a5ff00e94853 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92056) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 12:43:06.812750 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 583/1168] 10.222.0.30 () {58 vars in 1023 bytes} [Mon May 23 12:43:06 2022] POST /identity/v3/auth/tokens => generated 2108 bytes in 84 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) May 23 12:43:06.835015 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae85500d-c3a3-4989-9768-23849ca9bb10 None None] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:43:06.935768 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae85500d-c3a3-4989-9768-23849ca9bb10 None None] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:43:06.968601 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7988b2e3-aaff-450a-8a76-24021ab680a8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:43:06.969191 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7988b2e3-aaff-450a-8a76-24021ab680a8 None admin] REQUEST_METHOD: `POST` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:43:06.969478 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7988b2e3-aaff-450a-8a76-24021ab680a8 None admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:43:06.969757 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7988b2e3-aaff-450a-8a76-24021ab680a8 None admin] PATH_INFO: `/v3/domains` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:43:06.971110 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7988b2e3-aaff-450a-8a76-24021ab680a8 None admin] RBAC: Authorizing `identity:create_domain()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:43:06.972492 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7988b2e3-aaff-450a-8a76-24021ab680a8 None admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:43:07.069012 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 586/1169] 10.222.0.30 () {60 vars in 1138 bytes} [Mon May 23 12:43:06 2022] POST /identity/v3/domains => generated 267 bytes in 234 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) May 23 12:43:09.284526 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f9eed03-ef8e-4b0e-abe9-10e0d4967d7c None None] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:43:09.285089 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f9eed03-ef8e-4b0e-abe9-10e0d4967d7c None None] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:43:09.285089 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f9eed03-ef8e-4b0e-abe9-10e0d4967d7c None None] PATH_INFO: `/` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:43:09.286461 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 584/1170] 10.222.0.30 () {54 vars in 926 bytes} [Mon May 23 12:43:09 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) May 23 12:43:09.291648 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35207f98-818f-446a-a831-a1cb444b1072 None None] REQUEST_METHOD: `POST` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:43:09.292029 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35207f98-818f-446a-a831-a1cb444b1072 None None] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:43:09.292213 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35207f98-818f-446a-a831-a1cb444b1072 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:43:09.298108 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:43:09.308835 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.auth.core [None req-35207f98-818f-446a-a831-a1cb444b1072 None None] MFA Rules not processed for user `0f7119f8fa5c4445977671140ded91a4`. Rule list: `[]` (Enabled: `True`). {{(pid=92057) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 12:43:09.310042 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.fernet_utils [None req-35207f98-818f-446a-a831-a1cb444b1072 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92057) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 12:43:09.375181 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 587/1171] 10.222.0.30 () {58 vars in 1023 bytes} [Mon May 23 12:43:09 2022] POST /identity/v3/auth/tokens => generated 2108 bytes in 84 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) May 23 12:43:09.398977 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f9eed03-ef8e-4b0e-abe9-10e0d4967d7c None None] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:43:09.497721 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f9eed03-ef8e-4b0e-abe9-10e0d4967d7c None None] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:43:09.522994 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=100 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:43:09.531541 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b43f58e-8d69-4634-adc9-b719aaf176e5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:43:09.534698 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b43f58e-8d69-4634-adc9-b719aaf176e5 None admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:43:09.534698 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b43f58e-8d69-4634-adc9-b719aaf176e5 None admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:43:09.534698 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b43f58e-8d69-4634-adc9-b719aaf176e5 None admin] PATH_INFO: `/v3/domains/763b9ef8d6974f4bb38e9f1bd8d50160` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:43:09.543535 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b43f58e-8d69-4634-adc9-b719aaf176e5 None admin] RBAC: Authorizing `identity:get_domain(domain_id=763b9ef8d6974f4bb38e9f1bd8d50160)` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:43:09.544958 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b43f58e-8d69-4634-adc9-b719aaf176e5 None admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:43:09.546592 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 585/1172] 10.222.0.30 () {56 vars in 1183 bytes} [Mon May 23 12:43:09 2022] GET /identity/v3/domains/763b9ef8d6974f4bb38e9f1bd8d50160 => generated 267 bytes in 148 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) May 23 12:43:09.552364 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35207f98-818f-446a-a831-a1cb444b1072 None None] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:43:09.648848 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35207f98-818f-446a-a831-a1cb444b1072 None None] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:43:09.681340 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf122254-5cca-4f40-89a2-4a894bcdef70 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:43:09.682038 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf122254-5cca-4f40-89a2-4a894bcdef70 None admin] REQUEST_METHOD: `POST` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:43:09.682322 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf122254-5cca-4f40-89a2-4a894bcdef70 None admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:43:09.682597 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf122254-5cca-4f40-89a2-4a894bcdef70 None admin] PATH_INFO: `/v3/projects` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:43:09.683307 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf122254-5cca-4f40-89a2-4a894bcdef70 None admin] RBAC: Authorizing `identity:create_project()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:43:09.684748 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf122254-5cca-4f40-89a2-4a894bcdef70 None admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:43:09.688137 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=100 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:43:09.794871 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 588/1173] 10.222.0.30 () {60 vars in 1141 bytes} [Mon May 23 12:43:09 2022] POST /identity/v3/projects => generated 361 bytes in 243 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) May 23 12:43:12.066931 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a30e64d-d5f5-4385-b631-99beade113c8 None None] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:43:12.067459 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a30e64d-d5f5-4385-b631-99beade113c8 None None] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:43:12.067459 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a30e64d-d5f5-4385-b631-99beade113c8 None None] PATH_INFO: `/` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:43:12.068771 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 586/1174] 10.222.0.30 () {54 vars in 926 bytes} [Mon May 23 12:43:12 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) May 23 12:43:12.073964 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6527cd18-6d56-4764-90ee-48362515d6ce None None] REQUEST_METHOD: `POST` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:43:12.074338 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6527cd18-6d56-4764-90ee-48362515d6ce None None] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:43:12.074527 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6527cd18-6d56-4764-90ee-48362515d6ce None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:43:12.091355 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.auth.core [None req-6527cd18-6d56-4764-90ee-48362515d6ce None None] MFA Rules not processed for user `0f7119f8fa5c4445977671140ded91a4`. Rule list: `[]` (Enabled: `True`). {{(pid=92057) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 12:43:12.092518 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.fernet_utils [None req-6527cd18-6d56-4764-90ee-48362515d6ce None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92057) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 12:43:12.156258 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 589/1175] 10.222.0.30 () {58 vars in 1023 bytes} [Mon May 23 12:43:12 2022] POST /identity/v3/auth/tokens => generated 2108 bytes in 83 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) May 23 12:43:12.180786 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a30e64d-d5f5-4385-b631-99beade113c8 None None] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:43:12.279856 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a30e64d-d5f5-4385-b631-99beade113c8 None None] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:43:12.312250 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9dc826b0-d720-47a8-ab66-86a3d36370a0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:43:12.313090 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9dc826b0-d720-47a8-ab66-86a3d36370a0 None admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:43:12.313441 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9dc826b0-d720-47a8-ab66-86a3d36370a0 None admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:43:12.314283 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9dc826b0-d720-47a8-ab66-86a3d36370a0 None admin] PATH_INFO: `/v3/domains/763b9ef8d6974f4bb38e9f1bd8d50160` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:43:12.315400 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9dc826b0-d720-47a8-ab66-86a3d36370a0 None admin] RBAC: Authorizing `identity:get_domain(domain_id=763b9ef8d6974f4bb38e9f1bd8d50160)` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:43:12.316841 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9dc826b0-d720-47a8-ab66-86a3d36370a0 None admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:43:12.318517 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 587/1176] 10.222.0.30 () {56 vars in 1183 bytes} [Mon May 23 12:43:12 2022] GET /identity/v3/domains/763b9ef8d6974f4bb38e9f1bd8d50160 => generated 267 bytes in 138 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) May 23 12:43:12.324064 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6527cd18-6d56-4764-90ee-48362515d6ce None None] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:43:12.419973 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6527cd18-6d56-4764-90ee-48362515d6ce None None] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:43:12.436182 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:43:12.453165 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8df96323-cc50-4faa-bd3d-78eadaceaf31 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:43:12.455336 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8df96323-cc50-4faa-bd3d-78eadaceaf31 None admin] REQUEST_METHOD: `POST` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:43:12.455336 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8df96323-cc50-4faa-bd3d-78eadaceaf31 None admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:43:12.455336 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8df96323-cc50-4faa-bd3d-78eadaceaf31 None admin] PATH_INFO: `/v3/users` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:43:12.455799 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8df96323-cc50-4faa-bd3d-78eadaceaf31 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:43:12.457244 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8df96323-cc50-4faa-bd3d-78eadaceaf31 None admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:43:12.536623 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 590/1177] 10.222.0.30 () {60 vars in 1132 bytes} [Mon May 23 12:43:12 2022] POST /identity/v3/users => generated 311 bytes in 213 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) May 23 12:43:14.743276 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0215bcaa-09cb-4dae-897d-4b7396c41dbe None None] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:43:14.743823 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0215bcaa-09cb-4dae-897d-4b7396c41dbe None None] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:43:14.743823 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0215bcaa-09cb-4dae-897d-4b7396c41dbe None None] PATH_INFO: `/` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:43:14.745187 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 588/1178] 10.222.0.30 () {54 vars in 926 bytes} [Mon May 23 12:43:14 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) May 23 12:43:14.750530 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-281410a4-042d-4543-bfaf-b90be9d2e942 None None] REQUEST_METHOD: `POST` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:43:14.751555 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-281410a4-042d-4543-bfaf-b90be9d2e942 None None] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:43:14.751665 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-281410a4-042d-4543-bfaf-b90be9d2e942 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:43:14.768061 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.auth.core [None req-281410a4-042d-4543-bfaf-b90be9d2e942 None None] MFA Rules not processed for user `0f7119f8fa5c4445977671140ded91a4`. Rule list: `[]` (Enabled: `True`). {{(pid=92057) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 12:43:14.769221 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.fernet_utils [None req-281410a4-042d-4543-bfaf-b90be9d2e942 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92057) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 12:43:14.833073 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 591/1179] 10.222.0.30 () {58 vars in 1023 bytes} [Mon May 23 12:43:14 2022] POST /identity/v3/auth/tokens => generated 2108 bytes in 84 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) May 23 12:43:14.856708 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0215bcaa-09cb-4dae-897d-4b7396c41dbe None None] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:43:14.912191 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=100 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:43:14.956813 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0215bcaa-09cb-4dae-897d-4b7396c41dbe None None] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:43:14.997729 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f254cc7b-b08e-4feb-98d5-e7370141e07b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:43:14.998533 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f254cc7b-b08e-4feb-98d5-e7370141e07b None admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:43:14.998815 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f254cc7b-b08e-4feb-98d5-e7370141e07b None admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:43:14.999091 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f254cc7b-b08e-4feb-98d5-e7370141e07b None admin] PATH_INFO: `/v3/roles/admin` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:43:15.017139 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f254cc7b-b08e-4feb-98d5-e7370141e07b None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} May 23 12:43:15.026227 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application [None req-f254cc7b-b08e-4feb-98d5-e7370141e07b None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. May 23 12:43:15.026227 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application Traceback (most recent call last): May 23 12:43:15.026227 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request May 23 12:43:15.026227 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application rv = self.dispatch_request() May 23 12:43:15.026227 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request May 23 12:43:15.026227 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) May 23 12:43:15.026227 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper May 23 12:43:15.026227 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) May 23 12:43:15.026227 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view May 23 12:43:15.026227 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) May 23 12:43:15.026227 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request May 23 12:43:15.026227 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) May 23 12:43:15.026227 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get May 23 12:43:15.026227 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self._get_role(role_id) May 23 12:43:15.026227 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role May 23 12:43:15.026227 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') May 23 12:43:15.027336 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call May 23 12:43:15.027336 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( May 23 12:43:15.027336 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data May 23 12:43:15.027336 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) May 23 12:43:15.027336 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped May 23 12:43:15.027336 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) May 23 12:43:15.027336 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun May 23 12:43:15.027336 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) May 23 12:43:15.027336 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func May 23 12:43:15.027336 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self.get_or_create( May 23 12:43:15.027336 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create May 23 12:43:15.027336 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application with Lock( May 23 12:43:15.027336 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ May 23 12:43:15.027336 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self._enter() May 23 12:43:15.027336 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter May 23 12:43:15.027336 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) May 23 12:43:15.028370 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create May 23 12:43:15.028370 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self.creator() May 23 12:43:15.028370 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value May 23 12:43:15.028370 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application created_value = creator( May 23 12:43:15.028370 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role May 23 12:43:15.028370 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) May 23 12:43:15.028370 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role May 23 12:43:15.028370 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() May 23 12:43:15.028370 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role May 23 12:43:15.028370 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) May 23 12:43:15.028370 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. May 23 12:43:15.028370 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application May 23 12:43:15.028370 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 589/1180] 10.222.0.30 () {56 vars in 1096 bytes} [Mon May 23 12:43:14 2022] GET /identity/v3/roles/admin => generated 83 bytes in 171 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) May 23 12:43:15.034222 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-281410a4-042d-4543-bfaf-b90be9d2e942 None None] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:43:15.130718 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-281410a4-042d-4543-bfaf-b90be9d2e942 None None] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:43:15.132204 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:43:15.163713 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47189351-6221-4a12-835e-bb60a856b737 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:43:15.164584 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47189351-6221-4a12-835e-bb60a856b737 None admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:43:15.165029 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47189351-6221-4a12-835e-bb60a856b737 None admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:43:15.165663 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47189351-6221-4a12-835e-bb60a856b737 None admin] PATH_INFO: `/v3/roles` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:43:15.166469 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47189351-6221-4a12-835e-bb60a856b737 None admin] RBAC: Adding query filter params (name=admin) {{(pid=92057) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} May 23 12:43:15.166786 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47189351-6221-4a12-835e-bb60a856b737 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:43:15.168180 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47189351-6221-4a12-835e-bb60a856b737 None admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:43:15.177100 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 592/1181] 10.222.0.30 () {56 vars in 1099 bytes} [Mon May 23 12:43:15 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 143 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) May 23 12:43:15.182865 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f254cc7b-b08e-4feb-98d5-e7370141e07b None admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:43:15.279149 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f254cc7b-b08e-4feb-98d5-e7370141e07b None admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:43:15.292878 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=100 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:43:15.311716 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2c3df73-c789-4548-9752-a197e5d05d17 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:43:15.312561 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2c3df73-c789-4548-9752-a197e5d05d17 None admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:43:15.312836 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2c3df73-c789-4548-9752-a197e5d05d17 None admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:43:15.313105 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2c3df73-c789-4548-9752-a197e5d05d17 None admin] PATH_INFO: `/v3/users/79b6e3916b82451a9db7d043674619ee` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:43:15.326172 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2c3df73-c789-4548-9752-a197e5d05d17 None admin] RBAC: Authorizing `identity:get_user(user_id=79b6e3916b82451a9db7d043674619ee)` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:43:15.327599 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2c3df73-c789-4548-9752-a197e5d05d17 None admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:43:15.330019 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 590/1182] 10.222.0.30 () {56 vars in 1177 bytes} [Mon May 23 12:43:15 2022] GET /identity/v3/users/79b6e3916b82451a9db7d043674619ee => generated 311 bytes in 148 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) May 23 12:43:15.335736 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47189351-6221-4a12-835e-bb60a856b737 None admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:43:15.431036 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47189351-6221-4a12-835e-bb60a856b737 None admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:43:15.463412 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ea61921-58a2-4457-950f-109ca693b5cc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:43:15.464042 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ea61921-58a2-4457-950f-109ca693b5cc None admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:43:15.464327 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ea61921-58a2-4457-950f-109ca693b5cc None admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:43:15.464605 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ea61921-58a2-4457-950f-109ca693b5cc None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:43:15.493894 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=100 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:43:15.530078 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ea61921-58a2-4457-950f-109ca693b5cc None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:43:15.531460 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ea61921-58a2-4457-950f-109ca693b5cc None admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:43:15.626257 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 593/1183] 10.222.0.30 () {58 vars in 1282 bytes} [Mon May 23 12:43:15 2022] GET /identity/v3/auth/tokens => generated 2108 bytes in 291 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) May 23 12:43:15.632034 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2c3df73-c789-4548-9752-a197e5d05d17 None admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:43:15.730805 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2c3df73-c789-4548-9752-a197e5d05d17 None admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:43:15.763741 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd4b10cc-fb85-41da-be97-0ecc10ba3c68 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:43:15.764596 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd4b10cc-fb85-41da-be97-0ecc10ba3c68 None admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:43:15.764882 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd4b10cc-fb85-41da-be97-0ecc10ba3c68 None admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:43:15.765160 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd4b10cc-fb85-41da-be97-0ecc10ba3c68 None admin] PATH_INFO: `/v3/projects/68114c89d7fc4a4abe2a7702d3d79fee` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:43:15.766434 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd4b10cc-fb85-41da-be97-0ecc10ba3c68 None admin] RBAC: Authorizing `identity:get_project(project_id=68114c89d7fc4a4abe2a7702d3d79fee)` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:43:15.767910 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd4b10cc-fb85-41da-be97-0ecc10ba3c68 None admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:43:15.769602 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 591/1184] 10.222.0.30 () {56 vars in 1186 bytes} [Mon May 23 12:43:15 2022] GET /identity/v3/projects/68114c89d7fc4a4abe2a7702d3d79fee => generated 361 bytes in 138 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) May 23 12:43:15.776018 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ea61921-58a2-4457-950f-109ca693b5cc None admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:43:15.836080 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=100 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:43:15.873063 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ea61921-58a2-4457-950f-109ca693b5cc None admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:43:15.908794 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68f5fcc7-5db0-413f-8b59-897c3ade88ee None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:43:15.909442 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68f5fcc7-5db0-413f-8b59-897c3ade88ee None admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:43:15.909726 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68f5fcc7-5db0-413f-8b59-897c3ade88ee None admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:43:15.910033 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68f5fcc7-5db0-413f-8b59-897c3ade88ee None admin] PATH_INFO: `/v3/role_assignments` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:43:15.911804 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68f5fcc7-5db0-413f-8b59-897c3ade88ee None admin] RBAC: Adding query filter params (role.id=bc3c3c9d654940cbaa5177a6be23d1bc, scope.project.id=68114c89d7fc4a4abe2a7702d3d79fee, user.id=79b6e3916b82451a9db7d043674619ee) {{(pid=92057) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} May 23 12:43:15.912122 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68f5fcc7-5db0-413f-8b59-897c3ade88ee None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:43:15.913557 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68f5fcc7-5db0-413f-8b59-897c3ade88ee None admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:43:15.918879 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 594/1185] 10.222.0.30 () {56 vars in 1374 bytes} [Mon May 23 12:43:15 2022] GET /identity/v3/role_assignments?user.id=79b6e3916b82451a9db7d043674619ee&scope.project.id=68114c89d7fc4a4abe2a7702d3d79fee&role.id=bc3c3c9d654940cbaa5177a6be23d1bc => generated 259 bytes in 143 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) May 23 12:43:18.077731 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35e844f7-0fa3-4348-978e-e88b4aeece46 None None] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:43:18.078245 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35e844f7-0fa3-4348-978e-e88b4aeece46 None None] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:43:18.078245 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35e844f7-0fa3-4348-978e-e88b4aeece46 None None] PATH_INFO: `/` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:43:18.080270 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 592/1186] 10.222.0.30 () {54 vars in 926 bytes} [Mon May 23 12:43:18 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) May 23 12:43:18.085621 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ed5f33a-dab9-4a93-806a-e07fab4d8c58 None None] REQUEST_METHOD: `POST` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:43:18.086079 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ed5f33a-dab9-4a93-806a-e07fab4d8c58 None None] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:43:18.086275 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ed5f33a-dab9-4a93-806a-e07fab4d8c58 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:43:18.102887 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.auth.core [None req-8ed5f33a-dab9-4a93-806a-e07fab4d8c58 None None] MFA Rules not processed for user `0f7119f8fa5c4445977671140ded91a4`. Rule list: `[]` (Enabled: `True`). {{(pid=92057) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 12:43:18.104056 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.fernet_utils [None req-8ed5f33a-dab9-4a93-806a-e07fab4d8c58 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92057) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 12:43:18.172085 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 595/1187] 10.222.0.30 () {58 vars in 1023 bytes} [Mon May 23 12:43:18 2022] POST /identity/v3/auth/tokens => generated 2108 bytes in 87 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) May 23 12:43:18.195079 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35e844f7-0fa3-4348-978e-e88b4aeece46 None None] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:43:18.238448 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=100 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:43:18.295529 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35e844f7-0fa3-4348-978e-e88b4aeece46 None None] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:43:18.330291 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-140cbd42-6d87-4172-b1e2-a1e0893794c7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:43:18.331007 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-140cbd42-6d87-4172-b1e2-a1e0893794c7 None admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:43:18.331291 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-140cbd42-6d87-4172-b1e2-a1e0893794c7 None admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:43:18.331557 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-140cbd42-6d87-4172-b1e2-a1e0893794c7 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:43:18.342276 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-140cbd42-6d87-4172-b1e2-a1e0893794c7 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} May 23 12:43:18.350076 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application [None req-140cbd42-6d87-4172-b1e2-a1e0893794c7 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. May 23 12:43:18.350076 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application Traceback (most recent call last): May 23 12:43:18.350076 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request May 23 12:43:18.350076 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application rv = self.dispatch_request() May 23 12:43:18.350076 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request May 23 12:43:18.350076 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) May 23 12:43:18.350076 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper May 23 12:43:18.350076 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) May 23 12:43:18.350076 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view May 23 12:43:18.350076 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) May 23 12:43:18.350076 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request May 23 12:43:18.350076 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) May 23 12:43:18.350076 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get May 23 12:43:18.350076 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self._get_role(role_id) May 23 12:43:18.350076 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role May 23 12:43:18.350076 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') May 23 12:43:18.351197 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call May 23 12:43:18.351197 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( May 23 12:43:18.351197 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data May 23 12:43:18.351197 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) May 23 12:43:18.351197 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped May 23 12:43:18.351197 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) May 23 12:43:18.351197 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun May 23 12:43:18.351197 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) May 23 12:43:18.351197 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func May 23 12:43:18.351197 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self.get_or_create( May 23 12:43:18.351197 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create May 23 12:43:18.351197 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application with Lock( May 23 12:43:18.351197 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ May 23 12:43:18.351197 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self._enter() May 23 12:43:18.351197 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter May 23 12:43:18.351197 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) May 23 12:43:18.352388 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create May 23 12:43:18.352388 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self.creator() May 23 12:43:18.352388 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value May 23 12:43:18.352388 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application created_value = creator( May 23 12:43:18.352388 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role May 23 12:43:18.352388 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) May 23 12:43:18.352388 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role May 23 12:43:18.352388 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() May 23 12:43:18.352388 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role May 23 12:43:18.352388 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) May 23 12:43:18.352388 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. May 23 12:43:18.352388 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application May 23 12:43:18.352388 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 593/1188] 10.222.0.30 () {56 vars in 1096 bytes} [Mon May 23 12:43:18 2022] GET /identity/v3/roles/admin => generated 83 bytes in 157 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) May 23 12:43:18.357220 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ed5f33a-dab9-4a93-806a-e07fab4d8c58 None None] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:43:18.412791 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=100 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:43:18.452563 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ed5f33a-dab9-4a93-806a-e07fab4d8c58 None None] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:43:18.484426 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d48ac55e-3137-4dac-83af-71ae0c725e50 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:43:18.485638 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d48ac55e-3137-4dac-83af-71ae0c725e50 None admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:43:18.485937 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d48ac55e-3137-4dac-83af-71ae0c725e50 None admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:43:18.486215 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d48ac55e-3137-4dac-83af-71ae0c725e50 None admin] PATH_INFO: `/v3/roles` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:43:18.486945 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d48ac55e-3137-4dac-83af-71ae0c725e50 None admin] RBAC: Adding query filter params (name=admin) {{(pid=92057) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} May 23 12:43:18.487249 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d48ac55e-3137-4dac-83af-71ae0c725e50 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:43:18.488613 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d48ac55e-3137-4dac-83af-71ae0c725e50 None admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:43:18.496300 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 596/1189] 10.222.0.30 () {56 vars in 1099 bytes} [Mon May 23 12:43:18 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 140 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) May 23 12:43:18.503150 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-140cbd42-6d87-4172-b1e2-a1e0893794c7 None admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:43:18.600147 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=100 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:43:18.602931 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-140cbd42-6d87-4172-b1e2-a1e0893794c7 None admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:43:18.636188 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de817618-88f7-4e0d-9a44-3e80a7d15815 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:43:18.637046 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de817618-88f7-4e0d-9a44-3e80a7d15815 None admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:43:18.637328 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de817618-88f7-4e0d-9a44-3e80a7d15815 None admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:43:18.637604 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de817618-88f7-4e0d-9a44-3e80a7d15815 None admin] PATH_INFO: `/v3/users/79b6e3916b82451a9db7d043674619ee` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:43:18.638805 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de817618-88f7-4e0d-9a44-3e80a7d15815 None admin] RBAC: Authorizing `identity:get_user(user_id=79b6e3916b82451a9db7d043674619ee)` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:43:18.640243 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de817618-88f7-4e0d-9a44-3e80a7d15815 None admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:43:18.642536 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 594/1190] 10.222.0.30 () {56 vars in 1177 bytes} [Mon May 23 12:43:18 2022] GET /identity/v3/users/79b6e3916b82451a9db7d043674619ee => generated 311 bytes in 140 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) May 23 12:43:18.648503 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d48ac55e-3137-4dac-83af-71ae0c725e50 None admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:43:18.746019 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d48ac55e-3137-4dac-83af-71ae0c725e50 None admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:43:18.765824 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=100 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:43:18.779935 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7988232f-ba67-443c-ad66-01a4ef3d3f72 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:43:18.784024 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7988232f-ba67-443c-ad66-01a4ef3d3f72 None admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:43:18.784024 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7988232f-ba67-443c-ad66-01a4ef3d3f72 None admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:43:18.784024 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7988232f-ba67-443c-ad66-01a4ef3d3f72 None admin] PATH_INFO: `/v3/projects/68114c89d7fc4a4abe2a7702d3d79fee` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:43:18.784024 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7988232f-ba67-443c-ad66-01a4ef3d3f72 None admin] RBAC: Authorizing `identity:get_project(project_id=68114c89d7fc4a4abe2a7702d3d79fee)` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:43:18.784348 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7988232f-ba67-443c-ad66-01a4ef3d3f72 None admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:43:18.785959 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 597/1191] 10.222.0.30 () {56 vars in 1186 bytes} [Mon May 23 12:43:18 2022] GET /identity/v3/projects/68114c89d7fc4a4abe2a7702d3d79fee => generated 361 bytes in 138 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) May 23 12:43:18.791963 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de817618-88f7-4e0d-9a44-3e80a7d15815 None admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:43:18.892835 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de817618-88f7-4e0d-9a44-3e80a7d15815 None admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:43:18.927755 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-246ac4dd-0300-4467-9bf7-26fae76bd097 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:43:18.928621 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-246ac4dd-0300-4467-9bf7-26fae76bd097 None admin] REQUEST_METHOD: `PUT` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:43:18.928902 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-246ac4dd-0300-4467-9bf7-26fae76bd097 None admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:43:18.929176 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-246ac4dd-0300-4467-9bf7-26fae76bd097 None admin] PATH_INFO: `/v3/projects/68114c89d7fc4a4abe2a7702d3d79fee/users/79b6e3916b82451a9db7d043674619ee/roles/bc3c3c9d654940cbaa5177a6be23d1bc` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:43:18.931117 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-246ac4dd-0300-4467-9bf7-26fae76bd097 None admin] RBAC: Authorizing `identity:create_grant(project_id=68114c89d7fc4a4abe2a7702d3d79fee, user_id=79b6e3916b82451a9db7d043674619ee, role_id=bc3c3c9d654940cbaa5177a6be23d1bc)` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:43:18.932554 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-246ac4dd-0300-4467-9bf7-26fae76bd097 None admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:43:19.008409 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 595/1192] 10.222.0.30 () {58 vars in 1439 bytes} [Mon May 23 12:43:18 2022] PUT /identity/v3/projects/68114c89d7fc4a4abe2a7702d3d79fee/users/79b6e3916b82451a9db7d043674619ee/roles/bc3c3c9d654940cbaa5177a6be23d1bc => generated 0 bytes in 217 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) May 23 12:43:21.158086 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d71e35d-3ce2-4716-aeb0-7d0f7011fc6c None None] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:43:21.158671 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d71e35d-3ce2-4716-aeb0-7d0f7011fc6c None None] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:43:21.158671 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d71e35d-3ce2-4716-aeb0-7d0f7011fc6c None None] PATH_INFO: `/` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:43:21.160000 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 598/1193] 10.222.0.30 () {54 vars in 926 bytes} [Mon May 23 12:43:21 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) May 23 12:43:21.165071 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9a2861a-fffa-4f2f-8a84-a8f5f6452f85 None None] REQUEST_METHOD: `POST` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:43:21.165504 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9a2861a-fffa-4f2f-8a84-a8f5f6452f85 None None] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:43:21.165613 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9a2861a-fffa-4f2f-8a84-a8f5f6452f85 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:43:21.182482 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.auth.core [None req-d9a2861a-fffa-4f2f-8a84-a8f5f6452f85 None None] MFA Rules not processed for user `0f7119f8fa5c4445977671140ded91a4`. Rule list: `[]` (Enabled: `True`). {{(pid=92056) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 12:43:21.183639 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.fernet_utils [None req-d9a2861a-fffa-4f2f-8a84-a8f5f6452f85 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92056) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 12:43:21.205738 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:43:21.248168 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 596/1194] 10.222.0.30 () {58 vars in 1023 bytes} [Mon May 23 12:43:21 2022] POST /identity/v3/auth/tokens => generated 2108 bytes in 84 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) May 23 12:43:21.271897 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d71e35d-3ce2-4716-aeb0-7d0f7011fc6c None None] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:43:21.370157 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d71e35d-3ce2-4716-aeb0-7d0f7011fc6c None None] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:43:21.403433 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27a37b09-804a-4ebd-a360-b5b8af75bba9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:43:21.404212 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27a37b09-804a-4ebd-a360-b5b8af75bba9 None admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:43:21.404497 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27a37b09-804a-4ebd-a360-b5b8af75bba9 None admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:43:21.404774 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27a37b09-804a-4ebd-a360-b5b8af75bba9 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:43:21.415988 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-27a37b09-804a-4ebd-a360-b5b8af75bba9 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} May 23 12:43:21.423953 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application [None req-27a37b09-804a-4ebd-a360-b5b8af75bba9 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. May 23 12:43:21.423953 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application Traceback (most recent call last): May 23 12:43:21.423953 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request May 23 12:43:21.423953 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application rv = self.dispatch_request() May 23 12:43:21.423953 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request May 23 12:43:21.423953 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) May 23 12:43:21.423953 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper May 23 12:43:21.423953 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) May 23 12:43:21.423953 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view May 23 12:43:21.423953 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) May 23 12:43:21.423953 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request May 23 12:43:21.423953 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) May 23 12:43:21.423953 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get May 23 12:43:21.423953 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self._get_role(role_id) May 23 12:43:21.423953 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role May 23 12:43:21.423953 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') May 23 12:43:21.425093 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call May 23 12:43:21.425093 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( May 23 12:43:21.425093 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data May 23 12:43:21.425093 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) May 23 12:43:21.425093 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped May 23 12:43:21.425093 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) May 23 12:43:21.425093 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun May 23 12:43:21.425093 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) May 23 12:43:21.425093 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func May 23 12:43:21.425093 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self.get_or_create( May 23 12:43:21.425093 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create May 23 12:43:21.425093 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application with Lock( May 23 12:43:21.425093 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ May 23 12:43:21.425093 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self._enter() May 23 12:43:21.425093 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter May 23 12:43:21.425093 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) May 23 12:43:21.426199 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create May 23 12:43:21.426199 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self.creator() May 23 12:43:21.426199 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value May 23 12:43:21.426199 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application created_value = creator( May 23 12:43:21.426199 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role May 23 12:43:21.426199 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) May 23 12:43:21.426199 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role May 23 12:43:21.426199 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() May 23 12:43:21.426199 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role May 23 12:43:21.426199 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) May 23 12:43:21.426199 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. May 23 12:43:21.426199 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application May 23 12:43:21.426199 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 599/1195] 10.222.0.30 () {56 vars in 1096 bytes} [Mon May 23 12:43:21 2022] GET /identity/v3/roles/admin => generated 83 bytes in 154 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) May 23 12:43:21.431200 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9a2861a-fffa-4f2f-8a84-a8f5f6452f85 None None] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:43:21.539619 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9a2861a-fffa-4f2f-8a84-a8f5f6452f85 None None] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:43:21.573207 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-878062ad-32f1-4dd7-9251-29db41f16dfe None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:43:21.573862 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-878062ad-32f1-4dd7-9251-29db41f16dfe None admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:43:21.574170 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-878062ad-32f1-4dd7-9251-29db41f16dfe None admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:43:21.574445 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-878062ad-32f1-4dd7-9251-29db41f16dfe None admin] PATH_INFO: `/v3/roles` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:43:21.575186 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-878062ad-32f1-4dd7-9251-29db41f16dfe None admin] RBAC: Adding query filter params (name=admin) {{(pid=92056) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} May 23 12:43:21.575490 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-878062ad-32f1-4dd7-9251-29db41f16dfe None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:43:21.576856 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-878062ad-32f1-4dd7-9251-29db41f16dfe None admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:43:21.585651 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 597/1196] 10.222.0.30 () {56 vars in 1099 bytes} [Mon May 23 12:43:21 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 155 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) May 23 12:43:21.591513 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27a37b09-804a-4ebd-a360-b5b8af75bba9 None admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:43:21.639589 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=100 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:43:21.689540 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27a37b09-804a-4ebd-a360-b5b8af75bba9 None admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:43:21.723313 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e0a92a6-c347-4e5e-b20c-177607a84bc5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:43:21.724175 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e0a92a6-c347-4e5e-b20c-177607a84bc5 None admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:43:21.724460 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e0a92a6-c347-4e5e-b20c-177607a84bc5 None admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:43:21.724738 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e0a92a6-c347-4e5e-b20c-177607a84bc5 None admin] PATH_INFO: `/v3/users/79b6e3916b82451a9db7d043674619ee` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:43:21.725939 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e0a92a6-c347-4e5e-b20c-177607a84bc5 None admin] RBAC: Authorizing `identity:get_user(user_id=79b6e3916b82451a9db7d043674619ee)` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:43:21.727393 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e0a92a6-c347-4e5e-b20c-177607a84bc5 None admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:43:21.729017 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 600/1197] 10.222.0.30 () {56 vars in 1177 bytes} [Mon May 23 12:43:21 2022] GET /identity/v3/users/79b6e3916b82451a9db7d043674619ee => generated 311 bytes in 138 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) May 23 12:43:21.734676 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-878062ad-32f1-4dd7-9251-29db41f16dfe None admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:43:21.757367 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=100 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:43:21.833411 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-878062ad-32f1-4dd7-9251-29db41f16dfe None admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:43:21.865527 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7329df9d-3561-4f3c-b55f-afd6509de68b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:43:21.866188 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7329df9d-3561-4f3c-b55f-afd6509de68b None admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:43:21.866469 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7329df9d-3561-4f3c-b55f-afd6509de68b None admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:43:21.866739 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7329df9d-3561-4f3c-b55f-afd6509de68b None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:43:21.931668 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7329df9d-3561-4f3c-b55f-afd6509de68b None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:43:21.933101 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7329df9d-3561-4f3c-b55f-afd6509de68b None admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:43:21.953157 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=100 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:43:22.028340 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 598/1198] 10.222.0.30 () {58 vars in 1282 bytes} [Mon May 23 12:43:21 2022] GET /identity/v3/auth/tokens => generated 2108 bytes in 294 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) May 23 12:43:22.034568 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e0a92a6-c347-4e5e-b20c-177607a84bc5 None admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:43:22.133275 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e0a92a6-c347-4e5e-b20c-177607a84bc5 None admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:43:22.149136 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=100 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:43:22.165801 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02858093-3419-43a5-854c-d97ef4e62a7d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:43:22.166706 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02858093-3419-43a5-854c-d97ef4e62a7d None admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:43:22.166987 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02858093-3419-43a5-854c-d97ef4e62a7d None admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:43:22.167263 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02858093-3419-43a5-854c-d97ef4e62a7d None admin] PATH_INFO: `/v3/projects/68114c89d7fc4a4abe2a7702d3d79fee` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:43:22.168428 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-02858093-3419-43a5-854c-d97ef4e62a7d None admin] RBAC: Authorizing `identity:get_project(project_id=68114c89d7fc4a4abe2a7702d3d79fee)` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:43:22.169922 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-02858093-3419-43a5-854c-d97ef4e62a7d None admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:43:22.171689 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 601/1199] 10.222.0.30 () {56 vars in 1186 bytes} [Mon May 23 12:43:22 2022] GET /identity/v3/projects/68114c89d7fc4a4abe2a7702d3d79fee => generated 361 bytes in 138 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) May 23 12:43:22.178299 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7329df9d-3561-4f3c-b55f-afd6509de68b None admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:43:22.276494 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7329df9d-3561-4f3c-b55f-afd6509de68b None admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:43:22.296312 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=100 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:43:22.310203 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09eb93e4-5745-4f75-bd40-db7b9ea53b2c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:43:22.310840 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09eb93e4-5745-4f75-bd40-db7b9ea53b2c None admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:43:22.311125 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09eb93e4-5745-4f75-bd40-db7b9ea53b2c None admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:43:22.311401 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09eb93e4-5745-4f75-bd40-db7b9ea53b2c None admin] PATH_INFO: `/v3/role_assignments` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:43:22.312215 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09eb93e4-5745-4f75-bd40-db7b9ea53b2c None admin] RBAC: Adding query filter params (role.id=bc3c3c9d654940cbaa5177a6be23d1bc, scope.project.id=68114c89d7fc4a4abe2a7702d3d79fee, user.id=79b6e3916b82451a9db7d043674619ee) {{(pid=92056) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} May 23 12:43:22.312526 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09eb93e4-5745-4f75-bd40-db7b9ea53b2c None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:43:22.315023 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09eb93e4-5745-4f75-bd40-db7b9ea53b2c None admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:43:22.320650 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 599/1200] 10.222.0.30 () {56 vars in 1374 bytes} [Mon May 23 12:43:22 2022] GET /identity/v3/role_assignments?user.id=79b6e3916b82451a9db7d043674619ee&scope.project.id=68114c89d7fc4a4abe2a7702d3d79fee&role.id=bc3c3c9d654940cbaa5177a6be23d1bc => generated 608 bytes in 143 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) May 23 12:43:32.175329 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=8 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:43:32.341359 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=8 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:44:39.288782 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fe4f265-a577-4cee-b0d3-811f8056a062 None None] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:44:39.289418 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fe4f265-a577-4cee-b0d3-811f8056a062 None None] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:44:39.289418 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fe4f265-a577-4cee-b0d3-811f8056a062 None None] PATH_INFO: `/` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:44:39.290906 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 602/1201] 10.222.0.30 () {54 vars in 926 bytes} [Mon May 23 12:44:39 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) May 23 12:44:39.296207 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c5bdd03-6a28-44c3-bbca-d0dfbf414d5f None None] REQUEST_METHOD: `POST` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:44:39.296761 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c5bdd03-6a28-44c3-bbca-d0dfbf414d5f None None] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:44:39.296761 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c5bdd03-6a28-44c3-bbca-d0dfbf414d5f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:44:39.314169 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.auth.core [None req-2c5bdd03-6a28-44c3-bbca-d0dfbf414d5f None None] MFA Rules not processed for user `0f7119f8fa5c4445977671140ded91a4`. Rule list: `[]` (Enabled: `True`). {{(pid=92056) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 12:44:39.315929 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.fernet_utils [None req-2c5bdd03-6a28-44c3-bbca-d0dfbf414d5f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92056) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 12:44:39.384001 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 600/1202] 10.222.0.30 () {58 vars in 1023 bytes} [Mon May 23 12:44:39 2022] POST /identity/v3/auth/tokens => generated 2108 bytes in 89 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) May 23 12:44:39.408857 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2fe4f265-a577-4cee-b0d3-811f8056a062 None None] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:44:39.511085 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2fe4f265-a577-4cee-b0d3-811f8056a062 None None] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:44:39.546625 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aaad5f48-a258-4e4b-ac0c-4ed0bc03e5b0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:44:39.547945 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aaad5f48-a258-4e4b-ac0c-4ed0bc03e5b0 None admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:44:39.548417 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aaad5f48-a258-4e4b-ac0c-4ed0bc03e5b0 None admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:44:39.548931 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aaad5f48-a258-4e4b-ac0c-4ed0bc03e5b0 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:44:39.560156 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aaad5f48-a258-4e4b-ac0c-4ed0bc03e5b0 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:44:39.561619 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aaad5f48-a258-4e4b-ac0c-4ed0bc03e5b0 None admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:44:39.578896 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application [None req-aaad5f48-a258-4e4b-ac0c-4ed0bc03e5b0 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. May 23 12:44:39.578896 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application Traceback (most recent call last): May 23 12:44:39.578896 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request May 23 12:44:39.578896 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application rv = self.dispatch_request() May 23 12:44:39.578896 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request May 23 12:44:39.578896 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) May 23 12:44:39.578896 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper May 23 12:44:39.578896 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) May 23 12:44:39.578896 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view May 23 12:44:39.578896 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) May 23 12:44:39.578896 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request May 23 12:44:39.578896 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) May 23 12:44:39.578896 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get May 23 12:44:39.578896 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self._get_domain(domain_id) May 23 12:44:39.578896 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain May 23 12:44:39.580123 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) May 23 12:44:39.580123 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped May 23 12:44:39.580123 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) May 23 12:44:39.580123 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun May 23 12:44:39.580123 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) May 23 12:44:39.580123 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func May 23 12:44:39.580123 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self.get_or_create( May 23 12:44:39.580123 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create May 23 12:44:39.580123 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application with Lock( May 23 12:44:39.580123 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ May 23 12:44:39.580123 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self._enter() May 23 12:44:39.580123 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter May 23 12:44:39.580123 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) May 23 12:44:39.580123 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create May 23 12:44:39.580123 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self.creator() May 23 12:44:39.580123 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value May 23 12:44:39.580123 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application created_value = creator( May 23 12:44:39.581631 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain May 23 12:44:39.581631 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) May 23 12:44:39.581631 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. May 23 12:44:39.581631 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application May 23 12:44:39.581631 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 603/1203] 10.222.0.30 () {56 vars in 1108 bytes} [Mon May 23 12:44:39 2022] GET /identity/v3/domains/Default => generated 87 bytes in 172 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) May 23 12:44:39.586477 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c5bdd03-6a28-44c3-bbca-d0dfbf414d5f None None] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:44:39.686859 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c5bdd03-6a28-44c3-bbca-d0dfbf414d5f None None] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:44:39.716048 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=100 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:44:39.722439 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-789282dc-6bfc-4b3b-a61f-f08c6800f6a2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:44:39.723140 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-789282dc-6bfc-4b3b-a61f-f08c6800f6a2 None admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:44:39.723424 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-789282dc-6bfc-4b3b-a61f-f08c6800f6a2 None admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:44:39.723700 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-789282dc-6bfc-4b3b-a61f-f08c6800f6a2 None admin] PATH_INFO: `/v3/domains` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:44:39.724420 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-789282dc-6bfc-4b3b-a61f-f08c6800f6a2 None admin] RBAC: Adding query filter params (name=Default) {{(pid=92056) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} May 23 12:44:39.724728 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-789282dc-6bfc-4b3b-a61f-f08c6800f6a2 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:44:39.726192 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-789282dc-6bfc-4b3b-a61f-f08c6800f6a2 None admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:44:39.737895 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 601/1204] 10.222.0.30 () {56 vars in 1109 bytes} [Mon May 23 12:44:39 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 152 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) May 23 12:44:39.743693 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aaad5f48-a258-4e4b-ac0c-4ed0bc03e5b0 None admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:44:39.797468 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=100 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:44:39.845099 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aaad5f48-a258-4e4b-ac0c-4ed0bc03e5b0 None admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:44:39.880698 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08e0ad10-6ecf-4f9d-bae5-04bbac2ec0fb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:44:39.881361 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08e0ad10-6ecf-4f9d-bae5-04bbac2ec0fb None admin] REQUEST_METHOD: `POST` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:44:39.881678 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08e0ad10-6ecf-4f9d-bae5-04bbac2ec0fb None admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:44:39.881983 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08e0ad10-6ecf-4f9d-bae5-04bbac2ec0fb None admin] PATH_INFO: `/v3/users` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:44:39.887025 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08e0ad10-6ecf-4f9d-bae5-04bbac2ec0fb None admin] RBAC: Authorizing `identity:create_user()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:44:39.887025 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08e0ad10-6ecf-4f9d-bae5-04bbac2ec0fb None admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:44:39.964428 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 604/1205] 10.222.0.30 () {60 vars in 1132 bytes} [Mon May 23 12:44:39 2022] POST /identity/v3/users => generated 251 bytes in 221 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) May 23 12:44:42.242796 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e2fc3dd-d4d6-4147-b399-c49ef08f6110 None None] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:44:42.243358 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e2fc3dd-d4d6-4147-b399-c49ef08f6110 None None] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:44:42.243462 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e2fc3dd-d4d6-4147-b399-c49ef08f6110 None None] PATH_INFO: `/` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:44:42.244801 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 602/1206] 10.222.0.30 () {54 vars in 926 bytes} [Mon May 23 12:44:42 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) May 23 12:44:42.250133 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95be5c4d-a1db-41ff-8b31-30e2df388078 None None] REQUEST_METHOD: `POST` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:44:42.255784 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95be5c4d-a1db-41ff-8b31-30e2df388078 None None] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:44:42.255784 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95be5c4d-a1db-41ff-8b31-30e2df388078 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:44:42.268132 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.auth.core [None req-95be5c4d-a1db-41ff-8b31-30e2df388078 None None] MFA Rules not processed for user `0f7119f8fa5c4445977671140ded91a4`. Rule list: `[]` (Enabled: `True`). {{(pid=92057) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 12:44:42.272669 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.fernet_utils [None req-95be5c4d-a1db-41ff-8b31-30e2df388078 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92057) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 12:44:42.336795 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 605/1207] 10.222.0.30 () {58 vars in 1023 bytes} [Mon May 23 12:44:42 2022] POST /identity/v3/auth/tokens => generated 2108 bytes in 88 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) May 23 12:44:42.359943 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e2fc3dd-d4d6-4147-b399-c49ef08f6110 None None] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:44:42.464921 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e2fc3dd-d4d6-4147-b399-c49ef08f6110 None None] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:44:42.498253 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3848308c-abf0-4aed-8556-d180817cbe30 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:44:42.499006 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3848308c-abf0-4aed-8556-d180817cbe30 None admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:44:42.499285 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3848308c-abf0-4aed-8556-d180817cbe30 None admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:44:42.499559 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3848308c-abf0-4aed-8556-d180817cbe30 None admin] PATH_INFO: `/v3/roles/service` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:44:42.510518 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3848308c-abf0-4aed-8556-d180817cbe30 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} May 23 12:44:42.518738 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application [None req-3848308c-abf0-4aed-8556-d180817cbe30 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. May 23 12:44:42.518738 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application Traceback (most recent call last): May 23 12:44:42.518738 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request May 23 12:44:42.518738 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application rv = self.dispatch_request() May 23 12:44:42.518738 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request May 23 12:44:42.518738 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) May 23 12:44:42.518738 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper May 23 12:44:42.518738 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) May 23 12:44:42.518738 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view May 23 12:44:42.518738 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) May 23 12:44:42.518738 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request May 23 12:44:42.518738 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) May 23 12:44:42.518738 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get May 23 12:44:42.518738 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self._get_role(role_id) May 23 12:44:42.518738 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role May 23 12:44:42.518738 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') May 23 12:44:42.522844 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call May 23 12:44:42.522844 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( May 23 12:44:42.522844 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data May 23 12:44:42.522844 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) May 23 12:44:42.522844 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped May 23 12:44:42.522844 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) May 23 12:44:42.522844 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun May 23 12:44:42.522844 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) May 23 12:44:42.522844 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func May 23 12:44:42.522844 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self.get_or_create( May 23 12:44:42.522844 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create May 23 12:44:42.522844 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application with Lock( May 23 12:44:42.522844 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ May 23 12:44:42.522844 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self._enter() May 23 12:44:42.522844 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter May 23 12:44:42.522844 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) May 23 12:44:42.523737 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create May 23 12:44:42.523737 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self.creator() May 23 12:44:42.523737 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value May 23 12:44:42.523737 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application created_value = creator( May 23 12:44:42.523737 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role May 23 12:44:42.523737 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) May 23 12:44:42.523737 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role May 23 12:44:42.523737 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() May 23 12:44:42.523737 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role May 23 12:44:42.523737 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) May 23 12:44:42.523737 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. May 23 12:44:42.523737 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application May 23 12:44:42.523737 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 603/1208] 10.222.0.30 () {56 vars in 1102 bytes} [Mon May 23 12:44:42 2022] GET /identity/v3/roles/service => generated 85 bytes in 160 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) May 23 12:44:42.526143 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95be5c4d-a1db-41ff-8b31-30e2df388078 None None] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:44:42.563491 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:44:42.623465 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95be5c4d-a1db-41ff-8b31-30e2df388078 None None] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:44:42.657800 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23558725-0bf5-4098-8e7f-8d68d994fdeb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:44:42.658513 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23558725-0bf5-4098-8e7f-8d68d994fdeb None admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:44:42.658802 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23558725-0bf5-4098-8e7f-8d68d994fdeb None admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:44:42.659077 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23558725-0bf5-4098-8e7f-8d68d994fdeb None admin] PATH_INFO: `/v3/roles` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:44:42.659810 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-23558725-0bf5-4098-8e7f-8d68d994fdeb None admin] RBAC: Adding query filter params (name=service) {{(pid=92057) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} May 23 12:44:42.660114 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-23558725-0bf5-4098-8e7f-8d68d994fdeb None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:44:42.661492 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-23558725-0bf5-4098-8e7f-8d68d994fdeb None admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:44:42.669332 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 606/1209] 10.222.0.30 () {56 vars in 1103 bytes} [Mon May 23 12:44:42 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 144 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) May 23 12:44:42.675143 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3848308c-abf0-4aed-8556-d180817cbe30 None admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:44:42.726015 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=100 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:44:42.774423 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3848308c-abf0-4aed-8556-d180817cbe30 None admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:44:42.807826 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44269544-4f36-410e-9f83-54df623f92f8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:44:42.808650 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44269544-4f36-410e-9f83-54df623f92f8 None admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:44:42.808940 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44269544-4f36-410e-9f83-54df623f92f8 None admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:44:42.809217 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44269544-4f36-410e-9f83-54df623f92f8 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:44:42.819766 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44269544-4f36-410e-9f83-54df623f92f8 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:44:42.821302 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44269544-4f36-410e-9f83-54df623f92f8 None admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:44:42.839493 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application [None req-44269544-4f36-410e-9f83-54df623f92f8 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. May 23 12:44:42.839493 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application Traceback (most recent call last): May 23 12:44:42.839493 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request May 23 12:44:42.839493 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application rv = self.dispatch_request() May 23 12:44:42.839493 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request May 23 12:44:42.839493 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) May 23 12:44:42.839493 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper May 23 12:44:42.839493 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) May 23 12:44:42.839493 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view May 23 12:44:42.839493 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) May 23 12:44:42.839493 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request May 23 12:44:42.839493 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) May 23 12:44:42.839493 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get May 23 12:44:42.839493 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self._get_domain(domain_id) May 23 12:44:42.839493 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain May 23 12:44:42.840628 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) May 23 12:44:42.840628 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped May 23 12:44:42.840628 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) May 23 12:44:42.840628 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun May 23 12:44:42.840628 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) May 23 12:44:42.840628 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func May 23 12:44:42.840628 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self.get_or_create( May 23 12:44:42.840628 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create May 23 12:44:42.840628 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application with Lock( May 23 12:44:42.840628 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ May 23 12:44:42.840628 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self._enter() May 23 12:44:42.840628 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter May 23 12:44:42.840628 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) May 23 12:44:42.840628 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create May 23 12:44:42.840628 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self.creator() May 23 12:44:42.840628 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value May 23 12:44:42.840628 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application created_value = creator( May 23 12:44:42.841625 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain May 23 12:44:42.841625 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) May 23 12:44:42.841625 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. May 23 12:44:42.841625 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application May 23 12:44:42.841625 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 604/1210] 10.222.0.30 () {56 vars in 1108 bytes} [Mon May 23 12:44:42 2022] GET /identity/v3/domains/Default => generated 87 bytes in 166 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) May 23 12:44:42.846667 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23558725-0bf5-4098-8e7f-8d68d994fdeb None admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:44:42.959210 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=100 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:44:42.962130 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23558725-0bf5-4098-8e7f-8d68d994fdeb None admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:44:42.995226 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12e729d1-0d1b-4f36-90ef-55b1dc42d5c5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:44:42.995884 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12e729d1-0d1b-4f36-90ef-55b1dc42d5c5 None admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:44:42.996173 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12e729d1-0d1b-4f36-90ef-55b1dc42d5c5 None admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:44:42.996448 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12e729d1-0d1b-4f36-90ef-55b1dc42d5c5 None admin] PATH_INFO: `/v3/domains` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:44:42.997173 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12e729d1-0d1b-4f36-90ef-55b1dc42d5c5 None admin] RBAC: Adding query filter params (name=Default) {{(pid=92057) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} May 23 12:44:42.997506 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12e729d1-0d1b-4f36-90ef-55b1dc42d5c5 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:44:42.998925 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12e729d1-0d1b-4f36-90ef-55b1dc42d5c5 None admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:44:43.011237 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 607/1211] 10.222.0.30 () {56 vars in 1109 bytes} [Mon May 23 12:44:42 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 165 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) May 23 12:44:43.016782 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44269544-4f36-410e-9f83-54df623f92f8 None admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:44:43.116748 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44269544-4f36-410e-9f83-54df623f92f8 None admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:44:43.118290 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=100 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:44:43.149460 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e883e2d-d17f-4a62-a62e-59276b621308 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:44:43.150523 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e883e2d-d17f-4a62-a62e-59276b621308 None admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:44:43.150806 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e883e2d-d17f-4a62-a62e-59276b621308 None admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:44:43.151076 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e883e2d-d17f-4a62-a62e-59276b621308 None admin] PATH_INFO: `/v3/users/placement` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:44:43.158677 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e883e2d-d17f-4a62-a62e-59276b621308 None admin] RBAC: Authorizing `identity:get_user(user_id=placement)` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:44:43.162877 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e883e2d-d17f-4a62-a62e-59276b621308 None admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:44:43.175115 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application [None req-2e883e2d-d17f-4a62-a62e-59276b621308 None admin] Could not find user: placement.: keystone.exception.UserNotFound: Could not find user: placement. May 23 12:44:43.175115 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application Traceback (most recent call last): May 23 12:44:43.175115 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request May 23 12:44:43.175115 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application rv = self.dispatch_request() May 23 12:44:43.175115 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request May 23 12:44:43.175115 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) May 23 12:44:43.175115 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper May 23 12:44:43.175115 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) May 23 12:44:43.175115 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view May 23 12:44:43.175115 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) May 23 12:44:43.175115 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request May 23 12:44:43.175115 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) May 23 12:44:43.175115 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get May 23 12:44:43.175115 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self._get_user(user_id) May 23 12:44:43.175115 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user May 23 12:44:43.175115 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) May 23 12:44:43.176289 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped May 23 12:44:43.176289 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) May 23 12:44:43.176289 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper May 23 12:44:43.176289 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) May 23 12:44:43.176289 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper May 23 12:44:43.176289 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) May 23 12:44:43.176289 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun May 23 12:44:43.176289 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) May 23 12:44:43.176289 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func May 23 12:44:43.176289 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self.get_or_create( May 23 12:44:43.176289 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create May 23 12:44:43.176289 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application with Lock( May 23 12:44:43.176289 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ May 23 12:44:43.176289 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self._enter() May 23 12:44:43.176289 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter May 23 12:44:43.176289 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) May 23 12:44:43.176289 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create May 23 12:44:43.176289 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self.creator() May 23 12:44:43.177495 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value May 23 12:44:43.177495 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application created_value = creator( May 23 12:44:43.177495 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user May 23 12:44:43.177495 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) May 23 12:44:43.177495 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user May 23 12:44:43.177495 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) May 23 12:44:43.177495 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user May 23 12:44:43.177495 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) May 23 12:44:43.177495 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: placement. May 23 12:44:43.177495 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application May 23 12:44:43.177495 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 605/1212] 10.222.0.30 () {56 vars in 1108 bytes} [Mon May 23 12:44:43 2022] GET /identity/v3/users/placement => generated 87 bytes in 160 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) May 23 12:44:43.182082 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12e729d1-0d1b-4f36-90ef-55b1dc42d5c5 None admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:44:43.284050 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12e729d1-0d1b-4f36-90ef-55b1dc42d5c5 None admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:44:43.317531 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9a66884-7514-4b0e-82bc-c3d534b4f2ad None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:44:43.318258 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9a66884-7514-4b0e-82bc-c3d534b4f2ad None admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:44:43.318547 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9a66884-7514-4b0e-82bc-c3d534b4f2ad None admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:44:43.318826 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9a66884-7514-4b0e-82bc-c3d534b4f2ad None admin] PATH_INFO: `/v3/users` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:44:43.319660 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9a66884-7514-4b0e-82bc-c3d534b4f2ad None admin] RBAC: Adding query filter params (domain_id=default, name=placement) {{(pid=92057) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} May 23 12:44:43.319974 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9a66884-7514-4b0e-82bc-c3d534b4f2ad None admin] RBAC: Authorizing `identity:list_users()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:44:43.321410 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9a66884-7514-4b0e-82bc-c3d534b4f2ad None admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:44:43.338790 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 608/1213] 10.222.0.30 () {56 vars in 1143 bytes} [Mon May 23 12:44:43 2022] GET /identity/v3/users?domain_id=default&name=placement => generated 378 bytes in 157 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) May 23 12:44:43.344814 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e883e2d-d17f-4a62-a62e-59276b621308 None admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:44:43.442880 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e883e2d-d17f-4a62-a62e-59276b621308 None admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:44:43.476132 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-874fd720-daf5-48fc-8255-b0d79a083c8b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:44:43.476775 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-874fd720-daf5-48fc-8255-b0d79a083c8b None admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:44:43.477059 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-874fd720-daf5-48fc-8255-b0d79a083c8b None admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:44:43.477338 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-874fd720-daf5-48fc-8255-b0d79a083c8b None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:44:43.508310 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=100 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:44:43.544797 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-874fd720-daf5-48fc-8255-b0d79a083c8b None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:44:43.546350 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-874fd720-daf5-48fc-8255-b0d79a083c8b None admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:44:43.647621 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 606/1214] 10.222.0.30 () {58 vars in 1282 bytes} [Mon May 23 12:44:43 2022] GET /identity/v3/auth/tokens => generated 2108 bytes in 303 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) May 23 12:44:43.653840 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9a66884-7514-4b0e-82bc-c3d534b4f2ad None admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:44:43.672600 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=100 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:44:43.751428 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9a66884-7514-4b0e-82bc-c3d534b4f2ad None admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:44:43.785492 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f365927f-159d-412a-be76-1ce8b16d0dd6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:44:43.787594 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f365927f-159d-412a-be76-1ce8b16d0dd6 None admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:44:43.787594 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f365927f-159d-412a-be76-1ce8b16d0dd6 None admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:44:43.787594 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f365927f-159d-412a-be76-1ce8b16d0dd6 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:44:43.796729 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f365927f-159d-412a-be76-1ce8b16d0dd6 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:44:43.798211 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f365927f-159d-412a-be76-1ce8b16d0dd6 None admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:44:43.816466 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application [None req-f365927f-159d-412a-be76-1ce8b16d0dd6 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. May 23 12:44:43.816466 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application Traceback (most recent call last): May 23 12:44:43.816466 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request May 23 12:44:43.816466 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application rv = self.dispatch_request() May 23 12:44:43.816466 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request May 23 12:44:43.816466 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) May 23 12:44:43.816466 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper May 23 12:44:43.816466 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) May 23 12:44:43.816466 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view May 23 12:44:43.816466 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) May 23 12:44:43.816466 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request May 23 12:44:43.816466 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) May 23 12:44:43.816466 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get May 23 12:44:43.816466 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self._get_domain(domain_id) May 23 12:44:43.816466 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain May 23 12:44:43.817549 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) May 23 12:44:43.817549 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped May 23 12:44:43.817549 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) May 23 12:44:43.817549 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun May 23 12:44:43.817549 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) May 23 12:44:43.817549 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func May 23 12:44:43.817549 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self.get_or_create( May 23 12:44:43.817549 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create May 23 12:44:43.817549 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application with Lock( May 23 12:44:43.817549 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ May 23 12:44:43.817549 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self._enter() May 23 12:44:43.817549 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter May 23 12:44:43.817549 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) May 23 12:44:43.817549 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create May 23 12:44:43.817549 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self.creator() May 23 12:44:43.817549 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value May 23 12:44:43.817549 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application created_value = creator( May 23 12:44:43.818565 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain May 23 12:44:43.818565 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) May 23 12:44:43.818565 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. May 23 12:44:43.818565 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application May 23 12:44:43.818565 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 609/1215] 10.222.0.30 () {56 vars in 1108 bytes} [Mon May 23 12:44:43 2022] GET /identity/v3/domains/Default => generated 87 bytes in 164 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) May 23 12:44:43.823591 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-874fd720-daf5-48fc-8255-b0d79a083c8b None admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:44:43.883644 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=100 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:44:43.920438 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-874fd720-daf5-48fc-8255-b0d79a083c8b None admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:44:43.953569 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8791d48-c21b-42f6-b3f5-6aff4f516a16 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:44:43.954179 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8791d48-c21b-42f6-b3f5-6aff4f516a16 None admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:44:43.954461 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8791d48-c21b-42f6-b3f5-6aff4f516a16 None admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:44:43.954735 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8791d48-c21b-42f6-b3f5-6aff4f516a16 None admin] PATH_INFO: `/v3/domains` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:44:43.956063 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8791d48-c21b-42f6-b3f5-6aff4f516a16 None admin] RBAC: Adding query filter params (name=Default) {{(pid=92056) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} May 23 12:44:43.956503 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8791d48-c21b-42f6-b3f5-6aff4f516a16 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:44:43.957920 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8791d48-c21b-42f6-b3f5-6aff4f516a16 None admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:44:43.969845 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 607/1216] 10.222.0.30 () {56 vars in 1109 bytes} [Mon May 23 12:44:43 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 147 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) May 23 12:44:43.975331 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f365927f-159d-412a-be76-1ce8b16d0dd6 None admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:44:44.043452 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=100 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:44:44.072651 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f365927f-159d-412a-be76-1ce8b16d0dd6 None admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:44:44.105628 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66f41861-3b60-4cdb-b1ce-5c497a8ac506 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:44:44.106435 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66f41861-3b60-4cdb-b1ce-5c497a8ac506 None admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:44:44.117772 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66f41861-3b60-4cdb-b1ce-5c497a8ac506 None admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:44:44.117772 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66f41861-3b60-4cdb-b1ce-5c497a8ac506 None admin] PATH_INFO: `/v3/projects/service` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:44:44.117772 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66f41861-3b60-4cdb-b1ce-5c497a8ac506 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:44:44.117772 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66f41861-3b60-4cdb-b1ce-5c497a8ac506 None admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:44:44.129122 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application [None req-66f41861-3b60-4cdb-b1ce-5c497a8ac506 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. May 23 12:44:44.129122 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application Traceback (most recent call last): May 23 12:44:44.129122 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request May 23 12:44:44.129122 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application rv = self.dispatch_request() May 23 12:44:44.129122 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request May 23 12:44:44.129122 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) May 23 12:44:44.129122 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper May 23 12:44:44.129122 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) May 23 12:44:44.129122 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view May 23 12:44:44.129122 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) May 23 12:44:44.129122 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request May 23 12:44:44.129122 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) May 23 12:44:44.129122 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get May 23 12:44:44.129122 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self._get_project(project_id) May 23 12:44:44.129122 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project May 23 12:44:44.130258 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) May 23 12:44:44.130258 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped May 23 12:44:44.130258 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) May 23 12:44:44.130258 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun May 23 12:44:44.130258 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) May 23 12:44:44.130258 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func May 23 12:44:44.130258 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self.get_or_create( May 23 12:44:44.130258 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create May 23 12:44:44.130258 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application with Lock( May 23 12:44:44.130258 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ May 23 12:44:44.130258 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self._enter() May 23 12:44:44.130258 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter May 23 12:44:44.130258 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) May 23 12:44:44.130258 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create May 23 12:44:44.130258 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self.creator() May 23 12:44:44.130258 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value May 23 12:44:44.130258 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application created_value = creator( May 23 12:44:44.131197 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project May 23 12:44:44.131197 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) May 23 12:44:44.131197 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project May 23 12:44:44.131197 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() May 23 12:44:44.131197 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project May 23 12:44:44.131197 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) May 23 12:44:44.131197 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. May 23 12:44:44.131197 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application May 23 12:44:44.131197 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 610/1217] 10.222.0.30 () {56 vars in 1111 bytes} [Mon May 23 12:44:43 2022] GET /identity/v3/projects/service => generated 88 bytes in 156 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) May 23 12:44:44.136386 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8791d48-c21b-42f6-b3f5-6aff4f516a16 None admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:44:44.234581 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8791d48-c21b-42f6-b3f5-6aff4f516a16 None admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:44:44.256825 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=100 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:44:44.270290 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8243372f-c49c-46f4-8235-e81bbd0de0c5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:44:44.270941 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8243372f-c49c-46f4-8235-e81bbd0de0c5 None admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:44:44.271221 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8243372f-c49c-46f4-8235-e81bbd0de0c5 None admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:44:44.271492 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8243372f-c49c-46f4-8235-e81bbd0de0c5 None admin] PATH_INFO: `/v3/projects` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:44:44.272263 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8243372f-c49c-46f4-8235-e81bbd0de0c5 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=92056) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} May 23 12:44:44.272568 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8243372f-c49c-46f4-8235-e81bbd0de0c5 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:44:44.274026 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8243372f-c49c-46f4-8235-e81bbd0de0c5 None admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:44:44.286767 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 608/1218] 10.222.0.30 () {56 vars in 1148 bytes} [Mon May 23 12:44:44 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 151 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) May 23 12:44:44.292916 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66f41861-3b60-4cdb-b1ce-5c497a8ac506 None admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:44:44.394233 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66f41861-3b60-4cdb-b1ce-5c497a8ac506 None admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:44:44.422694 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=100 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:44:44.428878 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-454cb3c7-7b1d-49e8-b4c8-04295ca36441 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:44:44.429523 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-454cb3c7-7b1d-49e8-b4c8-04295ca36441 None admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:44:44.429807 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-454cb3c7-7b1d-49e8-b4c8-04295ca36441 None admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:44:44.430116 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-454cb3c7-7b1d-49e8-b4c8-04295ca36441 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:44:44.430984 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-454cb3c7-7b1d-49e8-b4c8-04295ca36441 None admin] RBAC: Adding query filter params (role.id=41ab28c1ae3d4941bd09fb5d35f1dc54, scope.project.id=9c389ae65aaa4542bb5a5c4fd0bacf5b, user.id=d5fb51f8d0fa46439cb29666970b885b) {{(pid=92057) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} May 23 12:44:44.431294 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-454cb3c7-7b1d-49e8-b4c8-04295ca36441 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:44:44.432736 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-454cb3c7-7b1d-49e8-b4c8-04295ca36441 None admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:44:44.438089 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 611/1219] 10.222.0.30 () {56 vars in 1374 bytes} [Mon May 23 12:44:44 2022] GET /identity/v3/role_assignments?user.id=d5fb51f8d0fa46439cb29666970b885b&scope.project.id=9c389ae65aaa4542bb5a5c4fd0bacf5b&role.id=41ab28c1ae3d4941bd09fb5d35f1dc54 => generated 259 bytes in 146 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) May 23 12:44:46.587230 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-445d8cef-76f6-4e09-87d1-bf831071ba26 None None] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:44:46.587754 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-445d8cef-76f6-4e09-87d1-bf831071ba26 None None] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:44:46.587865 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-445d8cef-76f6-4e09-87d1-bf831071ba26 None None] PATH_INFO: `/` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:44:46.589170 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 609/1220] 10.222.0.30 () {54 vars in 926 bytes} [Mon May 23 12:44:46 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) May 23 12:44:46.594422 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1686b126-e6d4-4eb9-bec0-3890fdc8aef1 None None] REQUEST_METHOD: `POST` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:44:46.594848 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1686b126-e6d4-4eb9-bec0-3890fdc8aef1 None None] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:44:46.594981 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1686b126-e6d4-4eb9-bec0-3890fdc8aef1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:44:46.612973 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.auth.core [None req-1686b126-e6d4-4eb9-bec0-3890fdc8aef1 None None] MFA Rules not processed for user `0f7119f8fa5c4445977671140ded91a4`. Rule list: `[]` (Enabled: `True`). {{(pid=92057) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 12:44:46.614185 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.fernet_utils [None req-1686b126-e6d4-4eb9-bec0-3890fdc8aef1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92057) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 12:44:46.679520 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 612/1221] 10.222.0.30 () {58 vars in 1023 bytes} [Mon May 23 12:44:46 2022] POST /identity/v3/auth/tokens => generated 2108 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) May 23 12:44:46.702372 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-445d8cef-76f6-4e09-87d1-bf831071ba26 None None] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:44:46.803691 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-445d8cef-76f6-4e09-87d1-bf831071ba26 None None] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:44:46.837349 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-620024d9-e6b5-4d88-884a-caa1f3368227 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:44:46.838099 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-620024d9-e6b5-4d88-884a-caa1f3368227 None admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:44:46.838378 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-620024d9-e6b5-4d88-884a-caa1f3368227 None admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:44:46.838648 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-620024d9-e6b5-4d88-884a-caa1f3368227 None admin] PATH_INFO: `/v3/roles/service` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:44:46.848572 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-620024d9-e6b5-4d88-884a-caa1f3368227 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} May 23 12:44:46.856328 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application [None req-620024d9-e6b5-4d88-884a-caa1f3368227 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. May 23 12:44:46.856328 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application Traceback (most recent call last): May 23 12:44:46.856328 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request May 23 12:44:46.856328 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application rv = self.dispatch_request() May 23 12:44:46.856328 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request May 23 12:44:46.856328 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) May 23 12:44:46.856328 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper May 23 12:44:46.856328 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) May 23 12:44:46.856328 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view May 23 12:44:46.856328 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) May 23 12:44:46.856328 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request May 23 12:44:46.856328 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) May 23 12:44:46.856328 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get May 23 12:44:46.856328 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self._get_role(role_id) May 23 12:44:46.856328 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role May 23 12:44:46.856328 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') May 23 12:44:46.857497 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call May 23 12:44:46.857497 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( May 23 12:44:46.857497 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data May 23 12:44:46.857497 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) May 23 12:44:46.857497 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped May 23 12:44:46.857497 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) May 23 12:44:46.857497 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun May 23 12:44:46.857497 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) May 23 12:44:46.857497 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func May 23 12:44:46.857497 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self.get_or_create( May 23 12:44:46.857497 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create May 23 12:44:46.857497 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application with Lock( May 23 12:44:46.857497 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ May 23 12:44:46.857497 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self._enter() May 23 12:44:46.857497 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter May 23 12:44:46.857497 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) May 23 12:44:46.858791 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create May 23 12:44:46.858791 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self.creator() May 23 12:44:46.858791 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value May 23 12:44:46.858791 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application created_value = creator( May 23 12:44:46.858791 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role May 23 12:44:46.858791 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) May 23 12:44:46.858791 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role May 23 12:44:46.858791 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() May 23 12:44:46.858791 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role May 23 12:44:46.858791 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) May 23 12:44:46.858791 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. May 23 12:44:46.858791 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application May 23 12:44:46.858791 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 610/1222] 10.222.0.30 () {56 vars in 1102 bytes} [Mon May 23 12:44:46 2022] GET /identity/v3/roles/service => generated 85 bytes in 156 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) May 23 12:44:46.864047 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1686b126-e6d4-4eb9-bec0-3890fdc8aef1 None None] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:44:46.968737 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1686b126-e6d4-4eb9-bec0-3890fdc8aef1 None None] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:44:46.992275 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=100 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:44:47.002704 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f9e464a-a215-47ae-b1cd-e8cf90a58eb4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:44:47.003408 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f9e464a-a215-47ae-b1cd-e8cf90a58eb4 None admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:44:47.003691 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f9e464a-a215-47ae-b1cd-e8cf90a58eb4 None admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:44:47.003964 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f9e464a-a215-47ae-b1cd-e8cf90a58eb4 None admin] PATH_INFO: `/v3/roles` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:44:47.004729 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f9e464a-a215-47ae-b1cd-e8cf90a58eb4 None admin] RBAC: Adding query filter params (name=service) {{(pid=92057) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} May 23 12:44:47.005032 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f9e464a-a215-47ae-b1cd-e8cf90a58eb4 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:44:47.006432 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f9e464a-a215-47ae-b1cd-e8cf90a58eb4 None admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:44:47.014127 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 613/1223] 10.222.0.30 () {56 vars in 1103 bytes} [Mon May 23 12:44:46 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 151 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) May 23 12:44:47.020132 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-620024d9-e6b5-4d88-884a-caa1f3368227 None admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:44:47.063804 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=100 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:44:47.120380 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-620024d9-e6b5-4d88-884a-caa1f3368227 None admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:44:47.154412 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d41dfb66-3ac6-453f-80cd-f91db7022bf6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:44:47.155190 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d41dfb66-3ac6-453f-80cd-f91db7022bf6 None admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:44:47.155472 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d41dfb66-3ac6-453f-80cd-f91db7022bf6 None admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:44:47.155850 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d41dfb66-3ac6-453f-80cd-f91db7022bf6 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:44:47.168390 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d41dfb66-3ac6-453f-80cd-f91db7022bf6 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:44:47.169892 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d41dfb66-3ac6-453f-80cd-f91db7022bf6 None admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:44:47.187835 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application [None req-d41dfb66-3ac6-453f-80cd-f91db7022bf6 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. May 23 12:44:47.187835 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application Traceback (most recent call last): May 23 12:44:47.187835 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request May 23 12:44:47.187835 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application rv = self.dispatch_request() May 23 12:44:47.187835 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request May 23 12:44:47.187835 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) May 23 12:44:47.187835 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper May 23 12:44:47.187835 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) May 23 12:44:47.187835 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view May 23 12:44:47.187835 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) May 23 12:44:47.187835 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request May 23 12:44:47.187835 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) May 23 12:44:47.187835 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get May 23 12:44:47.187835 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self._get_domain(domain_id) May 23 12:44:47.187835 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain May 23 12:44:47.188937 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) May 23 12:44:47.188937 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped May 23 12:44:47.188937 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) May 23 12:44:47.188937 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun May 23 12:44:47.188937 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) May 23 12:44:47.188937 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func May 23 12:44:47.188937 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self.get_or_create( May 23 12:44:47.188937 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create May 23 12:44:47.188937 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application with Lock( May 23 12:44:47.188937 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ May 23 12:44:47.188937 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self._enter() May 23 12:44:47.188937 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter May 23 12:44:47.188937 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) May 23 12:44:47.188937 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create May 23 12:44:47.188937 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self.creator() May 23 12:44:47.188937 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value May 23 12:44:47.188937 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application created_value = creator( May 23 12:44:47.189981 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain May 23 12:44:47.189981 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) May 23 12:44:47.189981 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. May 23 12:44:47.189981 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application May 23 12:44:47.189981 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 611/1224] 10.222.0.30 () {56 vars in 1108 bytes} [Mon May 23 12:44:47 2022] GET /identity/v3/domains/Default => generated 87 bytes in 169 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) May 23 12:44:47.195523 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f9e464a-a215-47ae-b1cd-e8cf90a58eb4 None admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:44:47.295852 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f9e464a-a215-47ae-b1cd-e8cf90a58eb4 None admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:44:47.329160 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70ce3f40-ff05-4c4f-a204-7a003e3540ff None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:44:47.329852 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70ce3f40-ff05-4c4f-a204-7a003e3540ff None admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:44:47.330168 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70ce3f40-ff05-4c4f-a204-7a003e3540ff None admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:44:47.331391 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70ce3f40-ff05-4c4f-a204-7a003e3540ff None admin] PATH_INFO: `/v3/domains` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:44:47.331810 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70ce3f40-ff05-4c4f-a204-7a003e3540ff None admin] RBAC: Adding query filter params (name=Default) {{(pid=92057) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} May 23 12:44:47.332114 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70ce3f40-ff05-4c4f-a204-7a003e3540ff None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:44:47.333483 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70ce3f40-ff05-4c4f-a204-7a003e3540ff None admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:44:47.344965 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 614/1225] 10.222.0.30 () {56 vars in 1109 bytes} [Mon May 23 12:44:47 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 150 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) May 23 12:44:47.350846 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d41dfb66-3ac6-453f-80cd-f91db7022bf6 None admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:44:47.443622 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=100 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:44:47.450037 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d41dfb66-3ac6-453f-80cd-f91db7022bf6 None admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:44:47.482280 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b98ec09-4ea6-4e51-8c9e-e7219b26b035 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:44:47.483120 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b98ec09-4ea6-4e51-8c9e-e7219b26b035 None admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:44:47.483402 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b98ec09-4ea6-4e51-8c9e-e7219b26b035 None admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:44:47.483679 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b98ec09-4ea6-4e51-8c9e-e7219b26b035 None admin] PATH_INFO: `/v3/users/placement` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:44:47.491731 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b98ec09-4ea6-4e51-8c9e-e7219b26b035 None admin] RBAC: Authorizing `identity:get_user(user_id=placement)` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:44:47.493179 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b98ec09-4ea6-4e51-8c9e-e7219b26b035 None admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:44:47.508101 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application [None req-1b98ec09-4ea6-4e51-8c9e-e7219b26b035 None admin] Could not find user: placement.: keystone.exception.UserNotFound: Could not find user: placement. May 23 12:44:47.508101 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application Traceback (most recent call last): May 23 12:44:47.508101 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request May 23 12:44:47.508101 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application rv = self.dispatch_request() May 23 12:44:47.508101 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request May 23 12:44:47.508101 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) May 23 12:44:47.508101 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper May 23 12:44:47.508101 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) May 23 12:44:47.508101 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view May 23 12:44:47.508101 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) May 23 12:44:47.508101 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request May 23 12:44:47.508101 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) May 23 12:44:47.508101 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get May 23 12:44:47.508101 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self._get_user(user_id) May 23 12:44:47.508101 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user May 23 12:44:47.508101 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) May 23 12:44:47.509271 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped May 23 12:44:47.509271 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) May 23 12:44:47.509271 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper May 23 12:44:47.509271 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) May 23 12:44:47.509271 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper May 23 12:44:47.509271 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) May 23 12:44:47.509271 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun May 23 12:44:47.509271 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) May 23 12:44:47.509271 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func May 23 12:44:47.509271 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self.get_or_create( May 23 12:44:47.509271 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create May 23 12:44:47.509271 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application with Lock( May 23 12:44:47.509271 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ May 23 12:44:47.509271 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self._enter() May 23 12:44:47.509271 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter May 23 12:44:47.509271 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) May 23 12:44:47.509271 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create May 23 12:44:47.509271 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self.creator() May 23 12:44:47.510435 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value May 23 12:44:47.510435 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application created_value = creator( May 23 12:44:47.510435 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user May 23 12:44:47.510435 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) May 23 12:44:47.510435 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user May 23 12:44:47.510435 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) May 23 12:44:47.510435 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user May 23 12:44:47.510435 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) May 23 12:44:47.510435 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: placement. May 23 12:44:47.510435 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application May 23 12:44:47.510435 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 612/1226] 10.222.0.30 () {56 vars in 1108 bytes} [Mon May 23 12:44:47 2022] GET /identity/v3/users/placement => generated 87 bytes in 159 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) May 23 12:44:47.515196 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70ce3f40-ff05-4c4f-a204-7a003e3540ff None admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:44:47.562986 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=100 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:44:47.616004 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70ce3f40-ff05-4c4f-a204-7a003e3540ff None admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:44:47.650424 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7a0497d-bf16-4fd4-b753-149504dc6558 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:44:47.651132 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7a0497d-bf16-4fd4-b753-149504dc6558 None admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:44:47.651458 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7a0497d-bf16-4fd4-b753-149504dc6558 None admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:44:47.651775 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7a0497d-bf16-4fd4-b753-149504dc6558 None admin] PATH_INFO: `/v3/users` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:44:47.652648 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7a0497d-bf16-4fd4-b753-149504dc6558 None admin] RBAC: Adding query filter params (domain_id=default, name=placement) {{(pid=92057) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} May 23 12:44:47.653001 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7a0497d-bf16-4fd4-b753-149504dc6558 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:44:47.654622 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7a0497d-bf16-4fd4-b753-149504dc6558 None admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:44:47.672504 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 615/1227] 10.222.0.30 () {56 vars in 1143 bytes} [Mon May 23 12:44:47 2022] GET /identity/v3/users?domain_id=default&name=placement => generated 378 bytes in 158 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) May 23 12:44:47.678888 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b98ec09-4ea6-4e51-8c9e-e7219b26b035 None admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:44:47.778314 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b98ec09-4ea6-4e51-8c9e-e7219b26b035 None admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:44:47.812687 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-060a91e5-4a94-4627-b3d1-a3d25c2d175a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:44:47.813476 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-060a91e5-4a94-4627-b3d1-a3d25c2d175a None admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:44:47.813773 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-060a91e5-4a94-4627-b3d1-a3d25c2d175a None admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:44:47.814067 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-060a91e5-4a94-4627-b3d1-a3d25c2d175a None admin] PATH_INFO: `/v3/domains/Default` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:44:47.824936 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-060a91e5-4a94-4627-b3d1-a3d25c2d175a None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:44:47.826418 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-060a91e5-4a94-4627-b3d1-a3d25c2d175a None admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:44:47.843025 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application [None req-060a91e5-4a94-4627-b3d1-a3d25c2d175a None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. May 23 12:44:47.843025 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application Traceback (most recent call last): May 23 12:44:47.843025 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request May 23 12:44:47.843025 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application rv = self.dispatch_request() May 23 12:44:47.843025 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request May 23 12:44:47.843025 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) May 23 12:44:47.843025 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper May 23 12:44:47.843025 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) May 23 12:44:47.843025 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view May 23 12:44:47.843025 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) May 23 12:44:47.843025 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request May 23 12:44:47.843025 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) May 23 12:44:47.843025 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get May 23 12:44:47.843025 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self._get_domain(domain_id) May 23 12:44:47.843025 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain May 23 12:44:47.844134 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) May 23 12:44:47.844134 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped May 23 12:44:47.844134 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) May 23 12:44:47.844134 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun May 23 12:44:47.844134 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) May 23 12:44:47.844134 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func May 23 12:44:47.844134 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self.get_or_create( May 23 12:44:47.844134 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create May 23 12:44:47.844134 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application with Lock( May 23 12:44:47.844134 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ May 23 12:44:47.844134 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self._enter() May 23 12:44:47.844134 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter May 23 12:44:47.844134 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) May 23 12:44:47.844134 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create May 23 12:44:47.844134 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self.creator() May 23 12:44:47.844134 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value May 23 12:44:47.844134 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application created_value = creator( May 23 12:44:47.845304 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain May 23 12:44:47.845304 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) May 23 12:44:47.845304 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. May 23 12:44:47.845304 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application May 23 12:44:47.845304 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 613/1228] 10.222.0.30 () {56 vars in 1108 bytes} [Mon May 23 12:44:47 2022] GET /identity/v3/domains/Default => generated 87 bytes in 166 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) May 23 12:44:47.850912 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7a0497d-bf16-4fd4-b753-149504dc6558 None admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:44:47.948821 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7a0497d-bf16-4fd4-b753-149504dc6558 None admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:44:47.951005 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=100 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:44:47.981378 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0cc3f762-e382-47e6-94bd-50e82a2e5216 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:44:47.982057 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cc3f762-e382-47e6-94bd-50e82a2e5216 None admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:44:47.982342 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cc3f762-e382-47e6-94bd-50e82a2e5216 None admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:44:47.982616 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cc3f762-e382-47e6-94bd-50e82a2e5216 None admin] PATH_INFO: `/v3/domains` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:44:47.983324 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0cc3f762-e382-47e6-94bd-50e82a2e5216 None admin] RBAC: Adding query filter params (name=Default) {{(pid=92057) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} May 23 12:44:47.983655 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0cc3f762-e382-47e6-94bd-50e82a2e5216 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:44:47.985002 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0cc3f762-e382-47e6-94bd-50e82a2e5216 None admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:44:47.996260 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 616/1229] 10.222.0.30 () {56 vars in 1109 bytes} [Mon May 23 12:44:47 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 146 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) May 23 12:44:48.001868 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-060a91e5-4a94-4627-b3d1-a3d25c2d175a None admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:44:48.011946 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=100 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:44:48.099200 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-060a91e5-4a94-4627-b3d1-a3d25c2d175a None admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:44:48.132085 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-651b59a9-f344-4525-9859-2c224a5618b1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:44:48.132871 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-651b59a9-f344-4525-9859-2c224a5618b1 None admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:44:48.133154 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-651b59a9-f344-4525-9859-2c224a5618b1 None admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:44:48.133429 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-651b59a9-f344-4525-9859-2c224a5618b1 None admin] PATH_INFO: `/v3/projects/service` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:44:48.139862 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-651b59a9-f344-4525-9859-2c224a5618b1 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:44:48.141321 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-651b59a9-f344-4525-9859-2c224a5618b1 None admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:44:48.157040 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application [None req-651b59a9-f344-4525-9859-2c224a5618b1 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. May 23 12:44:48.157040 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application Traceback (most recent call last): May 23 12:44:48.157040 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request May 23 12:44:48.157040 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application rv = self.dispatch_request() May 23 12:44:48.157040 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request May 23 12:44:48.157040 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) May 23 12:44:48.157040 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper May 23 12:44:48.157040 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) May 23 12:44:48.157040 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view May 23 12:44:48.157040 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) May 23 12:44:48.157040 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request May 23 12:44:48.157040 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) May 23 12:44:48.157040 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get May 23 12:44:48.157040 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self._get_project(project_id) May 23 12:44:48.157040 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project May 23 12:44:48.158101 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) May 23 12:44:48.158101 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped May 23 12:44:48.158101 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) May 23 12:44:48.158101 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun May 23 12:44:48.158101 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) May 23 12:44:48.158101 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func May 23 12:44:48.158101 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self.get_or_create( May 23 12:44:48.158101 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create May 23 12:44:48.158101 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application with Lock( May 23 12:44:48.158101 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ May 23 12:44:48.158101 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self._enter() May 23 12:44:48.158101 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter May 23 12:44:48.158101 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) May 23 12:44:48.158101 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create May 23 12:44:48.158101 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self.creator() May 23 12:44:48.158101 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value May 23 12:44:48.158101 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application created_value = creator( May 23 12:44:48.159713 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project May 23 12:44:48.159713 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) May 23 12:44:48.159713 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project May 23 12:44:48.159713 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() May 23 12:44:48.159713 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project May 23 12:44:48.159713 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) May 23 12:44:48.159713 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. May 23 12:44:48.159713 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application May 23 12:44:48.159713 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 614/1230] 10.222.0.30 () {56 vars in 1111 bytes} [Mon May 23 12:44:48 2022] GET /identity/v3/projects/service => generated 88 bytes in 157 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) May 23 12:44:48.164705 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0cc3f762-e382-47e6-94bd-50e82a2e5216 None admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:44:48.262773 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0cc3f762-e382-47e6-94bd-50e82a2e5216 None admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:44:48.296590 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e06289c4-5cd9-4da0-837c-acab092df7f8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:44:48.297239 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e06289c4-5cd9-4da0-837c-acab092df7f8 None admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:44:48.297524 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e06289c4-5cd9-4da0-837c-acab092df7f8 None admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:44:48.297798 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e06289c4-5cd9-4da0-837c-acab092df7f8 None admin] PATH_INFO: `/v3/projects` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:44:48.298605 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e06289c4-5cd9-4da0-837c-acab092df7f8 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=92057) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} May 23 12:44:48.298914 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e06289c4-5cd9-4da0-837c-acab092df7f8 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:44:48.300422 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e06289c4-5cd9-4da0-837c-acab092df7f8 None admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:44:48.312113 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 617/1231] 10.222.0.30 () {56 vars in 1148 bytes} [Mon May 23 12:44:48 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 148 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) May 23 12:44:48.318112 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-651b59a9-f344-4525-9859-2c224a5618b1 None admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:44:48.383201 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=100 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:44:48.417508 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-651b59a9-f344-4525-9859-2c224a5618b1 None admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:44:48.451636 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ce8e8d4-f98d-44be-b1ac-94625f2c9c91 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:44:48.452505 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ce8e8d4-f98d-44be-b1ac-94625f2c9c91 None admin] REQUEST_METHOD: `PUT` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:44:48.452779 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ce8e8d4-f98d-44be-b1ac-94625f2c9c91 None admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:44:48.453048 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ce8e8d4-f98d-44be-b1ac-94625f2c9c91 None admin] PATH_INFO: `/v3/projects/9c389ae65aaa4542bb5a5c4fd0bacf5b/users/d5fb51f8d0fa46439cb29666970b885b/roles/41ab28c1ae3d4941bd09fb5d35f1dc54` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:44:48.467442 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ce8e8d4-f98d-44be-b1ac-94625f2c9c91 None admin] RBAC: Authorizing `identity:create_grant(project_id=9c389ae65aaa4542bb5a5c4fd0bacf5b, user_id=d5fb51f8d0fa46439cb29666970b885b, role_id=41ab28c1ae3d4941bd09fb5d35f1dc54)` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:44:48.469619 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ce8e8d4-f98d-44be-b1ac-94625f2c9c91 None admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:44:48.552528 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 615/1232] 10.222.0.30 () {58 vars in 1439 bytes} [Mon May 23 12:44:48 2022] PUT /identity/v3/projects/9c389ae65aaa4542bb5a5c4fd0bacf5b/users/d5fb51f8d0fa46439cb29666970b885b/roles/41ab28c1ae3d4941bd09fb5d35f1dc54 => generated 0 bytes in 235 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) May 23 12:44:50.761371 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf79742b-b193-4945-863e-ce96c5ecfc7b None None] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:44:50.761862 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf79742b-b193-4945-863e-ce96c5ecfc7b None None] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:44:50.762023 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf79742b-b193-4945-863e-ce96c5ecfc7b None None] PATH_INFO: `/` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:44:50.763348 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 618/1233] 10.222.0.30 () {54 vars in 926 bytes} [Mon May 23 12:44:50 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) May 23 12:44:50.768515 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01112398-c29a-4df3-8bfc-51d162b87dd7 None None] REQUEST_METHOD: `POST` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:44:50.769311 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01112398-c29a-4df3-8bfc-51d162b87dd7 None None] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:44:50.769594 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01112398-c29a-4df3-8bfc-51d162b87dd7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:44:50.786332 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.auth.core [None req-01112398-c29a-4df3-8bfc-51d162b87dd7 None None] MFA Rules not processed for user `0f7119f8fa5c4445977671140ded91a4`. Rule list: `[]` (Enabled: `True`). {{(pid=92056) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 12:44:50.787549 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.fernet_utils [None req-01112398-c29a-4df3-8bfc-51d162b87dd7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92056) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 12:44:50.853391 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 616/1234] 10.222.0.30 () {58 vars in 1023 bytes} [Mon May 23 12:44:50 2022] POST /identity/v3/auth/tokens => generated 2108 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) May 23 12:44:50.877065 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf79742b-b193-4945-863e-ce96c5ecfc7b None None] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:44:50.902866 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=100 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:44:50.977487 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf79742b-b193-4945-863e-ce96c5ecfc7b None None] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:44:51.010924 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7f003e1-068e-48d1-9d51-ea3d7bb08907 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:44:51.011705 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7f003e1-068e-48d1-9d51-ea3d7bb08907 None admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:44:51.011983 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7f003e1-068e-48d1-9d51-ea3d7bb08907 None admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:44:51.012252 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7f003e1-068e-48d1-9d51-ea3d7bb08907 None admin] PATH_INFO: `/v3/roles/service` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:44:51.022422 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c7f003e1-068e-48d1-9d51-ea3d7bb08907 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} May 23 12:44:51.030407 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application [None req-c7f003e1-068e-48d1-9d51-ea3d7bb08907 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. May 23 12:44:51.030407 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application Traceback (most recent call last): May 23 12:44:51.030407 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request May 23 12:44:51.030407 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application rv = self.dispatch_request() May 23 12:44:51.030407 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request May 23 12:44:51.030407 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) May 23 12:44:51.030407 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper May 23 12:44:51.030407 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) May 23 12:44:51.030407 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view May 23 12:44:51.030407 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) May 23 12:44:51.030407 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request May 23 12:44:51.030407 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) May 23 12:44:51.030407 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get May 23 12:44:51.030407 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self._get_role(role_id) May 23 12:44:51.030407 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role May 23 12:44:51.030407 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') May 23 12:44:51.031544 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call May 23 12:44:51.031544 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( May 23 12:44:51.031544 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data May 23 12:44:51.031544 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) May 23 12:44:51.031544 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped May 23 12:44:51.031544 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) May 23 12:44:51.031544 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun May 23 12:44:51.031544 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) May 23 12:44:51.031544 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func May 23 12:44:51.031544 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self.get_or_create( May 23 12:44:51.031544 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create May 23 12:44:51.031544 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application with Lock( May 23 12:44:51.031544 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ May 23 12:44:51.031544 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self._enter() May 23 12:44:51.031544 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter May 23 12:44:51.031544 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) May 23 12:44:51.032633 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create May 23 12:44:51.032633 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self.creator() May 23 12:44:51.032633 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value May 23 12:44:51.032633 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application created_value = creator( May 23 12:44:51.032633 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role May 23 12:44:51.032633 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) May 23 12:44:51.032633 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role May 23 12:44:51.032633 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() May 23 12:44:51.032633 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role May 23 12:44:51.032633 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) May 23 12:44:51.032633 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. May 23 12:44:51.032633 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application May 23 12:44:51.032633 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 619/1235] 10.222.0.30 () {56 vars in 1102 bytes} [Mon May 23 12:44:50 2022] GET /identity/v3/roles/service => generated 85 bytes in 155 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) May 23 12:44:51.038119 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01112398-c29a-4df3-8bfc-51d162b87dd7 None None] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:44:51.083820 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:44:51.135304 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01112398-c29a-4df3-8bfc-51d162b87dd7 None None] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:44:51.170048 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a1be427-279b-486e-ad81-6a90a3f39b2e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:44:51.170715 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a1be427-279b-486e-ad81-6a90a3f39b2e None admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:44:51.170999 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a1be427-279b-486e-ad81-6a90a3f39b2e None admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:44:51.171277 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a1be427-279b-486e-ad81-6a90a3f39b2e None admin] PATH_INFO: `/v3/roles` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:44:51.172024 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a1be427-279b-486e-ad81-6a90a3f39b2e None admin] RBAC: Adding query filter params (name=service) {{(pid=92056) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} May 23 12:44:51.172335 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a1be427-279b-486e-ad81-6a90a3f39b2e None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:44:51.173777 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a1be427-279b-486e-ad81-6a90a3f39b2e None admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:44:51.182201 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 617/1236] 10.222.0.30 () {56 vars in 1103 bytes} [Mon May 23 12:44:51 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 145 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) May 23 12:44:51.188078 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7f003e1-068e-48d1-9d51-ea3d7bb08907 None admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:44:51.272881 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=100 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:44:51.288717 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7f003e1-068e-48d1-9d51-ea3d7bb08907 None admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:44:51.321719 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-628277a1-94ea-407b-b30e-65d53c04212a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:44:51.322532 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-628277a1-94ea-407b-b30e-65d53c04212a None admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:44:51.322867 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-628277a1-94ea-407b-b30e-65d53c04212a None admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:44:51.323221 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-628277a1-94ea-407b-b30e-65d53c04212a None admin] PATH_INFO: `/v3/domains/Default` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:44:51.334298 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-628277a1-94ea-407b-b30e-65d53c04212a None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:44:51.335867 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-628277a1-94ea-407b-b30e-65d53c04212a None admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:44:51.353331 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application [None req-628277a1-94ea-407b-b30e-65d53c04212a None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. May 23 12:44:51.353331 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application Traceback (most recent call last): May 23 12:44:51.353331 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request May 23 12:44:51.353331 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application rv = self.dispatch_request() May 23 12:44:51.353331 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request May 23 12:44:51.353331 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) May 23 12:44:51.353331 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper May 23 12:44:51.353331 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) May 23 12:44:51.353331 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view May 23 12:44:51.353331 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) May 23 12:44:51.353331 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request May 23 12:44:51.353331 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) May 23 12:44:51.353331 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get May 23 12:44:51.353331 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self._get_domain(domain_id) May 23 12:44:51.353331 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain May 23 12:44:51.354488 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) May 23 12:44:51.354488 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped May 23 12:44:51.354488 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) May 23 12:44:51.354488 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun May 23 12:44:51.354488 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) May 23 12:44:51.354488 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func May 23 12:44:51.354488 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self.get_or_create( May 23 12:44:51.354488 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create May 23 12:44:51.354488 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application with Lock( May 23 12:44:51.354488 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ May 23 12:44:51.354488 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self._enter() May 23 12:44:51.354488 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter May 23 12:44:51.354488 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) May 23 12:44:51.354488 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create May 23 12:44:51.354488 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self.creator() May 23 12:44:51.354488 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value May 23 12:44:51.354488 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application created_value = creator( May 23 12:44:51.355580 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain May 23 12:44:51.355580 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) May 23 12:44:51.355580 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. May 23 12:44:51.355580 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application May 23 12:44:51.355580 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 620/1237] 10.222.0.30 () {56 vars in 1108 bytes} [Mon May 23 12:44:51 2022] GET /identity/v3/domains/Default => generated 87 bytes in 167 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) May 23 12:44:51.360531 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a1be427-279b-486e-ad81-6a90a3f39b2e None admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:44:51.460817 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a1be427-279b-486e-ad81-6a90a3f39b2e None admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:44:51.469840 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=100 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:44:51.493206 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc49f128-77e1-4ee1-bc03-a0c8e7a88d12 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:44:51.493855 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc49f128-77e1-4ee1-bc03-a0c8e7a88d12 None admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:44:51.494165 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc49f128-77e1-4ee1-bc03-a0c8e7a88d12 None admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:44:51.494457 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc49f128-77e1-4ee1-bc03-a0c8e7a88d12 None admin] PATH_INFO: `/v3/domains` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:44:51.495156 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc49f128-77e1-4ee1-bc03-a0c8e7a88d12 None admin] RBAC: Adding query filter params (name=Default) {{(pid=92056) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} May 23 12:44:51.495463 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc49f128-77e1-4ee1-bc03-a0c8e7a88d12 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:44:51.496828 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc49f128-77e1-4ee1-bc03-a0c8e7a88d12 None admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:44:51.508634 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 618/1238] 10.222.0.30 () {56 vars in 1109 bytes} [Mon May 23 12:44:51 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 149 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) May 23 12:44:51.514469 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-628277a1-94ea-407b-b30e-65d53c04212a None admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:44:51.619431 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-628277a1-94ea-407b-b30e-65d53c04212a None admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:44:51.654097 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc27e811-2070-4924-8f41-c7ad941c7944 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:44:51.654951 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc27e811-2070-4924-8f41-c7ad941c7944 None admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:44:51.655243 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc27e811-2070-4924-8f41-c7ad941c7944 None admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:44:51.655524 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc27e811-2070-4924-8f41-c7ad941c7944 None admin] PATH_INFO: `/v3/users/placement` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:44:51.658451 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=100 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:44:51.663538 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc27e811-2070-4924-8f41-c7ad941c7944 None admin] RBAC: Authorizing `identity:get_user(user_id=placement)` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:44:51.664996 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc27e811-2070-4924-8f41-c7ad941c7944 None admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:44:51.679476 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application [None req-fc27e811-2070-4924-8f41-c7ad941c7944 None admin] Could not find user: placement.: keystone.exception.UserNotFound: Could not find user: placement. May 23 12:44:51.679476 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application Traceback (most recent call last): May 23 12:44:51.679476 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request May 23 12:44:51.679476 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application rv = self.dispatch_request() May 23 12:44:51.679476 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request May 23 12:44:51.679476 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) May 23 12:44:51.679476 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper May 23 12:44:51.679476 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) May 23 12:44:51.679476 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view May 23 12:44:51.679476 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) May 23 12:44:51.679476 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request May 23 12:44:51.679476 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) May 23 12:44:51.679476 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get May 23 12:44:51.679476 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self._get_user(user_id) May 23 12:44:51.679476 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user May 23 12:44:51.679476 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) May 23 12:44:51.680652 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped May 23 12:44:51.680652 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) May 23 12:44:51.680652 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper May 23 12:44:51.680652 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) May 23 12:44:51.680652 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper May 23 12:44:51.680652 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) May 23 12:44:51.680652 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun May 23 12:44:51.680652 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) May 23 12:44:51.680652 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func May 23 12:44:51.680652 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self.get_or_create( May 23 12:44:51.680652 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create May 23 12:44:51.680652 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application with Lock( May 23 12:44:51.680652 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ May 23 12:44:51.680652 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self._enter() May 23 12:44:51.680652 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter May 23 12:44:51.680652 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) May 23 12:44:51.680652 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create May 23 12:44:51.680652 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self.creator() May 23 12:44:51.681832 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value May 23 12:44:51.681832 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application created_value = creator( May 23 12:44:51.681832 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user May 23 12:44:51.681832 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) May 23 12:44:51.681832 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user May 23 12:44:51.681832 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) May 23 12:44:51.681832 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user May 23 12:44:51.681832 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) May 23 12:44:51.681832 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: placement. May 23 12:44:51.681832 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application May 23 12:44:51.682501 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 621/1239] 10.222.0.30 () {56 vars in 1108 bytes} [Mon May 23 12:44:51 2022] GET /identity/v3/users/placement => generated 87 bytes in 168 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) May 23 12:44:51.686487 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc49f128-77e1-4ee1-bc03-a0c8e7a88d12 None admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:44:51.786491 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc49f128-77e1-4ee1-bc03-a0c8e7a88d12 None admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:44:51.819754 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7d1c8f7-b4ec-44bb-8b78-a4987f04d367 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:44:51.820446 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7d1c8f7-b4ec-44bb-8b78-a4987f04d367 None admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:44:51.820738 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7d1c8f7-b4ec-44bb-8b78-a4987f04d367 None admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:44:51.820988 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7d1c8f7-b4ec-44bb-8b78-a4987f04d367 None admin] PATH_INFO: `/v3/users` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:44:51.821811 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d7d1c8f7-b4ec-44bb-8b78-a4987f04d367 None admin] RBAC: Adding query filter params (domain_id=default, name=placement) {{(pid=92056) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} May 23 12:44:51.823337 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d7d1c8f7-b4ec-44bb-8b78-a4987f04d367 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:44:51.824829 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d7d1c8f7-b4ec-44bb-8b78-a4987f04d367 None admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:44:51.843112 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 619/1240] 10.222.0.30 () {56 vars in 1143 bytes} [Mon May 23 12:44:51 2022] GET /identity/v3/users?domain_id=default&name=placement => generated 378 bytes in 157 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) May 23 12:44:51.850480 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc27e811-2070-4924-8f41-c7ad941c7944 None admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:44:51.952822 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc27e811-2070-4924-8f41-c7ad941c7944 None admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:44:51.986540 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a9e8ce2-96e2-40b0-b64e-fb8189328302 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:44:51.987710 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a9e8ce2-96e2-40b0-b64e-fb8189328302 None admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:44:51.987989 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a9e8ce2-96e2-40b0-b64e-fb8189328302 None admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:44:51.988260 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a9e8ce2-96e2-40b0-b64e-fb8189328302 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:44:52.047936 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=100 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:44:52.054801 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a9e8ce2-96e2-40b0-b64e-fb8189328302 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:44:52.056251 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a9e8ce2-96e2-40b0-b64e-fb8189328302 None admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:44:52.151495 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 622/1241] 10.222.0.30 () {58 vars in 1282 bytes} [Mon May 23 12:44:51 2022] GET /identity/v3/auth/tokens => generated 2108 bytes in 302 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) May 23 12:44:52.157478 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7d1c8f7-b4ec-44bb-8b78-a4987f04d367 None admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:44:52.184592 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=100 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:44:52.255280 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7d1c8f7-b4ec-44bb-8b78-a4987f04d367 None admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:44:52.289626 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2e1392f-c4bd-4eab-aad9-d8873f18a632 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:44:52.290465 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2e1392f-c4bd-4eab-aad9-d8873f18a632 None admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:44:52.290739 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2e1392f-c4bd-4eab-aad9-d8873f18a632 None admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:44:52.291009 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2e1392f-c4bd-4eab-aad9-d8873f18a632 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:44:52.301339 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f2e1392f-c4bd-4eab-aad9-d8873f18a632 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:44:52.302808 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f2e1392f-c4bd-4eab-aad9-d8873f18a632 None admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:44:52.320267 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application [None req-f2e1392f-c4bd-4eab-aad9-d8873f18a632 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. May 23 12:44:52.320267 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application Traceback (most recent call last): May 23 12:44:52.320267 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request May 23 12:44:52.320267 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application rv = self.dispatch_request() May 23 12:44:52.320267 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request May 23 12:44:52.320267 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) May 23 12:44:52.320267 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper May 23 12:44:52.320267 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) May 23 12:44:52.320267 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view May 23 12:44:52.320267 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) May 23 12:44:52.320267 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request May 23 12:44:52.320267 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) May 23 12:44:52.320267 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get May 23 12:44:52.320267 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self._get_domain(domain_id) May 23 12:44:52.320267 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain May 23 12:44:52.321360 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) May 23 12:44:52.321360 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped May 23 12:44:52.321360 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) May 23 12:44:52.321360 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun May 23 12:44:52.321360 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) May 23 12:44:52.321360 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func May 23 12:44:52.321360 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self.get_or_create( May 23 12:44:52.321360 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create May 23 12:44:52.321360 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application with Lock( May 23 12:44:52.321360 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ May 23 12:44:52.321360 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self._enter() May 23 12:44:52.321360 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter May 23 12:44:52.321360 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) May 23 12:44:52.321360 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create May 23 12:44:52.321360 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self.creator() May 23 12:44:52.321360 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value May 23 12:44:52.321360 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application created_value = creator( May 23 12:44:52.322511 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain May 23 12:44:52.322511 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) May 23 12:44:52.322511 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. May 23 12:44:52.322511 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application May 23 12:44:52.322511 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 620/1242] 10.222.0.30 () {56 vars in 1108 bytes} [Mon May 23 12:44:52 2022] GET /identity/v3/domains/Default => generated 87 bytes in 165 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) May 23 12:44:52.327142 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a9e8ce2-96e2-40b0-b64e-fb8189328302 None admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:44:52.419429 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=100 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:44:52.425959 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a9e8ce2-96e2-40b0-b64e-fb8189328302 None admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:44:52.458178 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b95459d-c9e6-4e4d-b51e-2244462db4ca None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:44:52.458825 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b95459d-c9e6-4e4d-b51e-2244462db4ca None admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:44:52.459105 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b95459d-c9e6-4e4d-b51e-2244462db4ca None admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:44:52.459380 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b95459d-c9e6-4e4d-b51e-2244462db4ca None admin] PATH_INFO: `/v3/domains` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:44:52.460078 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b95459d-c9e6-4e4d-b51e-2244462db4ca None admin] RBAC: Adding query filter params (name=Default) {{(pid=92057) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} May 23 12:44:52.460381 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b95459d-c9e6-4e4d-b51e-2244462db4ca None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:44:52.461837 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b95459d-c9e6-4e4d-b51e-2244462db4ca None admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:44:52.473180 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 623/1243] 10.222.0.30 () {56 vars in 1109 bytes} [Mon May 23 12:44:52 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 147 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) May 23 12:44:52.478859 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2e1392f-c4bd-4eab-aad9-d8873f18a632 None admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:44:52.557112 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=100 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:44:52.577236 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2e1392f-c4bd-4eab-aad9-d8873f18a632 None admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:44:52.611056 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f715335f-3b87-4539-8f57-7420526382d8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:44:52.611845 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f715335f-3b87-4539-8f57-7420526382d8 None admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:44:52.612157 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f715335f-3b87-4539-8f57-7420526382d8 None admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:44:52.612423 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f715335f-3b87-4539-8f57-7420526382d8 None admin] PATH_INFO: `/v3/projects/service` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:44:52.618945 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f715335f-3b87-4539-8f57-7420526382d8 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:44:52.620394 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f715335f-3b87-4539-8f57-7420526382d8 None admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:44:52.634493 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application [None req-f715335f-3b87-4539-8f57-7420526382d8 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. May 23 12:44:52.634493 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application Traceback (most recent call last): May 23 12:44:52.634493 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request May 23 12:44:52.634493 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application rv = self.dispatch_request() May 23 12:44:52.634493 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request May 23 12:44:52.634493 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) May 23 12:44:52.634493 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper May 23 12:44:52.634493 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) May 23 12:44:52.634493 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view May 23 12:44:52.634493 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) May 23 12:44:52.634493 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request May 23 12:44:52.634493 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) May 23 12:44:52.634493 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get May 23 12:44:52.634493 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self._get_project(project_id) May 23 12:44:52.634493 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project May 23 12:44:52.638410 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) May 23 12:44:52.638410 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped May 23 12:44:52.638410 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) May 23 12:44:52.638410 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun May 23 12:44:52.638410 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) May 23 12:44:52.638410 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func May 23 12:44:52.638410 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self.get_or_create( May 23 12:44:52.638410 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create May 23 12:44:52.638410 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application with Lock( May 23 12:44:52.638410 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ May 23 12:44:52.638410 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self._enter() May 23 12:44:52.638410 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter May 23 12:44:52.638410 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) May 23 12:44:52.638410 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create May 23 12:44:52.638410 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self.creator() May 23 12:44:52.638410 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value May 23 12:44:52.638410 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application created_value = creator( May 23 12:44:52.639640 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project May 23 12:44:52.639640 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) May 23 12:44:52.639640 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project May 23 12:44:52.639640 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() May 23 12:44:52.639640 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project May 23 12:44:52.639640 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) May 23 12:44:52.639640 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. May 23 12:44:52.639640 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application May 23 12:44:52.639640 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 621/1244] 10.222.0.30 () {56 vars in 1111 bytes} [Mon May 23 12:44:52 2022] GET /identity/v3/projects/service => generated 88 bytes in 158 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) May 23 12:44:52.641985 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b95459d-c9e6-4e4d-b51e-2244462db4ca None admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:44:52.743365 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b95459d-c9e6-4e4d-b51e-2244462db4ca None admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:44:52.777579 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8399d72f-cb23-4c72-bf36-5e665ab15b4f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:44:52.778266 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8399d72f-cb23-4c72-bf36-5e665ab15b4f None admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:44:52.778611 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8399d72f-cb23-4c72-bf36-5e665ab15b4f None admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:44:52.778892 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8399d72f-cb23-4c72-bf36-5e665ab15b4f None admin] PATH_INFO: `/v3/projects` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:44:52.779672 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8399d72f-cb23-4c72-bf36-5e665ab15b4f None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=92057) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} May 23 12:44:52.779988 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8399d72f-cb23-4c72-bf36-5e665ab15b4f None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:44:52.781539 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8399d72f-cb23-4c72-bf36-5e665ab15b4f None admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:44:52.794185 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 624/1245] 10.222.0.30 () {56 vars in 1148 bytes} [Mon May 23 12:44:52 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 153 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) May 23 12:44:52.800850 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f715335f-3b87-4539-8f57-7420526382d8 None admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:44:52.908221 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f715335f-3b87-4539-8f57-7420526382d8 None admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:44:52.951990 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e287f4e-372b-4473-b919-bba9b12207f2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:44:52.952648 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e287f4e-372b-4473-b919-bba9b12207f2 None admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:44:52.952984 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e287f4e-372b-4473-b919-bba9b12207f2 None admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:44:52.953262 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e287f4e-372b-4473-b919-bba9b12207f2 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:44:52.954162 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e287f4e-372b-4473-b919-bba9b12207f2 None admin] RBAC: Adding query filter params (role.id=41ab28c1ae3d4941bd09fb5d35f1dc54, scope.project.id=9c389ae65aaa4542bb5a5c4fd0bacf5b, user.id=d5fb51f8d0fa46439cb29666970b885b) {{(pid=92056) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} May 23 12:44:52.954477 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e287f4e-372b-4473-b919-bba9b12207f2 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:44:52.956727 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e287f4e-372b-4473-b919-bba9b12207f2 None admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:44:52.964810 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 622/1246] 10.222.0.30 () {56 vars in 1374 bytes} [Mon May 23 12:44:52 2022] GET /identity/v3/role_assignments?user.id=d5fb51f8d0fa46439cb29666970b885b&scope.project.id=9c389ae65aaa4542bb5a5c4fd0bacf5b&role.id=41ab28c1ae3d4941bd09fb5d35f1dc54 => generated 608 bytes in 165 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) May 23 12:44:55.192527 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0772af7d-e843-464c-8f3a-f852f25c2bfa None None] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:44:55.193210 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0772af7d-e843-464c-8f3a-f852f25c2bfa None None] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:44:55.193210 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0772af7d-e843-464c-8f3a-f852f25c2bfa None None] PATH_INFO: `/` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:44:55.194523 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 625/1247] 10.222.0.30 () {54 vars in 926 bytes} [Mon May 23 12:44:55 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) May 23 12:44:55.199596 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b5ade62-26ac-4fb0-8d24-a757ee411d2a None None] REQUEST_METHOD: `POST` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:44:55.200020 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b5ade62-26ac-4fb0-8d24-a757ee411d2a None None] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:44:55.200160 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b5ade62-26ac-4fb0-8d24-a757ee411d2a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:44:55.205256 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=100 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:44:55.218095 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.auth.core [None req-7b5ade62-26ac-4fb0-8d24-a757ee411d2a None None] MFA Rules not processed for user `0f7119f8fa5c4445977671140ded91a4`. Rule list: `[]` (Enabled: `True`). {{(pid=92056) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 12:44:55.219326 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.fernet_utils [None req-7b5ade62-26ac-4fb0-8d24-a757ee411d2a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92056) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 12:44:55.285702 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 623/1248] 10.222.0.30 () {58 vars in 1023 bytes} [Mon May 23 12:44:55 2022] POST /identity/v3/auth/tokens => generated 2108 bytes in 87 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) May 23 12:44:55.308674 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0772af7d-e843-464c-8f3a-f852f25c2bfa None None] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:44:55.326918 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=100 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:44:55.409110 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0772af7d-e843-464c-8f3a-f852f25c2bfa None None] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:44:55.442497 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-473c39bb-fe3f-45e6-8682-40dbf20b6958 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:44:55.443326 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-473c39bb-fe3f-45e6-8682-40dbf20b6958 None admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:44:55.443582 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-473c39bb-fe3f-45e6-8682-40dbf20b6958 None admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:44:55.443859 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-473c39bb-fe3f-45e6-8682-40dbf20b6958 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:44:55.453668 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-473c39bb-fe3f-45e6-8682-40dbf20b6958 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} May 23 12:44:55.462091 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application [None req-473c39bb-fe3f-45e6-8682-40dbf20b6958 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. May 23 12:44:55.462091 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application Traceback (most recent call last): May 23 12:44:55.462091 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request May 23 12:44:55.462091 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application rv = self.dispatch_request() May 23 12:44:55.462091 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request May 23 12:44:55.462091 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) May 23 12:44:55.462091 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper May 23 12:44:55.462091 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) May 23 12:44:55.462091 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view May 23 12:44:55.462091 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) May 23 12:44:55.462091 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request May 23 12:44:55.462091 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) May 23 12:44:55.462091 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get May 23 12:44:55.462091 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self._get_role(role_id) May 23 12:44:55.462091 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role May 23 12:44:55.462091 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') May 23 12:44:55.463235 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call May 23 12:44:55.463235 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( May 23 12:44:55.463235 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data May 23 12:44:55.463235 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) May 23 12:44:55.463235 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped May 23 12:44:55.463235 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) May 23 12:44:55.463235 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun May 23 12:44:55.463235 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) May 23 12:44:55.463235 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func May 23 12:44:55.463235 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self.get_or_create( May 23 12:44:55.463235 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create May 23 12:44:55.463235 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application with Lock( May 23 12:44:55.463235 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ May 23 12:44:55.463235 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self._enter() May 23 12:44:55.463235 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter May 23 12:44:55.463235 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) May 23 12:44:55.464464 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create May 23 12:44:55.464464 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self.creator() May 23 12:44:55.464464 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value May 23 12:44:55.464464 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application created_value = creator( May 23 12:44:55.464464 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role May 23 12:44:55.464464 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) May 23 12:44:55.464464 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role May 23 12:44:55.464464 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() May 23 12:44:55.464464 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role May 23 12:44:55.464464 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) May 23 12:44:55.464464 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. May 23 12:44:55.464464 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application May 23 12:44:55.464464 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 626/1249] 10.222.0.30 () {56 vars in 1096 bytes} [Mon May 23 12:44:55 2022] GET /identity/v3/roles/admin => generated 83 bytes in 155 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) May 23 12:44:55.469220 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b5ade62-26ac-4fb0-8d24-a757ee411d2a None None] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:44:55.570895 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b5ade62-26ac-4fb0-8d24-a757ee411d2a None None] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:44:55.601499 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=100 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:44:55.606545 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80a3bde6-b131-48fe-b2ce-b1d74c48bad8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:44:55.607210 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80a3bde6-b131-48fe-b2ce-b1d74c48bad8 None admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:44:55.607493 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80a3bde6-b131-48fe-b2ce-b1d74c48bad8 None admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:44:55.607769 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80a3bde6-b131-48fe-b2ce-b1d74c48bad8 None admin] PATH_INFO: `/v3/roles` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:44:55.608511 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80a3bde6-b131-48fe-b2ce-b1d74c48bad8 None admin] RBAC: Adding query filter params (name=admin) {{(pid=92056) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} May 23 12:44:55.609061 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80a3bde6-b131-48fe-b2ce-b1d74c48bad8 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:44:55.610489 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80a3bde6-b131-48fe-b2ce-b1d74c48bad8 None admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:44:55.619800 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 624/1250] 10.222.0.30 () {56 vars in 1099 bytes} [Mon May 23 12:44:55 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 151 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) May 23 12:44:55.625645 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-473c39bb-fe3f-45e6-8682-40dbf20b6958 None admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:44:55.708199 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=100 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:44:55.730546 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-473c39bb-fe3f-45e6-8682-40dbf20b6958 None admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:44:55.764563 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-268a6850-ebdf-4f56-8e48-ff887bdf0225 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:44:55.765345 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-268a6850-ebdf-4f56-8e48-ff887bdf0225 None admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:44:55.765630 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-268a6850-ebdf-4f56-8e48-ff887bdf0225 None admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:44:55.765927 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-268a6850-ebdf-4f56-8e48-ff887bdf0225 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:44:55.776936 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-268a6850-ebdf-4f56-8e48-ff887bdf0225 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:44:55.778424 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-268a6850-ebdf-4f56-8e48-ff887bdf0225 None admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:44:55.795530 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application [None req-268a6850-ebdf-4f56-8e48-ff887bdf0225 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. May 23 12:44:55.795530 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application Traceback (most recent call last): May 23 12:44:55.795530 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request May 23 12:44:55.795530 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application rv = self.dispatch_request() May 23 12:44:55.795530 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request May 23 12:44:55.795530 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) May 23 12:44:55.795530 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper May 23 12:44:55.795530 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) May 23 12:44:55.795530 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view May 23 12:44:55.795530 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) May 23 12:44:55.795530 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request May 23 12:44:55.795530 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) May 23 12:44:55.795530 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get May 23 12:44:55.795530 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self._get_domain(domain_id) May 23 12:44:55.795530 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain May 23 12:44:55.796580 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) May 23 12:44:55.796580 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped May 23 12:44:55.796580 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) May 23 12:44:55.796580 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun May 23 12:44:55.796580 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) May 23 12:44:55.796580 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func May 23 12:44:55.796580 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self.get_or_create( May 23 12:44:55.796580 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create May 23 12:44:55.796580 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application with Lock( May 23 12:44:55.796580 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ May 23 12:44:55.796580 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self._enter() May 23 12:44:55.796580 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter May 23 12:44:55.796580 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) May 23 12:44:55.796580 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create May 23 12:44:55.796580 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self.creator() May 23 12:44:55.796580 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value May 23 12:44:55.796580 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application created_value = creator( May 23 12:44:55.797614 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain May 23 12:44:55.797614 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) May 23 12:44:55.797614 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. May 23 12:44:55.797614 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application May 23 12:44:55.797614 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 627/1251] 10.222.0.30 () {56 vars in 1108 bytes} [Mon May 23 12:44:55 2022] GET /identity/v3/domains/Default => generated 87 bytes in 172 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) May 23 12:44:55.803673 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80a3bde6-b131-48fe-b2ce-b1d74c48bad8 None admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:44:55.907013 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80a3bde6-b131-48fe-b2ce-b1d74c48bad8 None admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:44:55.947904 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79ce6e3b-482e-4a7f-868f-c44e501ab0a9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:44:55.948523 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79ce6e3b-482e-4a7f-868f-c44e501ab0a9 None admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:44:55.948806 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79ce6e3b-482e-4a7f-868f-c44e501ab0a9 None admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:44:55.949080 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79ce6e3b-482e-4a7f-868f-c44e501ab0a9 None admin] PATH_INFO: `/v3/domains` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:44:55.949801 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79ce6e3b-482e-4a7f-868f-c44e501ab0a9 None admin] RBAC: Adding query filter params (name=Default) {{(pid=92056) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} May 23 12:44:55.950140 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79ce6e3b-482e-4a7f-868f-c44e501ab0a9 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:44:55.951523 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79ce6e3b-482e-4a7f-868f-c44e501ab0a9 None admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:44:55.963862 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 625/1252] 10.222.0.30 () {56 vars in 1109 bytes} [Mon May 23 12:44:55 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 161 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) May 23 12:44:55.969492 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-268a6850-ebdf-4f56-8e48-ff887bdf0225 None admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:44:56.068030 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-268a6850-ebdf-4f56-8e48-ff887bdf0225 None admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:44:56.093389 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=100 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:44:56.101829 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81686a81-3fff-4ddf-bc67-e5d78d610c97 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:44:56.102646 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81686a81-3fff-4ddf-bc67-e5d78d610c97 None admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:44:56.102933 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81686a81-3fff-4ddf-bc67-e5d78d610c97 None admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:44:56.103209 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81686a81-3fff-4ddf-bc67-e5d78d610c97 None admin] PATH_INFO: `/v3/users/placement` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:44:56.110448 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81686a81-3fff-4ddf-bc67-e5d78d610c97 None admin] RBAC: Authorizing `identity:get_user(user_id=placement)` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:44:56.111923 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81686a81-3fff-4ddf-bc67-e5d78d610c97 None admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:44:56.126864 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application [None req-81686a81-3fff-4ddf-bc67-e5d78d610c97 None admin] Could not find user: placement.: keystone.exception.UserNotFound: Could not find user: placement. May 23 12:44:56.126864 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application Traceback (most recent call last): May 23 12:44:56.126864 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request May 23 12:44:56.126864 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application rv = self.dispatch_request() May 23 12:44:56.126864 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request May 23 12:44:56.126864 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) May 23 12:44:56.126864 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper May 23 12:44:56.126864 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) May 23 12:44:56.126864 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view May 23 12:44:56.126864 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) May 23 12:44:56.126864 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request May 23 12:44:56.126864 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) May 23 12:44:56.126864 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get May 23 12:44:56.126864 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self._get_user(user_id) May 23 12:44:56.126864 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user May 23 12:44:56.126864 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) May 23 12:44:56.128017 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped May 23 12:44:56.128017 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) May 23 12:44:56.128017 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper May 23 12:44:56.128017 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) May 23 12:44:56.128017 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper May 23 12:44:56.128017 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) May 23 12:44:56.128017 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun May 23 12:44:56.128017 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) May 23 12:44:56.128017 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func May 23 12:44:56.128017 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self.get_or_create( May 23 12:44:56.128017 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create May 23 12:44:56.128017 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application with Lock( May 23 12:44:56.128017 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ May 23 12:44:56.128017 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self._enter() May 23 12:44:56.128017 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter May 23 12:44:56.128017 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) May 23 12:44:56.128017 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create May 23 12:44:56.128017 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self.creator() May 23 12:44:56.129230 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value May 23 12:44:56.129230 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application created_value = creator( May 23 12:44:56.129230 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user May 23 12:44:56.129230 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) May 23 12:44:56.129230 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user May 23 12:44:56.129230 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) May 23 12:44:56.129230 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user May 23 12:44:56.129230 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) May 23 12:44:56.129230 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: placement. May 23 12:44:56.129230 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application May 23 12:44:56.129230 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 628/1253] 10.222.0.30 () {56 vars in 1108 bytes} [Mon May 23 12:44:55 2022] GET /identity/v3/users/placement => generated 87 bytes in 159 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) May 23 12:44:56.134180 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79ce6e3b-482e-4a7f-868f-c44e501ab0a9 None admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:44:56.189884 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=100 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:44:56.232799 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79ce6e3b-482e-4a7f-868f-c44e501ab0a9 None admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:44:56.271649 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c2804c6-9c73-405b-bf66-1ac9ed34cf77 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:44:56.272315 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c2804c6-9c73-405b-bf66-1ac9ed34cf77 None admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:44:56.272600 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c2804c6-9c73-405b-bf66-1ac9ed34cf77 None admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:44:56.272880 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c2804c6-9c73-405b-bf66-1ac9ed34cf77 None admin] PATH_INFO: `/v3/users` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:44:56.273713 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c2804c6-9c73-405b-bf66-1ac9ed34cf77 None admin] RBAC: Adding query filter params (domain_id=default, name=placement) {{(pid=92056) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} May 23 12:44:56.274057 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c2804c6-9c73-405b-bf66-1ac9ed34cf77 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:44:56.275748 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c2804c6-9c73-405b-bf66-1ac9ed34cf77 None admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:44:56.292904 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 626/1254] 10.222.0.30 () {56 vars in 1143 bytes} [Mon May 23 12:44:56 2022] GET /identity/v3/users?domain_id=default&name=placement => generated 378 bytes in 159 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) May 23 12:44:56.298936 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81686a81-3fff-4ddf-bc67-e5d78d610c97 None admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:44:56.397965 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81686a81-3fff-4ddf-bc67-e5d78d610c97 None admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:44:56.431327 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5dbcf803-7d9a-4c9c-b4e0-59fe6f7fc2a2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:44:56.452531 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5dbcf803-7d9a-4c9c-b4e0-59fe6f7fc2a2 None admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:44:56.452531 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5dbcf803-7d9a-4c9c-b4e0-59fe6f7fc2a2 None admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:44:56.452531 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5dbcf803-7d9a-4c9c-b4e0-59fe6f7fc2a2 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:44:56.486957 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=100 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:44:56.498998 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5dbcf803-7d9a-4c9c-b4e0-59fe6f7fc2a2 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:44:56.500464 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5dbcf803-7d9a-4c9c-b4e0-59fe6f7fc2a2 None admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:44:56.596409 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 629/1255] 10.222.0.30 () {58 vars in 1282 bytes} [Mon May 23 12:44:56 2022] GET /identity/v3/auth/tokens => generated 2108 bytes in 298 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) May 23 12:44:56.603345 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c2804c6-9c73-405b-bf66-1ac9ed34cf77 None admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:44:56.705351 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c2804c6-9c73-405b-bf66-1ac9ed34cf77 None admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:44:56.720247 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=100 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:44:56.741614 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1337e95a-2765-497e-af24-7b2dfdfef42e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:44:56.742356 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1337e95a-2765-497e-af24-7b2dfdfef42e None admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:44:56.742646 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1337e95a-2765-497e-af24-7b2dfdfef42e None admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:44:56.742920 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1337e95a-2765-497e-af24-7b2dfdfef42e None admin] PATH_INFO: `/v3/domains/Default` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:44:56.752732 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1337e95a-2765-497e-af24-7b2dfdfef42e None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:44:56.755249 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1337e95a-2765-497e-af24-7b2dfdfef42e None admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:44:56.773398 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application [None req-1337e95a-2765-497e-af24-7b2dfdfef42e None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. May 23 12:44:56.773398 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application Traceback (most recent call last): May 23 12:44:56.773398 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request May 23 12:44:56.773398 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application rv = self.dispatch_request() May 23 12:44:56.773398 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request May 23 12:44:56.773398 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) May 23 12:44:56.773398 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper May 23 12:44:56.773398 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) May 23 12:44:56.773398 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view May 23 12:44:56.773398 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) May 23 12:44:56.773398 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request May 23 12:44:56.773398 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) May 23 12:44:56.773398 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get May 23 12:44:56.773398 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self._get_domain(domain_id) May 23 12:44:56.773398 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain May 23 12:44:56.774525 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) May 23 12:44:56.774525 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped May 23 12:44:56.774525 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) May 23 12:44:56.774525 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun May 23 12:44:56.774525 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) May 23 12:44:56.774525 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func May 23 12:44:56.774525 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self.get_or_create( May 23 12:44:56.774525 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create May 23 12:44:56.774525 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application with Lock( May 23 12:44:56.774525 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ May 23 12:44:56.774525 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self._enter() May 23 12:44:56.774525 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter May 23 12:44:56.774525 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) May 23 12:44:56.774525 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create May 23 12:44:56.774525 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self.creator() May 23 12:44:56.774525 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value May 23 12:44:56.774525 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application created_value = creator( May 23 12:44:56.775861 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain May 23 12:44:56.775861 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) May 23 12:44:56.775861 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. May 23 12:44:56.775861 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application May 23 12:44:56.775861 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 627/1256] 10.222.0.30 () {56 vars in 1108 bytes} [Mon May 23 12:44:56 2022] GET /identity/v3/domains/Default => generated 87 bytes in 172 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) May 23 12:44:56.780745 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5dbcf803-7d9a-4c9c-b4e0-59fe6f7fc2a2 None admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:44:56.865398 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=100 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:44:56.877245 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5dbcf803-7d9a-4c9c-b4e0-59fe6f7fc2a2 None admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:44:56.911649 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fba172a8-67e4-4cad-a0a3-2ddf4aaf9691 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:44:56.912364 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fba172a8-67e4-4cad-a0a3-2ddf4aaf9691 None admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:44:56.912641 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fba172a8-67e4-4cad-a0a3-2ddf4aaf9691 None admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:44:56.912915 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fba172a8-67e4-4cad-a0a3-2ddf4aaf9691 None admin] PATH_INFO: `/v3/domains` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:44:56.913634 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fba172a8-67e4-4cad-a0a3-2ddf4aaf9691 None admin] RBAC: Adding query filter params (name=Default) {{(pid=92057) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} May 23 12:44:56.913966 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fba172a8-67e4-4cad-a0a3-2ddf4aaf9691 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:44:56.915335 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fba172a8-67e4-4cad-a0a3-2ddf4aaf9691 None admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:44:56.926389 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 630/1257] 10.222.0.30 () {56 vars in 1109 bytes} [Mon May 23 12:44:56 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 146 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) May 23 12:44:56.932139 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1337e95a-2765-497e-af24-7b2dfdfef42e None admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:44:57.033922 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1337e95a-2765-497e-af24-7b2dfdfef42e None admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:44:57.067714 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-acbf95d2-9d66-4e75-80ef-5e0cbbddc003 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:44:57.068496 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acbf95d2-9d66-4e75-80ef-5e0cbbddc003 None admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:44:57.068782 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acbf95d2-9d66-4e75-80ef-5e0cbbddc003 None admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:44:57.069060 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acbf95d2-9d66-4e75-80ef-5e0cbbddc003 None admin] PATH_INFO: `/v3/projects/service` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:44:57.075621 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-acbf95d2-9d66-4e75-80ef-5e0cbbddc003 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:44:57.077072 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-acbf95d2-9d66-4e75-80ef-5e0cbbddc003 None admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:44:57.091602 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application [None req-acbf95d2-9d66-4e75-80ef-5e0cbbddc003 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. May 23 12:44:57.091602 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application Traceback (most recent call last): May 23 12:44:57.091602 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request May 23 12:44:57.091602 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application rv = self.dispatch_request() May 23 12:44:57.091602 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request May 23 12:44:57.091602 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) May 23 12:44:57.091602 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper May 23 12:44:57.091602 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) May 23 12:44:57.091602 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view May 23 12:44:57.091602 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) May 23 12:44:57.091602 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request May 23 12:44:57.091602 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) May 23 12:44:57.091602 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get May 23 12:44:57.091602 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self._get_project(project_id) May 23 12:44:57.091602 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project May 23 12:44:57.092677 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) May 23 12:44:57.092677 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped May 23 12:44:57.092677 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) May 23 12:44:57.092677 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun May 23 12:44:57.092677 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) May 23 12:44:57.092677 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func May 23 12:44:57.092677 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self.get_or_create( May 23 12:44:57.092677 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create May 23 12:44:57.092677 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application with Lock( May 23 12:44:57.092677 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ May 23 12:44:57.092677 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self._enter() May 23 12:44:57.092677 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter May 23 12:44:57.092677 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) May 23 12:44:57.092677 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create May 23 12:44:57.092677 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self.creator() May 23 12:44:57.092677 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value May 23 12:44:57.092677 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application created_value = creator( May 23 12:44:57.093739 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project May 23 12:44:57.093739 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) May 23 12:44:57.093739 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project May 23 12:44:57.093739 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() May 23 12:44:57.093739 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project May 23 12:44:57.093739 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) May 23 12:44:57.093739 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. May 23 12:44:57.093739 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application May 23 12:44:57.093739 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 628/1258] 10.222.0.30 () {56 vars in 1111 bytes} [Mon May 23 12:44:56 2022] GET /identity/v3/projects/service => generated 88 bytes in 161 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) May 23 12:44:57.099569 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fba172a8-67e4-4cad-a0a3-2ddf4aaf9691 None admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:44:57.197285 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fba172a8-67e4-4cad-a0a3-2ddf4aaf9691 None admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:44:57.230735 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7a0e146-af99-4b13-a0bc-514c78384eae None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:44:57.231387 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7a0e146-af99-4b13-a0bc-514c78384eae None admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:44:57.231666 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7a0e146-af99-4b13-a0bc-514c78384eae None admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:44:57.231938 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7a0e146-af99-4b13-a0bc-514c78384eae None admin] PATH_INFO: `/v3/projects` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:44:57.232709 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7a0e146-af99-4b13-a0bc-514c78384eae None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=92057) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} May 23 12:44:57.233015 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7a0e146-af99-4b13-a0bc-514c78384eae None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:44:57.234483 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7a0e146-af99-4b13-a0bc-514c78384eae None admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:44:57.247189 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 631/1259] 10.222.0.30 () {56 vars in 1148 bytes} [Mon May 23 12:44:57 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 148 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) May 23 12:44:57.253689 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-acbf95d2-9d66-4e75-80ef-5e0cbbddc003 None admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:44:57.283576 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=100 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:44:57.356997 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-acbf95d2-9d66-4e75-80ef-5e0cbbddc003 None admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:44:57.389485 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b6a6ed5-5e7d-4d1b-aeb8-a9a46db4da28 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:44:57.390165 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b6a6ed5-5e7d-4d1b-aeb8-a9a46db4da28 None admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:44:57.390456 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b6a6ed5-5e7d-4d1b-aeb8-a9a46db4da28 None admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:44:57.390737 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b6a6ed5-5e7d-4d1b-aeb8-a9a46db4da28 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:44:57.391561 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b6a6ed5-5e7d-4d1b-aeb8-a9a46db4da28 None admin] RBAC: Adding query filter params (role.id=bc3c3c9d654940cbaa5177a6be23d1bc, scope.project.id=9c389ae65aaa4542bb5a5c4fd0bacf5b, user.id=d5fb51f8d0fa46439cb29666970b885b) {{(pid=92056) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} May 23 12:44:57.391920 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b6a6ed5-5e7d-4d1b-aeb8-a9a46db4da28 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:44:57.393374 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b6a6ed5-5e7d-4d1b-aeb8-a9a46db4da28 None admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:44:57.400716 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 629/1260] 10.222.0.30 () {56 vars in 1374 bytes} [Mon May 23 12:44:57 2022] GET /identity/v3/role_assignments?user.id=d5fb51f8d0fa46439cb29666970b885b&scope.project.id=9c389ae65aaa4542bb5a5c4fd0bacf5b&role.id=bc3c3c9d654940cbaa5177a6be23d1bc => generated 259 bytes in 148 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) May 23 12:44:59.584683 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53357c14-48ac-48d0-8eee-33f93d8f2e13 None None] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:44:59.585208 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53357c14-48ac-48d0-8eee-33f93d8f2e13 None None] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:44:59.585208 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53357c14-48ac-48d0-8eee-33f93d8f2e13 None None] PATH_INFO: `/` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:44:59.586666 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 632/1261] 10.222.0.30 () {54 vars in 926 bytes} [Mon May 23 12:44:59 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) May 23 12:44:59.591766 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fd6dc62-2477-4226-818b-fe316a47ffaa None None] REQUEST_METHOD: `POST` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:44:59.592174 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fd6dc62-2477-4226-818b-fe316a47ffaa None None] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:44:59.592320 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fd6dc62-2477-4226-818b-fe316a47ffaa None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:44:59.609469 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.auth.core [None req-6fd6dc62-2477-4226-818b-fe316a47ffaa None None] MFA Rules not processed for user `0f7119f8fa5c4445977671140ded91a4`. Rule list: `[]` (Enabled: `True`). {{(pid=92056) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 12:44:59.610681 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.fernet_utils [None req-6fd6dc62-2477-4226-818b-fe316a47ffaa None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92056) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 12:44:59.688609 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 630/1262] 10.222.0.30 () {58 vars in 1023 bytes} [Mon May 23 12:44:59 2022] POST /identity/v3/auth/tokens => generated 2108 bytes in 98 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) May 23 12:44:59.712214 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53357c14-48ac-48d0-8eee-33f93d8f2e13 None None] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:44:59.724531 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=100 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:44:59.818732 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53357c14-48ac-48d0-8eee-33f93d8f2e13 None None] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:44:59.852211 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9326411-7256-4343-803f-caff866af4ef None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:44:59.852933 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9326411-7256-4343-803f-caff866af4ef None admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:44:59.853215 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9326411-7256-4343-803f-caff866af4ef None admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:44:59.853487 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9326411-7256-4343-803f-caff866af4ef None admin] PATH_INFO: `/v3/roles/admin` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:44:59.864010 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9326411-7256-4343-803f-caff866af4ef None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} May 23 12:44:59.871590 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application [None req-c9326411-7256-4343-803f-caff866af4ef None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. May 23 12:44:59.871590 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application Traceback (most recent call last): May 23 12:44:59.871590 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request May 23 12:44:59.871590 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application rv = self.dispatch_request() May 23 12:44:59.871590 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request May 23 12:44:59.871590 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) May 23 12:44:59.871590 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper May 23 12:44:59.871590 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) May 23 12:44:59.871590 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view May 23 12:44:59.871590 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) May 23 12:44:59.871590 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request May 23 12:44:59.871590 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) May 23 12:44:59.871590 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get May 23 12:44:59.871590 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self._get_role(role_id) May 23 12:44:59.871590 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role May 23 12:44:59.871590 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') May 23 12:44:59.872588 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call May 23 12:44:59.872588 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( May 23 12:44:59.872588 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data May 23 12:44:59.872588 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) May 23 12:44:59.872588 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped May 23 12:44:59.872588 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) May 23 12:44:59.872588 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun May 23 12:44:59.872588 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) May 23 12:44:59.872588 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func May 23 12:44:59.872588 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self.get_or_create( May 23 12:44:59.872588 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create May 23 12:44:59.872588 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application with Lock( May 23 12:44:59.872588 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ May 23 12:44:59.872588 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self._enter() May 23 12:44:59.872588 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter May 23 12:44:59.872588 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) May 23 12:44:59.873932 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create May 23 12:44:59.873932 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self.creator() May 23 12:44:59.873932 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value May 23 12:44:59.873932 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application created_value = creator( May 23 12:44:59.873932 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role May 23 12:44:59.873932 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) May 23 12:44:59.873932 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role May 23 12:44:59.873932 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() May 23 12:44:59.873932 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role May 23 12:44:59.873932 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) May 23 12:44:59.873932 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. May 23 12:44:59.873932 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application May 23 12:44:59.873932 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 633/1263] 10.222.0.30 () {56 vars in 1096 bytes} [Mon May 23 12:44:59 2022] GET /identity/v3/roles/admin => generated 83 bytes in 161 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) May 23 12:44:59.879041 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6fd6dc62-2477-4226-818b-fe316a47ffaa None None] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:44:59.898477 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=100 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:44:59.984897 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6fd6dc62-2477-4226-818b-fe316a47ffaa None None] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:45:00.019485 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11f860ed-ebd1-42bb-8ce1-db705ab67f96 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:45:00.020115 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11f860ed-ebd1-42bb-8ce1-db705ab67f96 None admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:45:00.020428 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11f860ed-ebd1-42bb-8ce1-db705ab67f96 None admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:45:00.020703 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11f860ed-ebd1-42bb-8ce1-db705ab67f96 None admin] PATH_INFO: `/v3/roles` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:45:00.021437 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11f860ed-ebd1-42bb-8ce1-db705ab67f96 None admin] RBAC: Adding query filter params (name=admin) {{(pid=92056) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} May 23 12:45:00.021738 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11f860ed-ebd1-42bb-8ce1-db705ab67f96 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:45:00.023129 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11f860ed-ebd1-42bb-8ce1-db705ab67f96 None admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:45:00.031167 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 631/1264] 10.222.0.30 () {56 vars in 1099 bytes} [Mon May 23 12:44:59 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 153 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) May 23 12:45:00.037101 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9326411-7256-4343-803f-caff866af4ef None admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:45:00.106542 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=100 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:45:00.139349 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9326411-7256-4343-803f-caff866af4ef None admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:45:00.171750 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60127c5a-23a7-4be9-b238-6bbe4f1eafc8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:45:00.172530 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60127c5a-23a7-4be9-b238-6bbe4f1eafc8 None admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:45:00.172814 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60127c5a-23a7-4be9-b238-6bbe4f1eafc8 None admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:45:00.173090 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60127c5a-23a7-4be9-b238-6bbe4f1eafc8 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:45:00.183758 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60127c5a-23a7-4be9-b238-6bbe4f1eafc8 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:45:00.185203 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60127c5a-23a7-4be9-b238-6bbe4f1eafc8 None admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:45:00.202204 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application [None req-60127c5a-23a7-4be9-b238-6bbe4f1eafc8 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. May 23 12:45:00.202204 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application Traceback (most recent call last): May 23 12:45:00.202204 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request May 23 12:45:00.202204 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application rv = self.dispatch_request() May 23 12:45:00.202204 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request May 23 12:45:00.202204 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) May 23 12:45:00.202204 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper May 23 12:45:00.202204 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) May 23 12:45:00.202204 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view May 23 12:45:00.202204 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) May 23 12:45:00.202204 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request May 23 12:45:00.202204 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) May 23 12:45:00.202204 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get May 23 12:45:00.202204 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self._get_domain(domain_id) May 23 12:45:00.202204 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain May 23 12:45:00.203116 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) May 23 12:45:00.203116 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped May 23 12:45:00.203116 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) May 23 12:45:00.203116 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun May 23 12:45:00.203116 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) May 23 12:45:00.203116 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func May 23 12:45:00.203116 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self.get_or_create( May 23 12:45:00.203116 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create May 23 12:45:00.203116 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application with Lock( May 23 12:45:00.203116 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ May 23 12:45:00.203116 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self._enter() May 23 12:45:00.203116 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter May 23 12:45:00.203116 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) May 23 12:45:00.203116 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create May 23 12:45:00.203116 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self.creator() May 23 12:45:00.203116 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value May 23 12:45:00.203116 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application created_value = creator( May 23 12:45:00.204076 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain May 23 12:45:00.204076 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) May 23 12:45:00.204076 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. May 23 12:45:00.204076 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application May 23 12:45:00.204076 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 634/1265] 10.222.0.30 () {56 vars in 1108 bytes} [Mon May 23 12:45:00 2022] GET /identity/v3/domains/Default => generated 87 bytes in 167 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) May 23 12:45:00.209257 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11f860ed-ebd1-42bb-8ce1-db705ab67f96 None admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:45:00.293171 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=100 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:45:00.312674 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11f860ed-ebd1-42bb-8ce1-db705ab67f96 None admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:45:00.345629 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03ee2950-0cde-4ca3-8c80-6c87b7026b7a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:45:00.346334 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03ee2950-0cde-4ca3-8c80-6c87b7026b7a None admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:45:00.346618 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03ee2950-0cde-4ca3-8c80-6c87b7026b7a None admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:45:00.347148 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03ee2950-0cde-4ca3-8c80-6c87b7026b7a None admin] PATH_INFO: `/v3/domains` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:45:00.347738 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03ee2950-0cde-4ca3-8c80-6c87b7026b7a None admin] RBAC: Adding query filter params (name=Default) {{(pid=92056) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} May 23 12:45:00.348042 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03ee2950-0cde-4ca3-8c80-6c87b7026b7a None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:45:00.349406 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03ee2950-0cde-4ca3-8c80-6c87b7026b7a None admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:45:00.360584 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 632/1266] 10.222.0.30 () {56 vars in 1109 bytes} [Mon May 23 12:45:00 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 152 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) May 23 12:45:00.366785 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60127c5a-23a7-4be9-b238-6bbe4f1eafc8 None admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:45:00.465557 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60127c5a-23a7-4be9-b238-6bbe4f1eafc8 None admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:45:00.484778 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=100 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:45:00.498587 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73e21b63-e658-4378-9c0f-b188622a6e0d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:45:00.499375 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73e21b63-e658-4378-9c0f-b188622a6e0d None admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:45:00.499659 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73e21b63-e658-4378-9c0f-b188622a6e0d None admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:45:00.499926 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73e21b63-e658-4378-9c0f-b188622a6e0d None admin] PATH_INFO: `/v3/users/placement` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:45:00.507337 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-73e21b63-e658-4378-9c0f-b188622a6e0d None admin] RBAC: Authorizing `identity:get_user(user_id=placement)` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:45:00.511905 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-73e21b63-e658-4378-9c0f-b188622a6e0d None admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:45:00.522600 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application [None req-73e21b63-e658-4378-9c0f-b188622a6e0d None admin] Could not find user: placement.: keystone.exception.UserNotFound: Could not find user: placement. May 23 12:45:00.522600 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application Traceback (most recent call last): May 23 12:45:00.522600 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request May 23 12:45:00.522600 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application rv = self.dispatch_request() May 23 12:45:00.522600 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request May 23 12:45:00.522600 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) May 23 12:45:00.522600 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper May 23 12:45:00.522600 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) May 23 12:45:00.522600 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view May 23 12:45:00.522600 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) May 23 12:45:00.522600 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request May 23 12:45:00.522600 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) May 23 12:45:00.522600 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get May 23 12:45:00.522600 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self._get_user(user_id) May 23 12:45:00.522600 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user May 23 12:45:00.522600 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) May 23 12:45:00.523742 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped May 23 12:45:00.523742 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) May 23 12:45:00.523742 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper May 23 12:45:00.523742 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) May 23 12:45:00.523742 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper May 23 12:45:00.523742 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) May 23 12:45:00.523742 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun May 23 12:45:00.523742 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) May 23 12:45:00.523742 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func May 23 12:45:00.523742 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self.get_or_create( May 23 12:45:00.523742 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create May 23 12:45:00.523742 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application with Lock( May 23 12:45:00.523742 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ May 23 12:45:00.523742 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self._enter() May 23 12:45:00.523742 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter May 23 12:45:00.523742 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) May 23 12:45:00.523742 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create May 23 12:45:00.523742 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self.creator() May 23 12:45:00.524956 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value May 23 12:45:00.524956 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application created_value = creator( May 23 12:45:00.524956 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user May 23 12:45:00.524956 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) May 23 12:45:00.524956 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user May 23 12:45:00.524956 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) May 23 12:45:00.524956 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user May 23 12:45:00.524956 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) May 23 12:45:00.524956 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: placement. May 23 12:45:00.524956 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application May 23 12:45:00.524956 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 635/1267] 10.222.0.30 () {56 vars in 1108 bytes} [Mon May 23 12:45:00 2022] GET /identity/v3/users/placement => generated 87 bytes in 157 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) May 23 12:45:00.529822 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03ee2950-0cde-4ca3-8c80-6c87b7026b7a None admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:45:00.629159 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03ee2950-0cde-4ca3-8c80-6c87b7026b7a None admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:45:00.662761 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dce1856f-451e-42d7-b42c-78fa77f14b42 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:45:00.663420 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dce1856f-451e-42d7-b42c-78fa77f14b42 None admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:45:00.663741 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dce1856f-451e-42d7-b42c-78fa77f14b42 None admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:45:00.664014 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dce1856f-451e-42d7-b42c-78fa77f14b42 None admin] PATH_INFO: `/v3/users` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:45:00.664836 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dce1856f-451e-42d7-b42c-78fa77f14b42 None admin] RBAC: Adding query filter params (domain_id=default, name=placement) {{(pid=92056) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} May 23 12:45:00.665142 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dce1856f-451e-42d7-b42c-78fa77f14b42 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:45:00.666628 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dce1856f-451e-42d7-b42c-78fa77f14b42 None admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:45:00.675475 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=100 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:45:00.683768 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 633/1268] 10.222.0.30 () {56 vars in 1143 bytes} [Mon May 23 12:45:00 2022] GET /identity/v3/users?domain_id=default&name=placement => generated 378 bytes in 154 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) May 23 12:45:00.689472 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73e21b63-e658-4378-9c0f-b188622a6e0d None admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:45:00.791967 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73e21b63-e658-4378-9c0f-b188622a6e0d None admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:45:00.825501 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef99153b-b4ea-4346-8311-21ad16546501 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:45:00.826577 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef99153b-b4ea-4346-8311-21ad16546501 None admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:45:00.826861 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef99153b-b4ea-4346-8311-21ad16546501 None admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:45:00.827147 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef99153b-b4ea-4346-8311-21ad16546501 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:45:00.839016 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef99153b-b4ea-4346-8311-21ad16546501 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:45:00.851832 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef99153b-b4ea-4346-8311-21ad16546501 None admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:45:00.857148 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application [None req-ef99153b-b4ea-4346-8311-21ad16546501 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. May 23 12:45:00.857148 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application Traceback (most recent call last): May 23 12:45:00.857148 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request May 23 12:45:00.857148 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application rv = self.dispatch_request() May 23 12:45:00.857148 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request May 23 12:45:00.857148 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) May 23 12:45:00.857148 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper May 23 12:45:00.857148 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) May 23 12:45:00.857148 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view May 23 12:45:00.857148 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) May 23 12:45:00.857148 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request May 23 12:45:00.857148 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) May 23 12:45:00.857148 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get May 23 12:45:00.857148 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self._get_domain(domain_id) May 23 12:45:00.857148 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain May 23 12:45:00.858291 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) May 23 12:45:00.858291 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped May 23 12:45:00.858291 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) May 23 12:45:00.858291 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun May 23 12:45:00.858291 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) May 23 12:45:00.858291 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func May 23 12:45:00.858291 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self.get_or_create( May 23 12:45:00.858291 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create May 23 12:45:00.858291 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application with Lock( May 23 12:45:00.858291 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ May 23 12:45:00.858291 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self._enter() May 23 12:45:00.858291 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter May 23 12:45:00.858291 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) May 23 12:45:00.858291 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create May 23 12:45:00.858291 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self.creator() May 23 12:45:00.858291 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value May 23 12:45:00.858291 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application created_value = creator( May 23 12:45:00.859426 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain May 23 12:45:00.859426 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) May 23 12:45:00.859426 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. May 23 12:45:00.859426 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application May 23 12:45:00.859426 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 636/1269] 10.222.0.30 () {56 vars in 1108 bytes} [Mon May 23 12:45:00 2022] GET /identity/v3/domains/Default => generated 87 bytes in 169 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) May 23 12:45:00.864558 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dce1856f-451e-42d7-b42c-78fa77f14b42 None admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:45:00.961219 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dce1856f-451e-42d7-b42c-78fa77f14b42 None admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:45:00.995845 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c712303-9acc-4b30-973e-d98f952a6468 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:45:00.996501 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c712303-9acc-4b30-973e-d98f952a6468 None admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:45:00.996783 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c712303-9acc-4b30-973e-d98f952a6468 None admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:45:00.997056 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c712303-9acc-4b30-973e-d98f952a6468 None admin] PATH_INFO: `/v3/domains` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:45:00.997827 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c712303-9acc-4b30-973e-d98f952a6468 None admin] RBAC: Adding query filter params (name=Default) {{(pid=92056) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} May 23 12:45:00.998189 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c712303-9acc-4b30-973e-d98f952a6468 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:45:00.999568 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c712303-9acc-4b30-973e-d98f952a6468 None admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:45:01.011389 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 634/1270] 10.222.0.30 () {56 vars in 1109 bytes} [Mon May 23 12:45:00 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 147 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) May 23 12:45:01.017151 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef99153b-b4ea-4346-8311-21ad16546501 None admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:45:01.048862 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=100 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:45:01.118059 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef99153b-b4ea-4346-8311-21ad16546501 None admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:45:01.151470 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6acd0c6a-b5a0-4143-880c-eebf1b9d95b3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:45:01.152250 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6acd0c6a-b5a0-4143-880c-eebf1b9d95b3 None admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:45:01.152536 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6acd0c6a-b5a0-4143-880c-eebf1b9d95b3 None admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:45:01.152815 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6acd0c6a-b5a0-4143-880c-eebf1b9d95b3 None admin] PATH_INFO: `/v3/projects/service` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:45:01.159673 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6acd0c6a-b5a0-4143-880c-eebf1b9d95b3 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:45:01.161095 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6acd0c6a-b5a0-4143-880c-eebf1b9d95b3 None admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:45:01.175767 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application [None req-6acd0c6a-b5a0-4143-880c-eebf1b9d95b3 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. May 23 12:45:01.175767 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application Traceback (most recent call last): May 23 12:45:01.175767 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request May 23 12:45:01.175767 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application rv = self.dispatch_request() May 23 12:45:01.175767 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request May 23 12:45:01.175767 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) May 23 12:45:01.175767 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper May 23 12:45:01.175767 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) May 23 12:45:01.175767 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view May 23 12:45:01.175767 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) May 23 12:45:01.175767 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request May 23 12:45:01.175767 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) May 23 12:45:01.175767 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get May 23 12:45:01.175767 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self._get_project(project_id) May 23 12:45:01.175767 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project May 23 12:45:01.176673 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) May 23 12:45:01.176673 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped May 23 12:45:01.176673 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) May 23 12:45:01.176673 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun May 23 12:45:01.176673 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) May 23 12:45:01.176673 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func May 23 12:45:01.176673 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self.get_or_create( May 23 12:45:01.176673 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create May 23 12:45:01.176673 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application with Lock( May 23 12:45:01.176673 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ May 23 12:45:01.176673 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self._enter() May 23 12:45:01.176673 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter May 23 12:45:01.176673 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) May 23 12:45:01.176673 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create May 23 12:45:01.176673 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self.creator() May 23 12:45:01.176673 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value May 23 12:45:01.176673 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application created_value = creator( May 23 12:45:01.177752 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project May 23 12:45:01.177752 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) May 23 12:45:01.177752 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project May 23 12:45:01.177752 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() May 23 12:45:01.177752 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project May 23 12:45:01.177752 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) May 23 12:45:01.177752 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. May 23 12:45:01.177752 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application May 23 12:45:01.177752 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 637/1271] 10.222.0.30 () {56 vars in 1111 bytes} [Mon May 23 12:45:01 2022] GET /identity/v3/projects/service => generated 88 bytes in 160 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) May 23 12:45:01.182810 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c712303-9acc-4b30-973e-d98f952a6468 None admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:45:01.241943 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=100 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:45:01.282826 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c712303-9acc-4b30-973e-d98f952a6468 None admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:45:01.316616 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9e79aa6-3c89-42e9-ae98-88de2e9ffcd4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:45:01.317289 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9e79aa6-3c89-42e9-ae98-88de2e9ffcd4 None admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:45:01.317576 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9e79aa6-3c89-42e9-ae98-88de2e9ffcd4 None admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:45:01.317853 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9e79aa6-3c89-42e9-ae98-88de2e9ffcd4 None admin] PATH_INFO: `/v3/projects` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:45:01.318686 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9e79aa6-3c89-42e9-ae98-88de2e9ffcd4 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=92056) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} May 23 12:45:01.318999 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9e79aa6-3c89-42e9-ae98-88de2e9ffcd4 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:45:01.320493 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9e79aa6-3c89-42e9-ae98-88de2e9ffcd4 None admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:45:01.332760 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 635/1272] 10.222.0.30 () {56 vars in 1148 bytes} [Mon May 23 12:45:01 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 150 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) May 23 12:45:01.338717 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6acd0c6a-b5a0-4143-880c-eebf1b9d95b3 None admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:45:01.428243 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=100 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:45:01.436641 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6acd0c6a-b5a0-4143-880c-eebf1b9d95b3 None admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:45:01.470406 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-767b1a9a-e836-4f17-a8fc-c589a16fda66 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:45:01.471287 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-767b1a9a-e836-4f17-a8fc-c589a16fda66 None admin] REQUEST_METHOD: `PUT` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:45:01.472160 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-767b1a9a-e836-4f17-a8fc-c589a16fda66 None admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:45:01.472933 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-767b1a9a-e836-4f17-a8fc-c589a16fda66 None admin] PATH_INFO: `/v3/projects/9c389ae65aaa4542bb5a5c4fd0bacf5b/users/d5fb51f8d0fa46439cb29666970b885b/roles/bc3c3c9d654940cbaa5177a6be23d1bc` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:45:01.474927 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-767b1a9a-e836-4f17-a8fc-c589a16fda66 None admin] RBAC: Authorizing `identity:create_grant(project_id=9c389ae65aaa4542bb5a5c4fd0bacf5b, user_id=d5fb51f8d0fa46439cb29666970b885b, role_id=bc3c3c9d654940cbaa5177a6be23d1bc)` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:45:01.476350 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-767b1a9a-e836-4f17-a8fc-c589a16fda66 None admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:45:01.564807 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 638/1273] 10.222.0.30 () {58 vars in 1439 bytes} [Mon May 23 12:45:01 2022] PUT /identity/v3/projects/9c389ae65aaa4542bb5a5c4fd0bacf5b/users/d5fb51f8d0fa46439cb29666970b885b/roles/bc3c3c9d654940cbaa5177a6be23d1bc => generated 0 bytes in 221 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) May 23 12:45:03.751824 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ab93ac2-7450-4591-9cbb-96a97fc1260b None None] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:45:03.752408 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ab93ac2-7450-4591-9cbb-96a97fc1260b None None] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:45:03.752408 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ab93ac2-7450-4591-9cbb-96a97fc1260b None None] PATH_INFO: `/` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:45:03.754455 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 636/1274] 10.222.0.30 () {54 vars in 926 bytes} [Mon May 23 12:45:03 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) May 23 12:45:03.759692 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06b0a923-52f7-485f-bb65-8156bdd2dcd3 None None] REQUEST_METHOD: `POST` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:45:03.760163 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06b0a923-52f7-485f-bb65-8156bdd2dcd3 None None] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:45:03.760270 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06b0a923-52f7-485f-bb65-8156bdd2dcd3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:45:03.777295 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.auth.core [None req-06b0a923-52f7-485f-bb65-8156bdd2dcd3 None None] MFA Rules not processed for user `0f7119f8fa5c4445977671140ded91a4`. Rule list: `[]` (Enabled: `True`). {{(pid=92057) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 12:45:03.778510 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.fernet_utils [None req-06b0a923-52f7-485f-bb65-8156bdd2dcd3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92057) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 12:45:03.844144 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 639/1275] 10.222.0.30 () {58 vars in 1023 bytes} [Mon May 23 12:45:03 2022] POST /identity/v3/auth/tokens => generated 2108 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) May 23 12:45:03.868177 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ab93ac2-7450-4591-9cbb-96a97fc1260b None None] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:45:03.967476 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ab93ac2-7450-4591-9cbb-96a97fc1260b None None] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:45:03.984049 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=100 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:45:04.000858 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e159a64-21a3-46c6-a00f-6f04ad3c6417 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:45:04.007430 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e159a64-21a3-46c6-a00f-6f04ad3c6417 None admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:45:04.007430 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e159a64-21a3-46c6-a00f-6f04ad3c6417 None admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:45:04.007430 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e159a64-21a3-46c6-a00f-6f04ad3c6417 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:45:04.012345 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e159a64-21a3-46c6-a00f-6f04ad3c6417 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} May 23 12:45:04.020143 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application [None req-6e159a64-21a3-46c6-a00f-6f04ad3c6417 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. May 23 12:45:04.020143 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application Traceback (most recent call last): May 23 12:45:04.020143 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request May 23 12:45:04.020143 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application rv = self.dispatch_request() May 23 12:45:04.020143 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request May 23 12:45:04.020143 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) May 23 12:45:04.020143 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper May 23 12:45:04.020143 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) May 23 12:45:04.020143 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view May 23 12:45:04.020143 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) May 23 12:45:04.020143 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request May 23 12:45:04.020143 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) May 23 12:45:04.020143 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get May 23 12:45:04.020143 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self._get_role(role_id) May 23 12:45:04.020143 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role May 23 12:45:04.020143 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') May 23 12:45:04.021377 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call May 23 12:45:04.021377 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( May 23 12:45:04.021377 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data May 23 12:45:04.021377 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) May 23 12:45:04.021377 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped May 23 12:45:04.021377 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) May 23 12:45:04.021377 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun May 23 12:45:04.021377 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) May 23 12:45:04.021377 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func May 23 12:45:04.021377 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self.get_or_create( May 23 12:45:04.021377 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create May 23 12:45:04.021377 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application with Lock( May 23 12:45:04.021377 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ May 23 12:45:04.021377 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self._enter() May 23 12:45:04.021377 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter May 23 12:45:04.021377 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) May 23 12:45:04.022536 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create May 23 12:45:04.022536 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self.creator() May 23 12:45:04.022536 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value May 23 12:45:04.022536 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application created_value = creator( May 23 12:45:04.022536 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role May 23 12:45:04.022536 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) May 23 12:45:04.022536 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role May 23 12:45:04.022536 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() May 23 12:45:04.022536 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role May 23 12:45:04.022536 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) May 23 12:45:04.022536 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. May 23 12:45:04.022536 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application May 23 12:45:04.022536 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 637/1276] 10.222.0.30 () {56 vars in 1096 bytes} [Mon May 23 12:45:03 2022] GET /identity/v3/roles/admin => generated 83 bytes in 154 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) May 23 12:45:04.027554 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06b0a923-52f7-485f-bb65-8156bdd2dcd3 None None] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:45:04.106891 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:45:04.124655 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06b0a923-52f7-485f-bb65-8156bdd2dcd3 None None] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:45:04.156979 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0cb34f1-dea1-4f9b-99a7-2c9ddabe1143 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:45:04.157694 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0cb34f1-dea1-4f9b-99a7-2c9ddabe1143 None admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:45:04.158013 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0cb34f1-dea1-4f9b-99a7-2c9ddabe1143 None admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:45:04.158290 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0cb34f1-dea1-4f9b-99a7-2c9ddabe1143 None admin] PATH_INFO: `/v3/roles` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:45:04.159020 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0cb34f1-dea1-4f9b-99a7-2c9ddabe1143 None admin] RBAC: Adding query filter params (name=admin) {{(pid=92057) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} May 23 12:45:04.159561 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0cb34f1-dea1-4f9b-99a7-2c9ddabe1143 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:45:04.161055 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0cb34f1-dea1-4f9b-99a7-2c9ddabe1143 None admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:45:04.168709 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 640/1277] 10.222.0.30 () {56 vars in 1099 bytes} [Mon May 23 12:45:04 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 142 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) May 23 12:45:04.174446 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e159a64-21a3-46c6-a00f-6f04ad3c6417 None admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:45:04.278628 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e159a64-21a3-46c6-a00f-6f04ad3c6417 None admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:45:04.312392 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0de8465e-ff98-4e79-b173-29f9500d4799 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:45:04.313172 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0de8465e-ff98-4e79-b173-29f9500d4799 None admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:45:04.313576 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0de8465e-ff98-4e79-b173-29f9500d4799 None admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:45:04.313855 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0de8465e-ff98-4e79-b173-29f9500d4799 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:45:04.324191 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0de8465e-ff98-4e79-b173-29f9500d4799 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:45:04.325634 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0de8465e-ff98-4e79-b173-29f9500d4799 None admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:45:04.342455 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application [None req-0de8465e-ff98-4e79-b173-29f9500d4799 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. May 23 12:45:04.342455 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application Traceback (most recent call last): May 23 12:45:04.342455 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request May 23 12:45:04.342455 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application rv = self.dispatch_request() May 23 12:45:04.342455 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request May 23 12:45:04.342455 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) May 23 12:45:04.342455 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper May 23 12:45:04.342455 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) May 23 12:45:04.342455 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view May 23 12:45:04.342455 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) May 23 12:45:04.342455 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request May 23 12:45:04.342455 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) May 23 12:45:04.342455 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get May 23 12:45:04.342455 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self._get_domain(domain_id) May 23 12:45:04.342455 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain May 23 12:45:04.343574 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) May 23 12:45:04.343574 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped May 23 12:45:04.343574 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) May 23 12:45:04.343574 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun May 23 12:45:04.343574 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) May 23 12:45:04.343574 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func May 23 12:45:04.343574 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self.get_or_create( May 23 12:45:04.343574 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create May 23 12:45:04.343574 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application with Lock( May 23 12:45:04.343574 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ May 23 12:45:04.343574 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self._enter() May 23 12:45:04.343574 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter May 23 12:45:04.343574 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) May 23 12:45:04.343574 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create May 23 12:45:04.343574 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self.creator() May 23 12:45:04.343574 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value May 23 12:45:04.343574 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application created_value = creator( May 23 12:45:04.344712 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain May 23 12:45:04.344712 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) May 23 12:45:04.344712 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. May 23 12:45:04.344712 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application May 23 12:45:04.344712 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 638/1278] 10.222.0.30 () {56 vars in 1108 bytes} [Mon May 23 12:45:04 2022] GET /identity/v3/domains/Default => generated 87 bytes in 170 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) May 23 12:45:04.350326 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0cb34f1-dea1-4f9b-99a7-2c9ddabe1143 None admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:45:04.451847 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0cb34f1-dea1-4f9b-99a7-2c9ddabe1143 None admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:45:04.484409 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5381baf-322b-454a-a7cd-895baaaede8e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:45:04.485072 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5381baf-322b-454a-a7cd-895baaaede8e None admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:45:04.485359 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5381baf-322b-454a-a7cd-895baaaede8e None admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:45:04.485640 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5381baf-322b-454a-a7cd-895baaaede8e None admin] PATH_INFO: `/v3/domains` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:45:04.486388 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e5381baf-322b-454a-a7cd-895baaaede8e None admin] RBAC: Adding query filter params (name=Default) {{(pid=92057) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} May 23 12:45:04.486699 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e5381baf-322b-454a-a7cd-895baaaede8e None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:45:04.488638 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e5381baf-322b-454a-a7cd-895baaaede8e None admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:45:04.500082 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 641/1279] 10.222.0.30 () {56 vars in 1109 bytes} [Mon May 23 12:45:04 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 150 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) May 23 12:45:04.506200 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0de8465e-ff98-4e79-b173-29f9500d4799 None admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:45:04.537283 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=100 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:45:04.605575 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0de8465e-ff98-4e79-b173-29f9500d4799 None admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:45:04.639052 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-112518b9-35a6-4c75-bddd-e0edcfa0b530 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:45:04.639831 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-112518b9-35a6-4c75-bddd-e0edcfa0b530 None admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:45:04.640109 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-112518b9-35a6-4c75-bddd-e0edcfa0b530 None admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:45:04.640380 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-112518b9-35a6-4c75-bddd-e0edcfa0b530 None admin] PATH_INFO: `/v3/users/placement` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:45:04.648446 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-112518b9-35a6-4c75-bddd-e0edcfa0b530 None admin] RBAC: Authorizing `identity:get_user(user_id=placement)` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:45:04.649935 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-112518b9-35a6-4c75-bddd-e0edcfa0b530 None admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:45:04.664641 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application [None req-112518b9-35a6-4c75-bddd-e0edcfa0b530 None admin] Could not find user: placement.: keystone.exception.UserNotFound: Could not find user: placement. May 23 12:45:04.664641 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application Traceback (most recent call last): May 23 12:45:04.664641 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request May 23 12:45:04.664641 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application rv = self.dispatch_request() May 23 12:45:04.664641 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request May 23 12:45:04.664641 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) May 23 12:45:04.664641 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper May 23 12:45:04.664641 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) May 23 12:45:04.664641 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view May 23 12:45:04.664641 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) May 23 12:45:04.664641 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request May 23 12:45:04.664641 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) May 23 12:45:04.664641 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get May 23 12:45:04.664641 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self._get_user(user_id) May 23 12:45:04.664641 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user May 23 12:45:04.664641 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) May 23 12:45:04.667356 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped May 23 12:45:04.667356 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) May 23 12:45:04.667356 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper May 23 12:45:04.667356 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) May 23 12:45:04.667356 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper May 23 12:45:04.667356 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) May 23 12:45:04.667356 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun May 23 12:45:04.667356 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) May 23 12:45:04.667356 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func May 23 12:45:04.667356 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self.get_or_create( May 23 12:45:04.667356 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create May 23 12:45:04.667356 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application with Lock( May 23 12:45:04.667356 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ May 23 12:45:04.667356 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self._enter() May 23 12:45:04.667356 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter May 23 12:45:04.667356 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) May 23 12:45:04.667356 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create May 23 12:45:04.667356 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self.creator() May 23 12:45:04.668412 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value May 23 12:45:04.668412 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application created_value = creator( May 23 12:45:04.668412 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user May 23 12:45:04.668412 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) May 23 12:45:04.668412 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user May 23 12:45:04.668412 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) May 23 12:45:04.668412 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user May 23 12:45:04.668412 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) May 23 12:45:04.668412 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: placement. May 23 12:45:04.668412 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application May 23 12:45:04.668412 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 639/1280] 10.222.0.30 () {56 vars in 1108 bytes} [Mon May 23 12:45:04 2022] GET /identity/v3/users/placement => generated 87 bytes in 160 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) May 23 12:45:04.671574 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5381baf-322b-454a-a7cd-895baaaede8e None admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:45:04.675489 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=100 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:45:04.768429 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5381baf-322b-454a-a7cd-895baaaede8e None admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:45:04.804161 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1f7ea38-f76b-4c56-90d5-13d68b1460a5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:45:04.804749 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1f7ea38-f76b-4c56-90d5-13d68b1460a5 None admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:45:04.805028 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1f7ea38-f76b-4c56-90d5-13d68b1460a5 None admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:45:04.805296 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1f7ea38-f76b-4c56-90d5-13d68b1460a5 None admin] PATH_INFO: `/v3/users` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:45:04.806140 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1f7ea38-f76b-4c56-90d5-13d68b1460a5 None admin] RBAC: Adding query filter params (domain_id=default, name=placement) {{(pid=92057) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} May 23 12:45:04.806447 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1f7ea38-f76b-4c56-90d5-13d68b1460a5 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:45:04.807876 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1f7ea38-f76b-4c56-90d5-13d68b1460a5 None admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:45:04.824914 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 642/1281] 10.222.0.30 () {56 vars in 1143 bytes} [Mon May 23 12:45:04 2022] GET /identity/v3/users?domain_id=default&name=placement => generated 378 bytes in 154 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) May 23 12:45:04.830708 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-112518b9-35a6-4c75-bddd-e0edcfa0b530 None admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:45:04.922655 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=100 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:45:04.933090 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-112518b9-35a6-4c75-bddd-e0edcfa0b530 None admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:45:04.967540 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a8e98c8-c1e2-4289-a661-d1fbd4dd7b56 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:45:04.968179 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a8e98c8-c1e2-4289-a661-d1fbd4dd7b56 None admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:45:04.968458 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a8e98c8-c1e2-4289-a661-d1fbd4dd7b56 None admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:45:04.968729 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a8e98c8-c1e2-4289-a661-d1fbd4dd7b56 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:45:05.035943 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a8e98c8-c1e2-4289-a661-d1fbd4dd7b56 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:45:05.037380 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a8e98c8-c1e2-4289-a661-d1fbd4dd7b56 None admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:45:05.125080 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=100 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:45:05.134905 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 640/1282] 10.222.0.30 () {58 vars in 1282 bytes} [Mon May 23 12:45:04 2022] GET /identity/v3/auth/tokens => generated 2108 bytes in 305 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) May 23 12:45:05.140850 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1f7ea38-f76b-4c56-90d5-13d68b1460a5 None admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:45:05.200730 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=100 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:45:05.239857 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1f7ea38-f76b-4c56-90d5-13d68b1460a5 None admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:45:05.274014 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b54058a9-6119-4602-a1fd-555f346e5c39 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:45:05.274798 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b54058a9-6119-4602-a1fd-555f346e5c39 None admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:45:05.275081 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b54058a9-6119-4602-a1fd-555f346e5c39 None admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:45:05.275351 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b54058a9-6119-4602-a1fd-555f346e5c39 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:45:05.285965 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b54058a9-6119-4602-a1fd-555f346e5c39 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:45:05.287434 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b54058a9-6119-4602-a1fd-555f346e5c39 None admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:45:05.305148 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application [None req-b54058a9-6119-4602-a1fd-555f346e5c39 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. May 23 12:45:05.305148 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application Traceback (most recent call last): May 23 12:45:05.305148 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request May 23 12:45:05.305148 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application rv = self.dispatch_request() May 23 12:45:05.305148 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request May 23 12:45:05.305148 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) May 23 12:45:05.305148 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper May 23 12:45:05.305148 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) May 23 12:45:05.305148 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view May 23 12:45:05.305148 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) May 23 12:45:05.305148 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request May 23 12:45:05.305148 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) May 23 12:45:05.305148 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get May 23 12:45:05.305148 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self._get_domain(domain_id) May 23 12:45:05.305148 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain May 23 12:45:05.306354 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) May 23 12:45:05.306354 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped May 23 12:45:05.306354 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) May 23 12:45:05.306354 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun May 23 12:45:05.306354 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) May 23 12:45:05.306354 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func May 23 12:45:05.306354 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self.get_or_create( May 23 12:45:05.306354 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create May 23 12:45:05.306354 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application with Lock( May 23 12:45:05.306354 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ May 23 12:45:05.306354 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self._enter() May 23 12:45:05.306354 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter May 23 12:45:05.306354 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) May 23 12:45:05.306354 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create May 23 12:45:05.306354 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self.creator() May 23 12:45:05.306354 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value May 23 12:45:05.306354 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application created_value = creator( May 23 12:45:05.307324 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain May 23 12:45:05.307324 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) May 23 12:45:05.307324 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. May 23 12:45:05.307324 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application May 23 12:45:05.307324 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 643/1283] 10.222.0.30 () {56 vars in 1108 bytes} [Mon May 23 12:45:05 2022] GET /identity/v3/domains/Default => generated 87 bytes in 166 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) May 23 12:45:05.312055 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a8e98c8-c1e2-4289-a661-d1fbd4dd7b56 None admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:45:05.428643 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a8e98c8-c1e2-4289-a661-d1fbd4dd7b56 None admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:45:05.465527 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79428ead-f108-4f8e-8c95-faaddca14a57 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:45:05.466296 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79428ead-f108-4f8e-8c95-faaddca14a57 None admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:45:05.466580 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79428ead-f108-4f8e-8c95-faaddca14a57 None admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:45:05.466863 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79428ead-f108-4f8e-8c95-faaddca14a57 None admin] PATH_INFO: `/v3/domains` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:45:05.467707 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79428ead-f108-4f8e-8c95-faaddca14a57 None admin] RBAC: Adding query filter params (name=Default) {{(pid=92056) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} May 23 12:45:05.468015 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79428ead-f108-4f8e-8c95-faaddca14a57 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:45:05.469427 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79428ead-f108-4f8e-8c95-faaddca14a57 None admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:45:05.482922 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 641/1284] 10.222.0.30 () {56 vars in 1109 bytes} [Mon May 23 12:45:05 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 171 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) May 23 12:45:05.488815 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b54058a9-6119-4602-a1fd-555f346e5c39 None admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:45:05.601015 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b54058a9-6119-4602-a1fd-555f346e5c39 None admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:45:05.615770 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=100 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:45:05.636740 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69cb911a-d02f-4831-a5a1-2d0f07a1b62e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:45:05.637549 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69cb911a-d02f-4831-a5a1-2d0f07a1b62e None admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:45:05.637830 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69cb911a-d02f-4831-a5a1-2d0f07a1b62e None admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:45:05.638134 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69cb911a-d02f-4831-a5a1-2d0f07a1b62e None admin] PATH_INFO: `/v3/projects/service` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:45:05.646431 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69cb911a-d02f-4831-a5a1-2d0f07a1b62e None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:45:05.647926 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69cb911a-d02f-4831-a5a1-2d0f07a1b62e None admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:45:05.663860 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application [None req-69cb911a-d02f-4831-a5a1-2d0f07a1b62e None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. May 23 12:45:05.663860 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application Traceback (most recent call last): May 23 12:45:05.663860 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request May 23 12:45:05.663860 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application rv = self.dispatch_request() May 23 12:45:05.663860 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request May 23 12:45:05.663860 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) May 23 12:45:05.663860 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper May 23 12:45:05.663860 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) May 23 12:45:05.663860 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view May 23 12:45:05.663860 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) May 23 12:45:05.663860 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request May 23 12:45:05.663860 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) May 23 12:45:05.663860 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get May 23 12:45:05.663860 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self._get_project(project_id) May 23 12:45:05.663860 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project May 23 12:45:05.665070 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) May 23 12:45:05.665070 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped May 23 12:45:05.665070 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) May 23 12:45:05.665070 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun May 23 12:45:05.665070 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) May 23 12:45:05.665070 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func May 23 12:45:05.665070 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self.get_or_create( May 23 12:45:05.665070 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create May 23 12:45:05.665070 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application with Lock( May 23 12:45:05.665070 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ May 23 12:45:05.665070 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self._enter() May 23 12:45:05.665070 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter May 23 12:45:05.665070 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) May 23 12:45:05.665070 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create May 23 12:45:05.665070 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self.creator() May 23 12:45:05.665070 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value May 23 12:45:05.665070 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application created_value = creator( May 23 12:45:05.666206 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project May 23 12:45:05.666206 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) May 23 12:45:05.666206 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project May 23 12:45:05.666206 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() May 23 12:45:05.666206 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project May 23 12:45:05.666206 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) May 23 12:45:05.666206 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. May 23 12:45:05.666206 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: ERROR keystone.server.flask.application May 23 12:45:05.666206 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 644/1285] 10.222.0.30 () {56 vars in 1111 bytes} [Mon May 23 12:45:05 2022] GET /identity/v3/projects/service => generated 88 bytes in 177 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) May 23 12:45:05.671211 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79428ead-f108-4f8e-8c95-faaddca14a57 None admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:45:05.732862 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=100 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:45:05.787082 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79428ead-f108-4f8e-8c95-faaddca14a57 None admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:45:05.820441 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dded1d94-f83e-4764-bac4-e9f8dc2e3779 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:45:05.821095 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dded1d94-f83e-4764-bac4-e9f8dc2e3779 None admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:45:05.821370 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dded1d94-f83e-4764-bac4-e9f8dc2e3779 None admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:45:05.821647 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dded1d94-f83e-4764-bac4-e9f8dc2e3779 None admin] PATH_INFO: `/v3/projects` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:45:05.822458 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dded1d94-f83e-4764-bac4-e9f8dc2e3779 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=92056) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} May 23 12:45:05.822765 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dded1d94-f83e-4764-bac4-e9f8dc2e3779 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:45:05.824184 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dded1d94-f83e-4764-bac4-e9f8dc2e3779 None admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:45:05.837516 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 642/1286] 10.222.0.30 () {56 vars in 1148 bytes} [Mon May 23 12:45:05 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 167 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) May 23 12:45:05.843726 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69cb911a-d02f-4831-a5a1-2d0f07a1b62e None admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:45:05.948584 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69cb911a-d02f-4831-a5a1-2d0f07a1b62e None admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:45:05.981970 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa544a19-c5ac-4b96-83cd-d43d1443535a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:45:05.982603 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa544a19-c5ac-4b96-83cd-d43d1443535a None admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:45:05.982884 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa544a19-c5ac-4b96-83cd-d43d1443535a None admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:45:05.983185 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa544a19-c5ac-4b96-83cd-d43d1443535a None admin] PATH_INFO: `/v3/role_assignments` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:45:05.983996 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa544a19-c5ac-4b96-83cd-d43d1443535a None admin] RBAC: Adding query filter params (role.id=bc3c3c9d654940cbaa5177a6be23d1bc, scope.project.id=9c389ae65aaa4542bb5a5c4fd0bacf5b, user.id=d5fb51f8d0fa46439cb29666970b885b) {{(pid=92057) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} May 23 12:45:05.984304 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa544a19-c5ac-4b96-83cd-d43d1443535a None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:45:05.985738 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa544a19-c5ac-4b96-83cd-d43d1443535a None admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:45:05.991958 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 645/1287] 10.222.0.30 () {56 vars in 1374 bytes} [Mon May 23 12:45:05 2022] GET /identity/v3/role_assignments?user.id=d5fb51f8d0fa46439cb29666970b885b&scope.project.id=9c389ae65aaa4542bb5a5c4fd0bacf5b&role.id=bc3c3c9d654940cbaa5177a6be23d1bc => generated 608 bytes in 149 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) May 23 12:45:08.266387 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d575cb6b-723a-4104-918d-257699dd0c79 None None] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:45:08.266933 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d575cb6b-723a-4104-918d-257699dd0c79 None None] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:45:08.266933 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d575cb6b-723a-4104-918d-257699dd0c79 None None] PATH_INFO: `/` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:45:08.268376 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 643/1288] 10.222.0.30 () {54 vars in 926 bytes} [Mon May 23 12:45:08 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) May 23 12:45:08.274720 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65508b3e-a8ca-468b-80c3-38268a2604f8 None None] REQUEST_METHOD: `POST` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:45:08.275167 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65508b3e-a8ca-468b-80c3-38268a2604f8 None None] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:45:08.275289 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65508b3e-a8ca-468b-80c3-38268a2604f8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:45:08.295763 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.auth.core [None req-65508b3e-a8ca-468b-80c3-38268a2604f8 None None] MFA Rules not processed for user `0f7119f8fa5c4445977671140ded91a4`. Rule list: `[]` (Enabled: `True`). {{(pid=92057) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 12:45:08.296951 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.fernet_utils [None req-65508b3e-a8ca-468b-80c3-38268a2604f8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92057) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 12:45:08.327668 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=100 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:45:08.364538 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 646/1289] 10.222.0.30 () {58 vars in 1023 bytes} [Mon May 23 12:45:08 2022] POST /identity/v3/auth/tokens => generated 2108 bytes in 91 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) May 23 12:45:08.387718 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d575cb6b-723a-4104-918d-257699dd0c79 None None] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:45:08.488781 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d575cb6b-723a-4104-918d-257699dd0c79 None None] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:45:08.499156 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=100 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:45:08.522350 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-161a2fa6-9e90-4ef9-b298-7b89894fda55 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:45:08.523066 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-161a2fa6-9e90-4ef9-b298-7b89894fda55 None admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:45:08.523381 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-161a2fa6-9e90-4ef9-b298-7b89894fda55 None admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:45:08.523652 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-161a2fa6-9e90-4ef9-b298-7b89894fda55 None admin] PATH_INFO: `/v3/services/placement` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:45:08.524294 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-161a2fa6-9e90-4ef9-b298-7b89894fda55 None admin] RBAC: Authorizing `identity:get_service(service_id=placement)` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:45:08.525968 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-161a2fa6-9e90-4ef9-b298-7b89894fda55 None admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:45:08.538462 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application [None req-161a2fa6-9e90-4ef9-b298-7b89894fda55 None admin] Could not find service: placement.: keystone.exception.ServiceNotFound: Could not find service: placement. May 23 12:45:08.538462 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application Traceback (most recent call last): May 23 12:45:08.538462 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service May 23 12:45:08.538462 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) May 23 12:45:08.538462 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service May 23 12:45:08.538462 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() May 23 12:45:08.538462 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service May 23 12:45:08.538462 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) May 23 12:45:08.538462 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: placement. May 23 12:45:08.538462 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application May 23 12:45:08.538462 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: May 23 12:45:08.538462 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application May 23 12:45:08.538462 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application Traceback (most recent call last): May 23 12:45:08.538462 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request May 23 12:45:08.538462 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application rv = self.dispatch_request() May 23 12:45:08.538462 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request May 23 12:45:08.538462 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) May 23 12:45:08.542214 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper May 23 12:45:08.542214 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) May 23 12:45:08.542214 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view May 23 12:45:08.542214 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) May 23 12:45:08.542214 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request May 23 12:45:08.542214 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) May 23 12:45:08.542214 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get May 23 12:45:08.542214 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self._get_service(service_id) May 23 12:45:08.542214 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service May 23 12:45:08.542214 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) May 23 12:45:08.542214 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped May 23 12:45:08.542214 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) May 23 12:45:08.542214 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun May 23 12:45:08.542214 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) May 23 12:45:08.542214 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func May 23 12:45:08.542214 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self.get_or_create( May 23 12:45:08.543299 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create May 23 12:45:08.543299 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application with Lock( May 23 12:45:08.543299 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ May 23 12:45:08.543299 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self._enter() May 23 12:45:08.543299 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter May 23 12:45:08.543299 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) May 23 12:45:08.543299 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create May 23 12:45:08.543299 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self.creator() May 23 12:45:08.543299 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value May 23 12:45:08.543299 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application created_value = creator( May 23 12:45:08.543299 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service May 23 12:45:08.543299 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) May 23 12:45:08.543299 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: placement. May 23 12:45:08.543299 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application May 23 12:45:08.543299 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 644/1290] 10.222.0.30 () {56 vars in 1117 bytes} [Mon May 23 12:45:08 2022] GET /identity/v3/services/placement => generated 90 bytes in 153 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) May 23 12:45:08.545700 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65508b3e-a8ca-468b-80c3-38268a2604f8 None None] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:45:08.645938 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65508b3e-a8ca-468b-80c3-38268a2604f8 None None] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:45:08.680286 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca6c396e-400f-44c5-9dd7-06e930bc0c21 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:45:08.680956 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca6c396e-400f-44c5-9dd7-06e930bc0c21 None admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:45:08.681236 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca6c396e-400f-44c5-9dd7-06e930bc0c21 None admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:45:08.681512 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca6c396e-400f-44c5-9dd7-06e930bc0c21 None admin] PATH_INFO: `/v3/services` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:45:08.682251 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca6c396e-400f-44c5-9dd7-06e930bc0c21 None admin] RBAC: Adding query filter params (name=placement) {{(pid=92057) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} May 23 12:45:08.682560 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca6c396e-400f-44c5-9dd7-06e930bc0c21 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:45:08.683943 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca6c396e-400f-44c5-9dd7-06e930bc0c21 None admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:45:08.688877 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 647/1291] 10.222.0.30 () {56 vars in 1116 bytes} [Mon May 23 12:45:08 2022] GET /identity/v3/services?name=placement => generated 126 bytes in 144 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) May 23 12:45:08.695174 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-161a2fa6-9e90-4ef9-b298-7b89894fda55 None admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:45:08.808156 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-161a2fa6-9e90-4ef9-b298-7b89894fda55 None admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:45:08.841351 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91499a12-fc1b-4626-b8ac-acff700fe163 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:45:08.842035 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91499a12-fc1b-4626-b8ac-acff700fe163 None admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:45:08.842321 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91499a12-fc1b-4626-b8ac-acff700fe163 None admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:45:08.842596 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91499a12-fc1b-4626-b8ac-acff700fe163 None admin] PATH_INFO: `/v3/services` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:45:08.843328 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-91499a12-fc1b-4626-b8ac-acff700fe163 None admin] RBAC: Adding query filter params (type=placement) {{(pid=92056) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} May 23 12:45:08.843636 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-91499a12-fc1b-4626-b8ac-acff700fe163 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:45:08.845025 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-91499a12-fc1b-4626-b8ac-acff700fe163 None admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:45:08.850386 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 645/1292] 10.222.0.30 () {56 vars in 1116 bytes} [Mon May 23 12:45:08 2022] GET /identity/v3/services?type=placement => generated 126 bytes in 156 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) May 23 12:45:11.007349 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bb1a311-ed42-4ab8-b374-5d0e16190731 None None] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:45:11.007913 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bb1a311-ed42-4ab8-b374-5d0e16190731 None None] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:45:11.007913 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bb1a311-ed42-4ab8-b374-5d0e16190731 None None] PATH_INFO: `/` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:45:11.009212 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 648/1293] 10.222.0.30 () {54 vars in 926 bytes} [Mon May 23 12:45:11 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) May 23 12:45:11.014544 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c591f1fd-be96-4b2e-84bd-5e997db3e592 None None] REQUEST_METHOD: `POST` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:45:11.014960 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c591f1fd-be96-4b2e-84bd-5e997db3e592 None None] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:45:11.015128 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c591f1fd-be96-4b2e-84bd-5e997db3e592 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:45:11.033289 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.auth.core [None req-c591f1fd-be96-4b2e-84bd-5e997db3e592 None None] MFA Rules not processed for user `0f7119f8fa5c4445977671140ded91a4`. Rule list: `[]` (Enabled: `True`). {{(pid=92056) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 12:45:11.034691 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.fernet_utils [None req-c591f1fd-be96-4b2e-84bd-5e997db3e592 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92056) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 12:45:11.063495 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=100 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:45:11.100275 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 646/1294] 10.222.0.30 () {58 vars in 1023 bytes} [Mon May 23 12:45:11 2022] POST /identity/v3/auth/tokens => generated 2108 bytes in 87 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) May 23 12:45:11.123136 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3bb1a311-ed42-4ab8-b374-5d0e16190731 None None] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:45:11.154647 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=100 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:45:11.224528 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3bb1a311-ed42-4ab8-b374-5d0e16190731 None None] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:45:11.259948 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c72cd4e-ef11-490d-a491-8113f969af92 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:45:11.260576 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c72cd4e-ef11-490d-a491-8113f969af92 None admin] REQUEST_METHOD: `POST` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:45:11.260867 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c72cd4e-ef11-490d-a491-8113f969af92 None admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:45:11.261146 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c72cd4e-ef11-490d-a491-8113f969af92 None admin] PATH_INFO: `/v3/services` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:45:11.261924 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c72cd4e-ef11-490d-a491-8113f969af92 None admin] RBAC: Authorizing `identity:create_service()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:45:11.263595 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c72cd4e-ef11-490d-a491-8113f969af92 None admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:45:11.366433 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 649/1295] 10.222.0.30 () {60 vars in 1141 bytes} [Mon May 23 12:45:11 2022] POST /identity/v3/services => generated 246 bytes in 244 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) May 23 12:45:13.592497 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1927ad32-4ea7-4c54-a30e-b91c98660982 None None] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:45:13.593064 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1927ad32-4ea7-4c54-a30e-b91c98660982 None None] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:45:13.593064 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1927ad32-4ea7-4c54-a30e-b91c98660982 None None] PATH_INFO: `/` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:45:13.594469 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 647/1296] 10.222.0.30 () {54 vars in 926 bytes} [Mon May 23 12:45:13 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) May 23 12:45:13.599536 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-556b2db6-b6b3-41ba-9451-a911a64c77e9 None None] REQUEST_METHOD: `POST` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:45:13.599974 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-556b2db6-b6b3-41ba-9451-a911a64c77e9 None None] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:45:13.600079 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-556b2db6-b6b3-41ba-9451-a911a64c77e9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:45:13.617313 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.auth.core [None req-556b2db6-b6b3-41ba-9451-a911a64c77e9 None None] MFA Rules not processed for user `0f7119f8fa5c4445977671140ded91a4`. Rule list: `[]` (Enabled: `True`). {{(pid=92057) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 12:45:13.618516 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.fernet_utils [None req-556b2db6-b6b3-41ba-9451-a911a64c77e9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92057) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 12:45:13.694085 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 650/1297] 10.222.0.30 () {58 vars in 1023 bytes} [Mon May 23 12:45:13 2022] POST /identity/v3/auth/tokens => generated 2211 bytes in 95 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) May 23 12:45:13.716500 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1927ad32-4ea7-4c54-a30e-b91c98660982 None None] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:45:13.826334 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1927ad32-4ea7-4c54-a30e-b91c98660982 None None] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:45:13.859770 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89889fc5-1c93-4197-82f1-92b764e5a7c9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:45:13.860594 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89889fc5-1c93-4197-82f1-92b764e5a7c9 None admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:45:13.860875 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89889fc5-1c93-4197-82f1-92b764e5a7c9 None admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:45:13.861147 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89889fc5-1c93-4197-82f1-92b764e5a7c9 None admin] PATH_INFO: `/v3/services/placement` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:45:13.861797 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89889fc5-1c93-4197-82f1-92b764e5a7c9 None admin] RBAC: Authorizing `identity:get_service(service_id=placement)` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:45:13.863895 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89889fc5-1c93-4197-82f1-92b764e5a7c9 None admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:45:13.875864 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application [None req-89889fc5-1c93-4197-82f1-92b764e5a7c9 None admin] Could not find service: placement.: keystone.exception.ServiceNotFound: Could not find service: placement. May 23 12:45:13.875864 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application Traceback (most recent call last): May 23 12:45:13.875864 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service May 23 12:45:13.875864 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) May 23 12:45:13.875864 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service May 23 12:45:13.875864 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() May 23 12:45:13.875864 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service May 23 12:45:13.875864 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) May 23 12:45:13.875864 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: placement. May 23 12:45:13.875864 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application May 23 12:45:13.875864 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: May 23 12:45:13.875864 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application May 23 12:45:13.875864 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application Traceback (most recent call last): May 23 12:45:13.875864 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request May 23 12:45:13.875864 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application rv = self.dispatch_request() May 23 12:45:13.875864 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request May 23 12:45:13.875864 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) May 23 12:45:13.877082 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper May 23 12:45:13.877082 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) May 23 12:45:13.877082 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view May 23 12:45:13.877082 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) May 23 12:45:13.877082 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request May 23 12:45:13.877082 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) May 23 12:45:13.877082 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get May 23 12:45:13.877082 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self._get_service(service_id) May 23 12:45:13.877082 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service May 23 12:45:13.877082 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) May 23 12:45:13.877082 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped May 23 12:45:13.877082 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) May 23 12:45:13.877082 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun May 23 12:45:13.877082 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) May 23 12:45:13.877082 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func May 23 12:45:13.877082 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self.get_or_create( May 23 12:45:13.878743 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create May 23 12:45:13.878743 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application with Lock( May 23 12:45:13.878743 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ May 23 12:45:13.878743 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self._enter() May 23 12:45:13.878743 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter May 23 12:45:13.878743 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) May 23 12:45:13.878743 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create May 23 12:45:13.878743 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self.creator() May 23 12:45:13.878743 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value May 23 12:45:13.878743 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application created_value = creator( May 23 12:45:13.878743 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service May 23 12:45:13.878743 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) May 23 12:45:13.878743 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: placement. May 23 12:45:13.878743 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application May 23 12:45:13.878743 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 648/1298] 10.222.0.30 () {56 vars in 1117 bytes} [Mon May 23 12:45:13 2022] GET /identity/v3/services/placement => generated 90 bytes in 161 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) May 23 12:45:13.883869 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-556b2db6-b6b3-41ba-9451-a911a64c77e9 None None] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:45:13.902101 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:45:13.989294 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-556b2db6-b6b3-41ba-9451-a911a64c77e9 None None] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:45:14.023006 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7a577c8-30dd-4a0a-aa76-59fbbebbc566 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:45:14.023853 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7a577c8-30dd-4a0a-aa76-59fbbebbc566 None admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:45:14.023953 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7a577c8-30dd-4a0a-aa76-59fbbebbc566 None admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:45:14.024201 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7a577c8-30dd-4a0a-aa76-59fbbebbc566 None admin] PATH_INFO: `/v3/services` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:45:14.024903 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c7a577c8-30dd-4a0a-aa76-59fbbebbc566 None admin] RBAC: Adding query filter params (name=placement) {{(pid=92057) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} May 23 12:45:14.025212 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c7a577c8-30dd-4a0a-aa76-59fbbebbc566 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:45:14.026892 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c7a577c8-30dd-4a0a-aa76-59fbbebbc566 None admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:45:14.031922 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 651/1299] 10.222.0.30 () {56 vars in 1116 bytes} [Mon May 23 12:45:13 2022] GET /identity/v3/services?name=placement => generated 358 bytes in 149 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) May 23 12:45:14.038471 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89889fc5-1c93-4197-82f1-92b764e5a7c9 None admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:45:14.070095 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=100 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:45:14.136062 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89889fc5-1c93-4197-82f1-92b764e5a7c9 None admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:45:14.169744 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1ee0cd8-2c6c-481d-8e54-9c5be3c81311 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:45:14.170429 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1ee0cd8-2c6c-481d-8e54-9c5be3c81311 None admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:45:14.170718 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1ee0cd8-2c6c-481d-8e54-9c5be3c81311 None admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:45:14.170994 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1ee0cd8-2c6c-481d-8e54-9c5be3c81311 None admin] PATH_INFO: `/v3/endpoints` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:45:14.171777 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1ee0cd8-2c6c-481d-8e54-9c5be3c81311 None admin] RBAC: Adding query filter params (interface=public, service_id=0977e426a86249b5bba6d2e948231beb, region_id=RegionOne) {{(pid=92056) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} May 23 12:45:14.172089 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1ee0cd8-2c6c-481d-8e54-9c5be3c81311 None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:45:14.173462 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1ee0cd8-2c6c-481d-8e54-9c5be3c81311 None admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:45:14.179118 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 649/1300] 10.222.0.30 () {56 vars in 1251 bytes} [Mon May 23 12:45:14 2022] GET /identity/v3/endpoints?service_id=0977e426a86249b5bba6d2e948231beb&interface=public®ion_id=RegionOne => generated 194 bytes in 141 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) May 23 12:45:14.184731 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7a577c8-30dd-4a0a-aa76-59fbbebbc566 None admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:45:14.266492 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=100 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:45:14.284739 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7a577c8-30dd-4a0a-aa76-59fbbebbc566 None admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:45:14.318559 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-287c2863-6a4a-46d8-957a-e39cf258cad0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:45:14.321628 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-287c2863-6a4a-46d8-957a-e39cf258cad0 None admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:45:14.321628 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-287c2863-6a4a-46d8-957a-e39cf258cad0 None admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:45:14.321628 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-287c2863-6a4a-46d8-957a-e39cf258cad0 None admin] PATH_INFO: `/v3/services` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:45:14.321628 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-287c2863-6a4a-46d8-957a-e39cf258cad0 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:45:14.322013 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-287c2863-6a4a-46d8-957a-e39cf258cad0 None admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:45:14.328574 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 652/1301] 10.222.0.30 () {56 vars in 1087 bytes} [Mon May 23 12:45:14 2022] GET /identity/v3/services => generated 2191 bytes in 144 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) May 23 12:45:16.520174 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf0ea643-e531-4baf-a294-be59f9b84ff8 None None] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:45:16.521202 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf0ea643-e531-4baf-a294-be59f9b84ff8 None None] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:45:16.521319 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf0ea643-e531-4baf-a294-be59f9b84ff8 None None] PATH_INFO: `/` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:45:16.522788 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 650/1302] 10.222.0.30 () {54 vars in 926 bytes} [Mon May 23 12:45:16 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) May 23 12:45:16.528141 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9dd1e33b-76fe-49c7-bfd4-29e09c3e5010 None None] REQUEST_METHOD: `POST` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:45:16.528547 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9dd1e33b-76fe-49c7-bfd4-29e09c3e5010 None None] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:45:16.528728 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9dd1e33b-76fe-49c7-bfd4-29e09c3e5010 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:45:16.545034 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.auth.core [None req-9dd1e33b-76fe-49c7-bfd4-29e09c3e5010 None None] MFA Rules not processed for user `0f7119f8fa5c4445977671140ded91a4`. Rule list: `[]` (Enabled: `True`). {{(pid=92057) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 12:45:16.546244 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.fernet_utils [None req-9dd1e33b-76fe-49c7-bfd4-29e09c3e5010 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92057) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 12:45:16.611381 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 653/1303] 10.222.0.30 () {58 vars in 1023 bytes} [Mon May 23 12:45:16 2022] POST /identity/v3/auth/tokens => generated 2211 bytes in 84 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) May 23 12:45:16.635116 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf0ea643-e531-4baf-a294-be59f9b84ff8 None None] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:45:16.730162 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=100 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:45:16.736734 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf0ea643-e531-4baf-a294-be59f9b84ff8 None None] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:45:16.770467 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a26c1d5-defb-4cf0-8bf3-5aa0f8759f89 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:45:16.771195 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a26c1d5-defb-4cf0-8bf3-5aa0f8759f89 None admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:45:16.771474 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a26c1d5-defb-4cf0-8bf3-5aa0f8759f89 None admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:45:16.771746 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a26c1d5-defb-4cf0-8bf3-5aa0f8759f89 None admin] PATH_INFO: `/v3/services/placement` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:45:16.772393 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a26c1d5-defb-4cf0-8bf3-5aa0f8759f89 None admin] RBAC: Authorizing `identity:get_service(service_id=placement)` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:45:16.773755 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a26c1d5-defb-4cf0-8bf3-5aa0f8759f89 None admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:45:16.785762 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application [None req-6a26c1d5-defb-4cf0-8bf3-5aa0f8759f89 None admin] Could not find service: placement.: keystone.exception.ServiceNotFound: Could not find service: placement. May 23 12:45:16.785762 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application Traceback (most recent call last): May 23 12:45:16.785762 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service May 23 12:45:16.785762 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) May 23 12:45:16.785762 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service May 23 12:45:16.785762 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() May 23 12:45:16.785762 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service May 23 12:45:16.785762 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) May 23 12:45:16.785762 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: placement. May 23 12:45:16.785762 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application May 23 12:45:16.785762 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: May 23 12:45:16.785762 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application May 23 12:45:16.785762 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application Traceback (most recent call last): May 23 12:45:16.785762 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request May 23 12:45:16.785762 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application rv = self.dispatch_request() May 23 12:45:16.785762 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request May 23 12:45:16.785762 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) May 23 12:45:16.787397 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper May 23 12:45:16.787397 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) May 23 12:45:16.787397 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view May 23 12:45:16.787397 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) May 23 12:45:16.787397 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request May 23 12:45:16.787397 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) May 23 12:45:16.787397 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get May 23 12:45:16.787397 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self._get_service(service_id) May 23 12:45:16.787397 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service May 23 12:45:16.787397 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) May 23 12:45:16.787397 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped May 23 12:45:16.787397 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) May 23 12:45:16.787397 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun May 23 12:45:16.787397 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) May 23 12:45:16.787397 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func May 23 12:45:16.787397 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self.get_or_create( May 23 12:45:16.788472 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create May 23 12:45:16.788472 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application with Lock( May 23 12:45:16.788472 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ May 23 12:45:16.788472 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self._enter() May 23 12:45:16.788472 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter May 23 12:45:16.788472 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) May 23 12:45:16.788472 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create May 23 12:45:16.788472 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application return self.creator() May 23 12:45:16.788472 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value May 23 12:45:16.788472 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application created_value = creator( May 23 12:45:16.788472 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service May 23 12:45:16.788472 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) May 23 12:45:16.788472 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: placement. May 23 12:45:16.788472 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: ERROR keystone.server.flask.application May 23 12:45:16.788472 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 651/1304] 10.222.0.30 () {56 vars in 1117 bytes} [Mon May 23 12:45:16 2022] GET /identity/v3/services/placement => generated 90 bytes in 152 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) May 23 12:45:16.793168 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9dd1e33b-76fe-49c7-bfd4-29e09c3e5010 None None] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:45:16.868159 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=100 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:45:16.890213 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9dd1e33b-76fe-49c7-bfd4-29e09c3e5010 None None] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:45:16.921867 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43064d67-d033-4d6b-967f-afbc3748521e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:45:16.922541 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43064d67-d033-4d6b-967f-afbc3748521e None admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:45:16.922820 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43064d67-d033-4d6b-967f-afbc3748521e None admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:45:16.923090 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43064d67-d033-4d6b-967f-afbc3748521e None admin] PATH_INFO: `/v3/services` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:45:16.923779 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43064d67-d033-4d6b-967f-afbc3748521e None admin] RBAC: Adding query filter params (name=placement) {{(pid=92057) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} May 23 12:45:16.924084 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43064d67-d033-4d6b-967f-afbc3748521e None admin] RBAC: Authorizing `identity:list_services()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:45:16.926010 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43064d67-d033-4d6b-967f-afbc3748521e None admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:45:16.930964 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 654/1305] 10.222.0.30 () {56 vars in 1116 bytes} [Mon May 23 12:45:16 2022] GET /identity/v3/services?name=placement => generated 358 bytes in 138 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) May 23 12:45:16.936780 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a26c1d5-defb-4cf0-8bf3-5aa0f8759f89 None admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:45:17.036753 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a26c1d5-defb-4cf0-8bf3-5aa0f8759f89 None admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:45:17.069222 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d392697-a71c-4228-8ee2-bf7f9317d1c0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:45:17.069859 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d392697-a71c-4228-8ee2-bf7f9317d1c0 None admin] REQUEST_METHOD: `POST` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:45:17.070171 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d392697-a71c-4228-8ee2-bf7f9317d1c0 None admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:45:17.070441 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d392697-a71c-4228-8ee2-bf7f9317d1c0 None admin] PATH_INFO: `/v3/endpoints` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:45:17.071173 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d392697-a71c-4228-8ee2-bf7f9317d1c0 None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:45:17.072593 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d392697-a71c-4228-8ee2-bf7f9317d1c0 None admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:45:17.130411 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 652/1306] 10.222.0.30 () {60 vars in 1144 bytes} [Mon May 23 12:45:16 2022] POST /identity/v3/endpoints => generated 331 bytes in 194 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) May 23 12:45:26.964582 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=30 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:45:27.125746 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=91,keystone:INSERT=1 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:51:00.092899 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47dfae8e-d2a3-4192-82b7-69b4ca873963 None None] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:51:00.093869 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47dfae8e-d2a3-4192-82b7-69b4ca873963 None None] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:51:00.094394 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47dfae8e-d2a3-4192-82b7-69b4ca873963 None None] PATH_INFO: `/` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:51:00.101742 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9e08d20-91eb-4670-bed4-c8d2d0e08793 None None] REQUEST_METHOD: `POST` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:51:00.102613 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9e08d20-91eb-4670-bed4-c8d2d0e08793 None None] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:51:00.103116 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9e08d20-91eb-4670-bed4-c8d2d0e08793 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:51:00.106989 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 655/1307] 10.222.0.30 () {54 vars in 926 bytes} [Mon May 23 12:51:00 2022] GET /identity => generated 270 bytes in 15 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) May 23 12:51:00.227751 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.auth.core [None req-d9e08d20-91eb-4670-bed4-c8d2d0e08793 None None] MFA Rules not processed for user `0f7119f8fa5c4445977671140ded91a4`. Rule list: `[]` (Enabled: `True`). {{(pid=92056) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 12:51:00.227751 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.fernet_utils [None req-d9e08d20-91eb-4670-bed4-c8d2d0e08793 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92056) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 12:51:00.374225 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 653/1308] 10.222.0.30 () {58 vars in 1023 bytes} [Mon May 23 12:51:00 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 273 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) May 23 12:51:00.418784 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0e6d070-b110-4675-9fab-ebdd93e56e1b None None] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:51:00.419420 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0e6d070-b110-4675-9fab-ebdd93e56e1b None None] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:51:00.419905 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0e6d070-b110-4675-9fab-ebdd93e56e1b None None] PATH_INFO: `/` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:51:00.421863 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 656/1309] 10.222.0.30 () {54 vars in 964 bytes} [Mon May 23 12:51:00 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) May 23 12:51:00.438620 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49f536e4-569d-4157-a207-e399607cbb2f None None] REQUEST_METHOD: `POST` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:51:00.438850 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49f536e4-569d-4157-a207-e399607cbb2f None None] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:51:00.439128 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49f536e4-569d-4157-a207-e399607cbb2f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:51:00.521953 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.auth.core [None req-49f536e4-569d-4157-a207-e399607cbb2f None None] MFA Rules not processed for user `2eb83ca38e79449095f3a2f947bbb06a`. Rule list: `[]` (Enabled: `True`). {{(pid=92056) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 12:51:00.523189 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.fernet_utils [None req-49f536e4-569d-4157-a207-e399607cbb2f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92056) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 12:51:00.546689 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=100 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:51:00.607086 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 654/1310] 10.222.0.30 () {58 vars in 1061 bytes} [Mon May 23 12:51:00 2022] POST /identity/v3/auth/tokens => generated 3111 bytes in 169 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) May 23 12:51:00.616426 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0e6d070-b110-4675-9fab-ebdd93e56e1b None None] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:51:00.618398 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.fernet_utils [None req-d0e6d070-b110-4675-9fab-ebdd93e56e1b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92057) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 12:51:00.736943 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0e6d070-b110-4675-9fab-ebdd93e56e1b None None] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:51:00.816575 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3155bbf8-0d34-45dc-9d7b-11df0bc021c1 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2eb83ca38e79449095f3a2f947bbb06a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9c389ae65aaa4542bb5a5c4fd0bacf5b', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:51:00.818072 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3155bbf8-0d34-45dc-9d7b-11df0bc021c1 service neutron] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:51:00.818451 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3155bbf8-0d34-45dc-9d7b-11df0bc021c1 service neutron] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:51:00.818848 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3155bbf8-0d34-45dc-9d7b-11df0bc021c1 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:51:00.820499 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.fernet_utils [None req-3155bbf8-0d34-45dc-9d7b-11df0bc021c1 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92057) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 12:51:00.881222 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=100 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:51:00.941351 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3155bbf8-0d34-45dc-9d7b-11df0bc021c1 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:51:00.942970 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3155bbf8-0d34-45dc-9d7b-11df0bc021c1 service neutron] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:51:00.962903 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 657/1311] 10.222.0.30 () {60 vars in 1365 bytes} [Mon May 23 12:51:00 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 347 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) May 23 12:51:03.461516 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32e90e58-b26c-4cc6-aea2-46f8f722f92a None None] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:51:03.462048 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32e90e58-b26c-4cc6-aea2-46f8f722f92a None None] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:51:03.462365 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32e90e58-b26c-4cc6-aea2-46f8f722f92a None None] PATH_INFO: `/` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:51:03.466077 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 655/1312] 10.222.0.30 () {54 vars in 926 bytes} [Mon May 23 12:51:03 2022] GET /identity => generated 270 bytes in 6 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) May 23 12:51:03.469284 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-942ca132-f9e4-46b7-bed6-eea5e0f1a727 None None] REQUEST_METHOD: `POST` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:51:03.469760 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-942ca132-f9e4-46b7-bed6-eea5e0f1a727 None None] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:51:03.469914 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-942ca132-f9e4-46b7-bed6-eea5e0f1a727 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:51:03.551824 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.auth.core [None req-942ca132-f9e4-46b7-bed6-eea5e0f1a727 None None] MFA Rules not processed for user `0f7119f8fa5c4445977671140ded91a4`. Rule list: `[]` (Enabled: `True`). {{(pid=92057) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 12:51:03.553267 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.fernet_utils [None req-942ca132-f9e4-46b7-bed6-eea5e0f1a727 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92057) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 12:51:03.657401 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 658/1313] 10.222.0.30 () {58 vars in 1023 bytes} [Mon May 23 12:51:03 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 189 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) May 23 12:51:03.700216 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32e90e58-b26c-4cc6-aea2-46f8f722f92a None None] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:51:03.701807 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.fernet_utils [None req-32e90e58-b26c-4cc6-aea2-46f8f722f92a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92056) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 12:51:03.812449 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32e90e58-b26c-4cc6-aea2-46f8f722f92a None None] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:51:03.891792 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=100 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:51:03.894585 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f2117c8-15ea-44f6-9f38-12fc3edfa371 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2eb83ca38e79449095f3a2f947bbb06a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9c389ae65aaa4542bb5a5c4fd0bacf5b', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:51:03.895758 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f2117c8-15ea-44f6-9f38-12fc3edfa371 service neutron] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:51:03.896184 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f2117c8-15ea-44f6-9f38-12fc3edfa371 service neutron] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:51:03.896461 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f2117c8-15ea-44f6-9f38-12fc3edfa371 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:51:03.898279 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.fernet_utils [None req-7f2117c8-15ea-44f6-9f38-12fc3edfa371 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92056) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 12:51:04.015573 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f2117c8-15ea-44f6-9f38-12fc3edfa371 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:51:04.016955 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f2117c8-15ea-44f6-9f38-12fc3edfa371 service neutron] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:51:04.036799 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 656/1314] 10.222.0.30 () {60 vars in 1365 bytes} [Mon May 23 12:51:03 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 337 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) May 23 12:51:06.719901 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-300b0f2b-a2b2-4050-b2ee-8df572212cf6 None None] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:51:06.720412 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-300b0f2b-a2b2-4050-b2ee-8df572212cf6 None None] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:51:06.720528 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-300b0f2b-a2b2-4050-b2ee-8df572212cf6 None None] PATH_INFO: `/` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:51:06.721834 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 659/1315] 10.222.0.30 () {54 vars in 926 bytes} [Mon May 23 12:51:06 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) May 23 12:51:06.727174 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e776f3f-41fc-4c13-94a1-1f3485079b7e None None] REQUEST_METHOD: `POST` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:51:06.727591 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e776f3f-41fc-4c13-94a1-1f3485079b7e None None] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:51:06.727734 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e776f3f-41fc-4c13-94a1-1f3485079b7e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:51:06.791228 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.auth.core [None req-3e776f3f-41fc-4c13-94a1-1f3485079b7e None None] MFA Rules not processed for user `b3478ea957234bb99fde27e6c3d0dfd3`. Rule list: `[]` (Enabled: `True`). {{(pid=92056) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 12:51:06.792431 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.fernet_utils [None req-3e776f3f-41fc-4c13-94a1-1f3485079b7e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92056) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 12:51:06.846102 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=100 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:51:06.890331 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 657/1316] 10.222.0.30 () {58 vars in 1023 bytes} [Mon May 23 12:51:06 2022] POST /identity/v3/auth/tokens => generated 3233 bytes in 164 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) May 23 12:51:06.930797 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-300b0f2b-a2b2-4050-b2ee-8df572212cf6 None None] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:51:06.932305 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.fernet_utils [None req-300b0f2b-a2b2-4050-b2ee-8df572212cf6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92057) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 12:51:06.942494 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=100 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:51:07.033308 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-300b0f2b-a2b2-4050-b2ee-8df572212cf6 None None] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:51:07.105570 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb19ed03-a519-4c68-9ba1-445ab56792a3 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2eb83ca38e79449095f3a2f947bbb06a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9c389ae65aaa4542bb5a5c4fd0bacf5b', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:51:07.106232 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb19ed03-a519-4c68-9ba1-445ab56792a3 service neutron] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:51:07.106514 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb19ed03-a519-4c68-9ba1-445ab56792a3 service neutron] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:51:07.106788 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb19ed03-a519-4c68-9ba1-445ab56792a3 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:51:07.108384 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.fernet_utils [None req-fb19ed03-a519-4c68-9ba1-445ab56792a3 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92057) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 12:51:07.168766 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=100 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:51:07.215044 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb19ed03-a519-4c68-9ba1-445ab56792a3 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:51:07.216479 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb19ed03-a519-4c68-9ba1-445ab56792a3 service neutron] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:51:07.235490 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 660/1317] 10.222.0.30 () {60 vars in 1365 bytes} [Mon May 23 12:51:06 2022] GET /identity/v3/auth/tokens => generated 3233 bytes in 305 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) May 23 12:51:10.262817 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db8a4e34-6f66-4cc5-bfc7-bcbbbe716c01 None None] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:51:10.263559 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db8a4e34-6f66-4cc5-bfc7-bcbbbe716c01 None None] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:51:10.263559 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db8a4e34-6f66-4cc5-bfc7-bcbbbe716c01 None None] PATH_INFO: `/` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:51:10.264662 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 658/1318] 10.222.0.30 () {54 vars in 926 bytes} [Mon May 23 12:51:10 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) May 23 12:51:10.271366 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c545bce-1f60-4c18-9df3-06086d2b0281 None None] REQUEST_METHOD: `POST` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:51:10.271833 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c545bce-1f60-4c18-9df3-06086d2b0281 None None] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:51:10.272015 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c545bce-1f60-4c18-9df3-06086d2b0281 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:51:10.342737 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.auth.core [None req-0c545bce-1f60-4c18-9df3-06086d2b0281 None None] MFA Rules not processed for user `0f7119f8fa5c4445977671140ded91a4`. Rule list: `[]` (Enabled: `True`). {{(pid=92057) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 12:51:10.344008 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.fernet_utils [None req-0c545bce-1f60-4c18-9df3-06086d2b0281 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92057) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 12:51:10.438805 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 661/1319] 10.222.0.30 () {58 vars in 1023 bytes} [Mon May 23 12:51:10 2022] POST /identity/v3/auth/tokens => generated 3228 bytes in 170 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) May 23 12:51:10.479200 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db8a4e34-6f66-4cc5-bfc7-bcbbbe716c01 None None] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:51:10.480714 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.fernet_utils [None req-db8a4e34-6f66-4cc5-bfc7-bcbbbe716c01 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92056) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 12:51:10.582942 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db8a4e34-6f66-4cc5-bfc7-bcbbbe716c01 None None] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:51:10.653743 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec6516fc-9633-48e0-9bcc-1c78b51b6fb7 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2eb83ca38e79449095f3a2f947bbb06a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9c389ae65aaa4542bb5a5c4fd0bacf5b', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:51:10.654412 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec6516fc-9633-48e0-9bcc-1c78b51b6fb7 service neutron] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:51:10.654696 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec6516fc-9633-48e0-9bcc-1c78b51b6fb7 service neutron] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:51:10.654972 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec6516fc-9633-48e0-9bcc-1c78b51b6fb7 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:51:10.656596 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.fernet_utils [None req-ec6516fc-9633-48e0-9bcc-1c78b51b6fb7 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92056) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 12:51:10.664596 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=100 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:51:10.764068 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec6516fc-9633-48e0-9bcc-1c78b51b6fb7 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:51:10.765525 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec6516fc-9633-48e0-9bcc-1c78b51b6fb7 service neutron] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:51:10.783977 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 659/1320] 10.222.0.30 () {60 vars in 1365 bytes} [Mon May 23 12:51:10 2022] GET /identity/v3/auth/tokens => generated 3228 bytes in 305 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) May 23 12:51:13.978464 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6e38592-686e-43e6-9971-c71e8148e6a0 None None] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:51:13.978889 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6e38592-686e-43e6-9971-c71e8148e6a0 None None] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:51:13.979076 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6e38592-686e-43e6-9971-c71e8148e6a0 None None] PATH_INFO: `/` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:51:13.980405 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 662/1321] 10.222.0.30 () {54 vars in 926 bytes} [Mon May 23 12:51:13 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) May 23 12:51:13.985765 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-068c120e-5cfc-4355-a877-27b6cd0f0ddb None None] REQUEST_METHOD: `POST` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:51:13.986207 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-068c120e-5cfc-4355-a877-27b6cd0f0ddb None None] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:51:13.986372 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-068c120e-5cfc-4355-a877-27b6cd0f0ddb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:51:14.051407 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.auth.core [None req-068c120e-5cfc-4355-a877-27b6cd0f0ddb None None] MFA Rules not processed for user `0f7119f8fa5c4445977671140ded91a4`. Rule list: `[]` (Enabled: `True`). {{(pid=92056) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 12:51:14.052605 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.fernet_utils [None req-068c120e-5cfc-4355-a877-27b6cd0f0ddb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92056) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 12:51:14.109429 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=100 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:51:14.148847 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 660/1322] 10.222.0.30 () {58 vars in 1023 bytes} [Mon May 23 12:51:13 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 164 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) May 23 12:51:14.184613 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6e38592-686e-43e6-9971-c71e8148e6a0 None None] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:51:14.186145 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.fernet_utils [None req-c6e38592-686e-43e6-9971-c71e8148e6a0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92057) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 12:51:14.205500 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=100 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:51:14.288520 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6e38592-686e-43e6-9971-c71e8148e6a0 None None] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:51:14.362680 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25f75b64-98ca-463b-89e7-2f9b39934491 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2eb83ca38e79449095f3a2f947bbb06a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9c389ae65aaa4542bb5a5c4fd0bacf5b', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:51:14.364045 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25f75b64-98ca-463b-89e7-2f9b39934491 service neutron] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:51:14.364319 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25f75b64-98ca-463b-89e7-2f9b39934491 service neutron] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:51:14.364599 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25f75b64-98ca-463b-89e7-2f9b39934491 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:51:14.366276 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.fernet_utils [None req-25f75b64-98ca-463b-89e7-2f9b39934491 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92057) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 12:51:14.437720 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=100 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:51:14.475948 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25f75b64-98ca-463b-89e7-2f9b39934491 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:51:14.477386 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25f75b64-98ca-463b-89e7-2f9b39934491 service neutron] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:51:14.495376 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 663/1323] 10.222.0.30 () {60 vars in 1365 bytes} [Mon May 23 12:51:14 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 311 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) May 23 12:51:16.766824 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ecdd094-2cb3-4d6a-99ab-62e012904a67 None None] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:51:16.767382 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ecdd094-2cb3-4d6a-99ab-62e012904a67 None None] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:51:16.767382 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ecdd094-2cb3-4d6a-99ab-62e012904a67 None None] PATH_INFO: `/` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:51:16.768753 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 661/1324] 10.222.0.30 () {54 vars in 926 bytes} [Mon May 23 12:51:16 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) May 23 12:51:16.774637 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee943a53-32a8-4f3e-85ae-626ffd5ae95f None None] REQUEST_METHOD: `POST` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:51:16.775064 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee943a53-32a8-4f3e-85ae-626ffd5ae95f None None] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:51:16.775202 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee943a53-32a8-4f3e-85ae-626ffd5ae95f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:51:16.844701 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.auth.core [None req-ee943a53-32a8-4f3e-85ae-626ffd5ae95f None None] MFA Rules not processed for user `0f7119f8fa5c4445977671140ded91a4`. Rule list: `[]` (Enabled: `True`). {{(pid=92057) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 12:51:16.845985 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.fernet_utils [None req-ee943a53-32a8-4f3e-85ae-626ffd5ae95f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92057) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 12:51:16.960362 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 664/1325] 10.222.0.30 () {58 vars in 1023 bytes} [Mon May 23 12:51:16 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 187 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) May 23 12:51:17.011261 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-703407bf-fa0f-4b9b-ba25-62d8a55e1f55 None None] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:51:17.011697 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-703407bf-fa0f-4b9b-ba25-62d8a55e1f55 None None] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:51:17.012089 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-703407bf-fa0f-4b9b-ba25-62d8a55e1f55 None None] PATH_INFO: `/` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:51:17.013544 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 662/1326] 10.222.0.30 () {54 vars in 964 bytes} [Mon May 23 12:51:17 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) May 23 12:51:17.022091 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb8fd357-88b3-4191-a523-6df3d093b384 None None] REQUEST_METHOD: `POST` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:51:17.024126 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb8fd357-88b3-4191-a523-6df3d093b384 None None] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:51:17.024126 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb8fd357-88b3-4191-a523-6df3d093b384 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:51:17.055346 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=100 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:51:17.104341 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.auth.core [None req-bb8fd357-88b3-4191-a523-6df3d093b384 None None] MFA Rules not processed for user `2eb83ca38e79449095f3a2f947bbb06a`. Rule list: `[]` (Enabled: `True`). {{(pid=92057) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 12:51:17.105558 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.fernet_utils [None req-bb8fd357-88b3-4191-a523-6df3d093b384 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92057) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 12:51:17.185716 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 665/1327] 10.222.0.30 () {58 vars in 1061 bytes} [Mon May 23 12:51:17 2022] POST /identity/v3/auth/tokens => generated 3111 bytes in 166 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) May 23 12:51:17.194856 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-703407bf-fa0f-4b9b-ba25-62d8a55e1f55 None None] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:51:17.196473 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.fernet_utils [None req-703407bf-fa0f-4b9b-ba25-62d8a55e1f55 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92056) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 12:51:17.323633 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-703407bf-fa0f-4b9b-ba25-62d8a55e1f55 None None] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:51:17.413254 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3db4bf54-dae1-49df-afc4-1c6a678c45bb service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2eb83ca38e79449095f3a2f947bbb06a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9c389ae65aaa4542bb5a5c4fd0bacf5b', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:51:17.413918 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3db4bf54-dae1-49df-afc4-1c6a678c45bb service neutron] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:51:17.414201 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3db4bf54-dae1-49df-afc4-1c6a678c45bb service neutron] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:51:17.414476 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3db4bf54-dae1-49df-afc4-1c6a678c45bb service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:51:17.416097 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.fernet_utils [None req-3db4bf54-dae1-49df-afc4-1c6a678c45bb service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92056) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 12:51:17.434888 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=100 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:51:17.539121 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3db4bf54-dae1-49df-afc4-1c6a678c45bb service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:51:17.540570 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3db4bf54-dae1-49df-afc4-1c6a678c45bb service neutron] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:51:17.560231 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 663/1328] 10.222.0.30 () {60 vars in 1365 bytes} [Mon May 23 12:51:17 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 366 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) May 23 12:51:20.056543 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a3949a5-4243-42dc-b797-cb97ee116dbd None None] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:51:20.057127 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a3949a5-4243-42dc-b797-cb97ee116dbd None None] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:51:20.057127 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a3949a5-4243-42dc-b797-cb97ee116dbd None None] PATH_INFO: `/` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:51:20.058527 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 666/1329] 10.222.0.30 () {54 vars in 926 bytes} [Mon May 23 12:51:20 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) May 23 12:51:20.064008 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-234574c4-2a05-49d0-8d47-befd1fc2f098 None None] REQUEST_METHOD: `POST` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:51:20.064444 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-234574c4-2a05-49d0-8d47-befd1fc2f098 None None] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:51:20.064610 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-234574c4-2a05-49d0-8d47-befd1fc2f098 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:51:20.132387 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.auth.core [None req-234574c4-2a05-49d0-8d47-befd1fc2f098 None None] MFA Rules not processed for user `b3478ea957234bb99fde27e6c3d0dfd3`. Rule list: `[]` (Enabled: `True`). {{(pid=92056) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 12:51:20.133580 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.fernet_utils [None req-234574c4-2a05-49d0-8d47-befd1fc2f098 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92056) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 12:51:20.198873 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=100 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:51:20.230061 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 664/1330] 10.222.0.30 () {58 vars in 1023 bytes} [Mon May 23 12:51:20 2022] POST /identity/v3/auth/tokens => generated 3233 bytes in 167 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) May 23 12:51:20.267650 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a3949a5-4243-42dc-b797-cb97ee116dbd None None] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:51:20.269199 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.fernet_utils [None req-0a3949a5-4243-42dc-b797-cb97ee116dbd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92057) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 12:51:20.377460 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a3949a5-4243-42dc-b797-cb97ee116dbd None None] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:51:20.398491 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=100 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:51:20.455328 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e0753bd-4a2c-4cc7-8ff9-d76f5320f6f6 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2eb83ca38e79449095f3a2f947bbb06a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9c389ae65aaa4542bb5a5c4fd0bacf5b', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:51:20.456087 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e0753bd-4a2c-4cc7-8ff9-d76f5320f6f6 service neutron] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:51:20.456389 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e0753bd-4a2c-4cc7-8ff9-d76f5320f6f6 service neutron] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:51:20.456730 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e0753bd-4a2c-4cc7-8ff9-d76f5320f6f6 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:51:20.458288 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.fernet_utils [None req-7e0753bd-4a2c-4cc7-8ff9-d76f5320f6f6 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92057) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 12:51:20.573699 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e0753bd-4a2c-4cc7-8ff9-d76f5320f6f6 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:51:20.575345 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e0753bd-4a2c-4cc7-8ff9-d76f5320f6f6 service neutron] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:51:20.594024 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 667/1331] 10.222.0.30 () {60 vars in 1365 bytes} [Mon May 23 12:51:20 2022] GET /identity/v3/auth/tokens => generated 3233 bytes in 327 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) May 23 12:51:23.214633 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-caa52ae0-3827-4f7d-853d-f072a6b0f2f7 None None] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:51:23.215131 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-caa52ae0-3827-4f7d-853d-f072a6b0f2f7 None None] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:51:23.215131 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-caa52ae0-3827-4f7d-853d-f072a6b0f2f7 None None] PATH_INFO: `/` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:51:23.216472 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 665/1332] 10.222.0.30 () {54 vars in 926 bytes} [Mon May 23 12:51:23 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) May 23 12:51:23.222117 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3348ab84-54d1-421e-9c9c-3999caf0b7f4 None None] REQUEST_METHOD: `POST` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:51:23.222706 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3348ab84-54d1-421e-9c9c-3999caf0b7f4 None None] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:51:23.222820 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3348ab84-54d1-421e-9c9c-3999caf0b7f4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:51:23.288048 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=100 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:51:23.290997 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.auth.core [None req-3348ab84-54d1-421e-9c9c-3999caf0b7f4 None None] MFA Rules not processed for user `0f7119f8fa5c4445977671140ded91a4`. Rule list: `[]` (Enabled: `True`). {{(pid=92057) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 12:51:23.292291 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.fernet_utils [None req-3348ab84-54d1-421e-9c9c-3999caf0b7f4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92057) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 12:51:23.395751 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 668/1333] 10.222.0.30 () {58 vars in 1023 bytes} [Mon May 23 12:51:23 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 175 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) May 23 12:51:23.432253 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-caa52ae0-3827-4f7d-853d-f072a6b0f2f7 None None] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:51:23.433779 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.fernet_utils [None req-caa52ae0-3827-4f7d-853d-f072a6b0f2f7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92056) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 12:51:23.542269 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-caa52ae0-3827-4f7d-853d-f072a6b0f2f7 None None] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:51:23.621441 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69d774e9-bf59-4af6-b742-ed32bb4bde37 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2eb83ca38e79449095f3a2f947bbb06a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9c389ae65aaa4542bb5a5c4fd0bacf5b', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:51:23.622121 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69d774e9-bf59-4af6-b742-ed32bb4bde37 service neutron] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:51:23.622410 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69d774e9-bf59-4af6-b742-ed32bb4bde37 service neutron] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:51:23.622692 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69d774e9-bf59-4af6-b742-ed32bb4bde37 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:51:23.624320 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.fernet_utils [None req-69d774e9-bf59-4af6-b742-ed32bb4bde37 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92056) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 12:51:23.651687 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=100 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:51:23.734045 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69d774e9-bf59-4af6-b742-ed32bb4bde37 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:51:23.735435 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69d774e9-bf59-4af6-b742-ed32bb4bde37 service neutron] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:51:23.753687 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 666/1334] 10.222.0.30 () {60 vars in 1365 bytes} [Mon May 23 12:51:23 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 322 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) May 23 12:51:26.024534 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45538ea8-16b5-42be-9c49-d471a7e31539 None None] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:51:26.025089 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45538ea8-16b5-42be-9c49-d471a7e31539 None None] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:51:26.025089 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45538ea8-16b5-42be-9c49-d471a7e31539 None None] PATH_INFO: `/` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:51:26.026483 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 669/1335] 10.222.0.30 () {54 vars in 926 bytes} [Mon May 23 12:51:26 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) May 23 12:51:26.031974 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a0b471c-f6e4-4fa3-89d9-04df214de63c None None] REQUEST_METHOD: `POST` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:51:26.032395 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a0b471c-f6e4-4fa3-89d9-04df214de63c None None] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:51:26.032541 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a0b471c-f6e4-4fa3-89d9-04df214de63c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:51:26.097862 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.auth.core [None req-9a0b471c-f6e4-4fa3-89d9-04df214de63c None None] MFA Rules not processed for user `0f7119f8fa5c4445977671140ded91a4`. Rule list: `[]` (Enabled: `True`). {{(pid=92056) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 12:51:26.099087 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.fernet_utils [None req-9a0b471c-f6e4-4fa3-89d9-04df214de63c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92056) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 12:51:26.176939 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=100 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:51:26.199880 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 667/1336] 10.222.0.30 () {58 vars in 1023 bytes} [Mon May 23 12:51:26 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 169 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) May 23 12:51:26.240557 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45538ea8-16b5-42be-9c49-d471a7e31539 None None] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:51:26.242144 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.fernet_utils [None req-45538ea8-16b5-42be-9c49-d471a7e31539 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92057) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 12:51:26.343204 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45538ea8-16b5-42be-9c49-d471a7e31539 None None] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:51:26.373419 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=100 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:51:26.421045 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ee31293-a675-4834-8bce-945eb1c253a8 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2eb83ca38e79449095f3a2f947bbb06a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9c389ae65aaa4542bb5a5c4fd0bacf5b', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:51:26.421688 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ee31293-a675-4834-8bce-945eb1c253a8 service neutron] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:51:26.422001 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ee31293-a675-4834-8bce-945eb1c253a8 service neutron] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:51:26.422280 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ee31293-a675-4834-8bce-945eb1c253a8 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:51:26.423908 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.fernet_utils [None req-4ee31293-a675-4834-8bce-945eb1c253a8 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92057) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 12:51:26.533892 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ee31293-a675-4834-8bce-945eb1c253a8 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:51:26.536357 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ee31293-a675-4834-8bce-945eb1c253a8 service neutron] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:51:26.554893 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 670/1337] 10.222.0.30 () {60 vars in 1365 bytes} [Mon May 23 12:51:26 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 315 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) May 23 12:51:29.335079 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2dfb5aec-8685-4c11-8226-361c10e98f6e None None] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:51:29.335686 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2dfb5aec-8685-4c11-8226-361c10e98f6e None None] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:51:29.335686 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2dfb5aec-8685-4c11-8226-361c10e98f6e None None] PATH_INFO: `/` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:51:29.337015 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 668/1338] 10.222.0.30 () {54 vars in 926 bytes} [Mon May 23 12:51:29 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) May 23 12:51:29.342100 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1783ff14-5ad0-46fc-84a5-a273e80a700c None None] REQUEST_METHOD: `POST` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:51:29.342485 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1783ff14-5ad0-46fc-84a5-a273e80a700c None None] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:51:29.342664 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1783ff14-5ad0-46fc-84a5-a273e80a700c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:51:29.415663 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.auth.core [None req-1783ff14-5ad0-46fc-84a5-a273e80a700c None None] MFA Rules not processed for user `0f7119f8fa5c4445977671140ded91a4`. Rule list: `[]` (Enabled: `True`). {{(pid=92057) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 12:51:29.416884 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.fernet_utils [None req-1783ff14-5ad0-46fc-84a5-a273e80a700c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92057) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 12:51:29.426109 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=100 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:51:29.516822 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 671/1339] 10.222.0.30 () {58 vars in 1023 bytes} [Mon May 23 12:51:29 2022] POST /identity/v3/auth/tokens => generated 3228 bytes in 176 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) May 23 12:51:29.552871 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2dfb5aec-8685-4c11-8226-361c10e98f6e None None] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:51:29.554409 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.fernet_utils [None req-2dfb5aec-8685-4c11-8226-361c10e98f6e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92056) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 12:51:29.657350 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2dfb5aec-8685-4c11-8226-361c10e98f6e None None] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:51:29.729175 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82650bbf-1e1d-4e71-b6f4-81aa98df0754 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2eb83ca38e79449095f3a2f947bbb06a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9c389ae65aaa4542bb5a5c4fd0bacf5b', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:51:29.729814 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82650bbf-1e1d-4e71-b6f4-81aa98df0754 service neutron] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:51:29.730125 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82650bbf-1e1d-4e71-b6f4-81aa98df0754 service neutron] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:51:29.730402 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82650bbf-1e1d-4e71-b6f4-81aa98df0754 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:51:29.732025 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.fernet_utils [None req-82650bbf-1e1d-4e71-b6f4-81aa98df0754 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92056) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 12:51:29.769484 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=100 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:51:29.842853 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82650bbf-1e1d-4e71-b6f4-81aa98df0754 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:51:29.844288 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82650bbf-1e1d-4e71-b6f4-81aa98df0754 service neutron] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:51:29.862957 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 669/1340] 10.222.0.30 () {60 vars in 1365 bytes} [Mon May 23 12:51:29 2022] GET /identity/v3/auth/tokens => generated 3228 bytes in 311 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) May 23 12:51:34.622536 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d361a885-b8ab-4f75-a0fb-53fafbba1bc6 None None] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:51:34.623058 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d361a885-b8ab-4f75-a0fb-53fafbba1bc6 None None] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:51:34.623152 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d361a885-b8ab-4f75-a0fb-53fafbba1bc6 None None] PATH_INFO: `/` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:51:34.624649 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 672/1341] 10.222.0.30 () {54 vars in 926 bytes} [Mon May 23 12:51:34 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) May 23 12:51:34.630534 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f2aba59-0a5e-4c9a-be87-a6518977fac1 None None] REQUEST_METHOD: `POST` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:51:34.631052 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f2aba59-0a5e-4c9a-be87-a6518977fac1 None None] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:51:34.631159 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f2aba59-0a5e-4c9a-be87-a6518977fac1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:51:34.708485 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.auth.core [None req-6f2aba59-0a5e-4c9a-be87-a6518977fac1 None None] MFA Rules not processed for user `0f7119f8fa5c4445977671140ded91a4`. Rule list: `[]` (Enabled: `True`). {{(pid=92056) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 12:51:34.709722 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.fernet_utils [None req-6f2aba59-0a5e-4c9a-be87-a6518977fac1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92056) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 12:51:34.803848 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=100 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:51:34.817303 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 670/1342] 10.222.0.30 () {58 vars in 1023 bytes} [Mon May 23 12:51:34 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 188 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) May 23 12:51:34.856234 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d361a885-b8ab-4f75-a0fb-53fafbba1bc6 None None] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:51:34.858354 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.fernet_utils [None req-d361a885-b8ab-4f75-a0fb-53fafbba1bc6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92057) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 12:51:34.953048 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d361a885-b8ab-4f75-a0fb-53fafbba1bc6 None None] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:51:34.957745 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ebf727c1-044b-40c4-aa02-86113729ada0 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2eb83ca38e79449095f3a2f947bbb06a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9c389ae65aaa4542bb5a5c4fd0bacf5b', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:51:34.958423 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebf727c1-044b-40c4-aa02-86113729ada0 service neutron] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:51:34.963861 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebf727c1-044b-40c4-aa02-86113729ada0 service neutron] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:51:34.963861 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebf727c1-044b-40c4-aa02-86113729ada0 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:51:34.968025 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ebf727c1-044b-40c4-aa02-86113729ada0 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:51:34.969467 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ebf727c1-044b-40c4-aa02-86113729ada0 service neutron] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:51:34.979372 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 673/1343] 10.222.0.30 () {60 vars in 1365 bytes} [Mon May 23 12:51:34 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 124 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) May 23 12:51:38.152437 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0add546-7b6d-48f7-88c4-6baa61ceb1e0 None None] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:51:38.152993 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0add546-7b6d-48f7-88c4-6baa61ceb1e0 None None] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:51:38.152993 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0add546-7b6d-48f7-88c4-6baa61ceb1e0 None None] PATH_INFO: `/` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:51:38.154405 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 671/1344] 10.222.0.30 () {54 vars in 926 bytes} [Mon May 23 12:51:38 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) May 23 12:51:38.159691 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3ff14a5-e72a-474a-9cf4-d95e1dfba08e None None] REQUEST_METHOD: `POST` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:51:38.160113 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3ff14a5-e72a-474a-9cf4-d95e1dfba08e None None] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:51:38.160359 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3ff14a5-e72a-474a-9cf4-d95e1dfba08e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:51:38.187875 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.auth.core [None req-c3ff14a5-e72a-474a-9cf4-d95e1dfba08e None None] MFA Rules not processed for user `0f7119f8fa5c4445977671140ded91a4`. Rule list: `[]` (Enabled: `True`). {{(pid=92057) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 12:51:38.189043 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.fernet_utils [None req-c3ff14a5-e72a-474a-9cf4-d95e1dfba08e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92057) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 12:51:38.217611 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=100 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:51:38.268142 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 674/1345] 10.222.0.30 () {58 vars in 1023 bytes} [Mon May 23 12:51:38 2022] POST /identity/v3/auth/tokens => generated 3228 bytes in 109 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) May 23 12:51:38.305207 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0add546-7b6d-48f7-88c4-6baa61ceb1e0 None None] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:51:38.316518 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0add546-7b6d-48f7-88c4-6baa61ceb1e0 None None] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:51:38.320021 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f83442b-7792-47e5-a89f-0e3f7be1100b service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2eb83ca38e79449095f3a2f947bbb06a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9c389ae65aaa4542bb5a5c4fd0bacf5b', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:51:38.320651 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f83442b-7792-47e5-a89f-0e3f7be1100b service neutron] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:51:38.320927 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f83442b-7792-47e5-a89f-0e3f7be1100b service neutron] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:51:38.321197 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f83442b-7792-47e5-a89f-0e3f7be1100b service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:51:38.329962 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f83442b-7792-47e5-a89f-0e3f7be1100b service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:51:38.331416 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f83442b-7792-47e5-a89f-0e3f7be1100b service neutron] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:51:38.340663 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 672/1346] 10.222.0.30 () {60 vars in 1365 bytes} [Mon May 23 12:51:38 2022] GET /identity/v3/auth/tokens => generated 3228 bytes in 36 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) May 23 12:51:42.994486 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-564edf30-e0ca-4051-80d8-b32aba99b97b None None] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:51:42.995049 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-564edf30-e0ca-4051-80d8-b32aba99b97b None None] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:51:42.995049 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-564edf30-e0ca-4051-80d8-b32aba99b97b None None] PATH_INFO: `/` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:51:42.996365 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 675/1347] 10.222.0.30 () {54 vars in 926 bytes} [Mon May 23 12:51:42 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) May 23 12:51:43.001645 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fc3e408-a2d8-4f9e-b1a2-0a5beccd60d0 None None] REQUEST_METHOD: `POST` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:51:43.002744 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fc3e408-a2d8-4f9e-b1a2-0a5beccd60d0 None None] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:51:43.002903 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fc3e408-a2d8-4f9e-b1a2-0a5beccd60d0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:51:43.021026 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.auth.core [None req-7fc3e408-a2d8-4f9e-b1a2-0a5beccd60d0 None None] MFA Rules not processed for user `0f7119f8fa5c4445977671140ded91a4`. Rule list: `[]` (Enabled: `True`). {{(pid=92056) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 12:51:43.022279 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.fernet_utils [None req-7fc3e408-a2d8-4f9e-b1a2-0a5beccd60d0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92056) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 12:51:43.030397 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 673/1348] 10.222.0.30 () {58 vars in 1023 bytes} [Mon May 23 12:51:43 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 30 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) May 23 12:51:43.067770 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-564edf30-e0ca-4051-80d8-b32aba99b97b None None] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:51:43.078258 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-564edf30-e0ca-4051-80d8-b32aba99b97b None None] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:51:43.082352 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f689a78-db0e-4040-b178-99bd9f3df582 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2eb83ca38e79449095f3a2f947bbb06a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9c389ae65aaa4542bb5a5c4fd0bacf5b', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:51:43.082977 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f689a78-db0e-4040-b178-99bd9f3df582 service neutron] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:51:43.083258 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f689a78-db0e-4040-b178-99bd9f3df582 service neutron] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:51:43.083531 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f689a78-db0e-4040-b178-99bd9f3df582 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:51:43.092376 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f689a78-db0e-4040-b178-99bd9f3df582 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:51:43.093805 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f689a78-db0e-4040-b178-99bd9f3df582 service neutron] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:51:43.103168 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 676/1349] 10.222.0.30 () {60 vars in 1365 bytes} [Mon May 23 12:51:43 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 36 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) May 23 12:51:48.455712 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c23504f6-aece-44d3-af13-5f623874673b None None] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:51:48.456232 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c23504f6-aece-44d3-af13-5f623874673b None None] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:51:48.456353 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c23504f6-aece-44d3-af13-5f623874673b None None] PATH_INFO: `/` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:51:48.462090 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 674/1350] 10.222.0.30 () {54 vars in 921 bytes} [Mon May 23 12:51:48 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) May 23 12:51:48.467288 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e6c78f5-0dfa-4dce-a210-fa7b1dae3d40 None None] REQUEST_METHOD: `POST` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:51:48.467744 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e6c78f5-0dfa-4dce-a210-fa7b1dae3d40 None None] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:51:48.467857 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e6c78f5-0dfa-4dce-a210-fa7b1dae3d40 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:51:48.544371 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.auth.core [None req-2e6c78f5-0dfa-4dce-a210-fa7b1dae3d40 None None] MFA Rules not processed for user `d5fb51f8d0fa46439cb29666970b885b`. Rule list: `[]` (Enabled: `True`). {{(pid=92057) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 12:51:48.547867 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.fernet_utils [None req-2e6c78f5-0dfa-4dce-a210-fa7b1dae3d40 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92057) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 12:51:48.630334 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 677/1351] 10.222.0.30 () {58 vars in 1018 bytes} [Mon May 23 12:51:48 2022] POST /identity/v3/auth/tokens => generated 3298 bytes in 164 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) May 23 12:51:53.015192 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=13 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:51:53.104421 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbbda4ec-3645-49d6-8be0-b5a37acdf0ba None None] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:51:53.105078 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbbda4ec-3645-49d6-8be0-b5a37acdf0ba None None] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:51:53.105449 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbbda4ec-3645-49d6-8be0-b5a37acdf0ba None None] PATH_INFO: `/` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:51:53.107079 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 675/1352] 10.222.0.30 () {54 vars in 921 bytes} [Mon May 23 12:51:53 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) May 23 12:51:53.113435 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31d05d7a-40d9-4956-8c71-9fd95cbfb60c None None] REQUEST_METHOD: `POST` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:51:53.113965 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31d05d7a-40d9-4956-8c71-9fd95cbfb60c None None] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:51:53.114076 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31d05d7a-40d9-4956-8c71-9fd95cbfb60c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:51:53.132490 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.auth.core [None req-31d05d7a-40d9-4956-8c71-9fd95cbfb60c None None] MFA Rules not processed for user `d5fb51f8d0fa46439cb29666970b885b`. Rule list: `[]` (Enabled: `True`). {{(pid=92057) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 12:51:53.133668 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.fernet_utils [None req-31d05d7a-40d9-4956-8c71-9fd95cbfb60c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92057) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 12:51:53.144268 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 678/1353] 10.222.0.30 () {58 vars in 1018 bytes} [Mon May 23 12:51:53 2022] POST /identity/v3/auth/tokens => generated 3298 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) May 23 12:51:54.441032 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5bd669b-8acd-44ea-9ce3-240741b1abf8 None None] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:51:54.441562 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5bd669b-8acd-44ea-9ce3-240741b1abf8 None None] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:51:54.441677 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5bd669b-8acd-44ea-9ce3-240741b1abf8 None None] PATH_INFO: `/` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:51:54.443179 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 676/1354] 10.222.0.30 () {54 vars in 921 bytes} [Mon May 23 12:51:54 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) May 23 12:51:54.448233 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fc32b29-ea68-4d26-adf8-a5a0c27bc80c None None] REQUEST_METHOD: `POST` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:51:54.448660 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fc32b29-ea68-4d26-adf8-a5a0c27bc80c None None] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:51:54.448785 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fc32b29-ea68-4d26-adf8-a5a0c27bc80c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:51:54.466848 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.auth.core [None req-7fc32b29-ea68-4d26-adf8-a5a0c27bc80c None None] MFA Rules not processed for user `d5fb51f8d0fa46439cb29666970b885b`. Rule list: `[]` (Enabled: `True`). {{(pid=92057) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 12:51:54.468020 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.fernet_utils [None req-7fc32b29-ea68-4d26-adf8-a5a0c27bc80c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92057) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 12:51:54.477159 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 679/1355] 10.222.0.30 () {58 vars in 1018 bytes} [Mon May 23 12:51:54 2022] POST /identity/v3/auth/tokens => generated 3298 bytes in 30 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) May 23 12:51:58.470566 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13c79cf3-2572-4b86-aa4a-2c9f293e3196 None None] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:51:58.471110 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13c79cf3-2572-4b86-aa4a-2c9f293e3196 None None] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:51:58.471524 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13c79cf3-2572-4b86-aa4a-2c9f293e3196 None None] PATH_INFO: `/` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:51:58.473061 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 677/1356] 10.222.0.30 () {54 vars in 926 bytes} [Mon May 23 12:51:58 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) May 23 12:51:58.478511 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd514156-b660-4c25-919c-aee04338ede6 None None] REQUEST_METHOD: `POST` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:51:58.479000 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd514156-b660-4c25-919c-aee04338ede6 None None] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:51:58.479102 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd514156-b660-4c25-919c-aee04338ede6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:51:58.496995 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.auth.core [None req-dd514156-b660-4c25-919c-aee04338ede6 None None] MFA Rules not processed for user `0f7119f8fa5c4445977671140ded91a4`. Rule list: `[]` (Enabled: `True`). {{(pid=92057) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 12:51:58.498331 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.fernet_utils [None req-dd514156-b660-4c25-919c-aee04338ede6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92057) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 12:51:58.506295 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 680/1357] 10.222.0.30 () {58 vars in 1023 bytes} [Mon May 23 12:51:58 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 29 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) May 23 12:51:58.563172 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f765e949-b409-41b8-8bec-51a8cc7c9370 None None] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:51:58.563763 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f765e949-b409-41b8-8bec-51a8cc7c9370 None None] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:51:58.564405 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f765e949-b409-41b8-8bec-51a8cc7c9370 None None] PATH_INFO: `/` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:51:58.566698 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 678/1358] 10.222.0.30 () {54 vars in 961 bytes} [Mon May 23 12:51:58 2022] GET /identity => generated 270 bytes in 8 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) May 23 12:51:58.574257 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4d59750-a149-4e8c-906d-d09634baa3a4 None None] REQUEST_METHOD: `POST` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:51:58.574913 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4d59750-a149-4e8c-906d-d09634baa3a4 None None] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:51:58.575693 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4d59750-a149-4e8c-906d-d09634baa3a4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:51:58.621353 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=100 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:51:58.631253 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.auth.core [None req-d4d59750-a149-4e8c-906d-d09634baa3a4 None None] MFA Rules not processed for user `cbe9931d94d44cc7ace0e83a7e5015fc`. Rule list: `[]` (Enabled: `True`). {{(pid=92057) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 12:51:58.632837 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.fernet_utils [None req-d4d59750-a149-4e8c-906d-d09634baa3a4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92057) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 12:51:58.730154 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 681/1359] 10.222.0.30 () {58 vars in 1058 bytes} [Mon May 23 12:51:58 2022] POST /identity/v3/auth/tokens => generated 3362 bytes in 157 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) May 23 12:51:58.738630 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f765e949-b409-41b8-8bec-51a8cc7c9370 None None] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:51:58.753983 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f765e949-b409-41b8-8bec-51a8cc7c9370 None None] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:51:58.759604 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-310d7dc0-1b9b-4e00-9de5-fceeb64f5569 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cbe9931d94d44cc7ace0e83a7e5015fc', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9c389ae65aaa4542bb5a5c4fd0bacf5b', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'member', 'reader', 'service', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:51:58.760246 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-310d7dc0-1b9b-4e00-9de5-fceeb64f5569 service nova] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:51:58.766352 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-310d7dc0-1b9b-4e00-9de5-fceeb64f5569 service nova] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:51:58.766352 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-310d7dc0-1b9b-4e00-9de5-fceeb64f5569 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:51:58.770699 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-310d7dc0-1b9b-4e00-9de5-fceeb64f5569 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:51:58.772318 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-310d7dc0-1b9b-4e00-9de5-fceeb64f5569 service nova] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:51:58.797194 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 679/1360] 10.222.0.30 () {60 vars in 1365 bytes} [Mon May 23 12:51:58 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 59 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) May 23 12:52:01.244122 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3891ba8d-7a78-4eff-bbe4-8cc63842a0e4 None None] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:52:01.244631 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3891ba8d-7a78-4eff-bbe4-8cc63842a0e4 None None] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:52:01.244631 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3891ba8d-7a78-4eff-bbe4-8cc63842a0e4 None None] PATH_INFO: `/` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:52:01.246119 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 682/1361] 10.222.0.30 () {54 vars in 926 bytes} [Mon May 23 12:52:01 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) May 23 12:52:01.251871 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52e44ae7-7058-4144-8282-3bc1a122072c None None] REQUEST_METHOD: `POST` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:52:01.252283 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52e44ae7-7058-4144-8282-3bc1a122072c None None] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:52:01.252426 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52e44ae7-7058-4144-8282-3bc1a122072c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:52:01.270769 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.auth.core [None req-52e44ae7-7058-4144-8282-3bc1a122072c None None] MFA Rules not processed for user `0f7119f8fa5c4445977671140ded91a4`. Rule list: `[]` (Enabled: `True`). {{(pid=92056) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 12:52:01.272178 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.fernet_utils [None req-52e44ae7-7058-4144-8282-3bc1a122072c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92056) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 12:52:01.280271 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 680/1362] 10.222.0.30 () {58 vars in 1023 bytes} [Mon May 23 12:52:01 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 30 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) May 23 12:52:01.324793 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3891ba8d-7a78-4eff-bbe4-8cc63842a0e4 None None] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:52:01.340373 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3891ba8d-7a78-4eff-bbe4-8cc63842a0e4 None None] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:52:01.346373 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0708cbc6-bc7c-4dc5-9b3c-764a2c6eab04 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cbe9931d94d44cc7ace0e83a7e5015fc', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9c389ae65aaa4542bb5a5c4fd0bacf5b', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'member', 'reader', 'service', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:52:01.347202 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0708cbc6-bc7c-4dc5-9b3c-764a2c6eab04 service nova] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:52:01.347631 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0708cbc6-bc7c-4dc5-9b3c-764a2c6eab04 service nova] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:52:01.348054 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0708cbc6-bc7c-4dc5-9b3c-764a2c6eab04 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:52:01.357069 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0708cbc6-bc7c-4dc5-9b3c-764a2c6eab04 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:52:01.358827 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0708cbc6-bc7c-4dc5-9b3c-764a2c6eab04 service nova] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:52:01.369671 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 683/1363] 10.222.0.30 () {60 vars in 1365 bytes} [Mon May 23 12:52:01 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 45 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) May 23 12:52:04.116720 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbb4b140-1064-4200-8694-610288f0f5c4 None None] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:52:04.117248 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbb4b140-1064-4200-8694-610288f0f5c4 None None] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:52:04.117360 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbb4b140-1064-4200-8694-610288f0f5c4 None None] PATH_INFO: `/` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:52:04.119084 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 681/1364] 10.222.0.30 () {54 vars in 926 bytes} [Mon May 23 12:52:04 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) May 23 12:52:04.124656 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0133f94f-ea80-49c9-9f1f-b8c89bb79604 None None] REQUEST_METHOD: `POST` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:52:04.125456 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0133f94f-ea80-49c9-9f1f-b8c89bb79604 None None] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:52:04.126001 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0133f94f-ea80-49c9-9f1f-b8c89bb79604 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:52:04.143925 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.auth.core [None req-0133f94f-ea80-49c9-9f1f-b8c89bb79604 None None] MFA Rules not processed for user `0f7119f8fa5c4445977671140ded91a4`. Rule list: `[]` (Enabled: `True`). {{(pid=92057) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 12:52:04.145124 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.fernet_utils [None req-0133f94f-ea80-49c9-9f1f-b8c89bb79604 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92057) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 12:52:04.153301 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 684/1365] 10.222.0.30 () {58 vars in 1023 bytes} [Mon May 23 12:52:04 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 30 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) May 23 12:52:04.200855 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af1c7dc4-fad8-4402-94bd-1b3893683854 None None] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:52:04.201441 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af1c7dc4-fad8-4402-94bd-1b3893683854 None None] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:52:04.201790 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af1c7dc4-fad8-4402-94bd-1b3893683854 None None] PATH_INFO: `/` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:52:04.203628 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 682/1366] 10.222.0.30 () {54 vars in 961 bytes} [Mon May 23 12:52:04 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) May 23 12:52:04.209469 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47ceb079-6022-4bf8-be34-0fb4469f0496 None None] REQUEST_METHOD: `POST` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:52:04.209835 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47ceb079-6022-4bf8-be34-0fb4469f0496 None None] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:52:04.210161 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47ceb079-6022-4bf8-be34-0fb4469f0496 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:52:04.228936 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.auth.core [None req-47ceb079-6022-4bf8-be34-0fb4469f0496 None None] MFA Rules not processed for user `cbe9931d94d44cc7ace0e83a7e5015fc`. Rule list: `[]` (Enabled: `True`). {{(pid=92057) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 12:52:04.230138 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.fernet_utils [None req-47ceb079-6022-4bf8-be34-0fb4469f0496 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92057) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 12:52:04.240556 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 685/1367] 10.222.0.30 () {58 vars in 1058 bytes} [Mon May 23 12:52:04 2022] POST /identity/v3/auth/tokens => generated 3362 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) May 23 12:52:04.248511 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af1c7dc4-fad8-4402-94bd-1b3893683854 None None] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:52:04.263506 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af1c7dc4-fad8-4402-94bd-1b3893683854 None None] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:52:04.269482 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13f130de-4c3b-4150-92e6-a2ca017a0ef3 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cbe9931d94d44cc7ace0e83a7e5015fc', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9c389ae65aaa4542bb5a5c4fd0bacf5b', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'member', 'reader', 'service', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:52:04.270148 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13f130de-4c3b-4150-92e6-a2ca017a0ef3 service nova] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:52:04.270464 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13f130de-4c3b-4150-92e6-a2ca017a0ef3 service nova] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:52:04.270740 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13f130de-4c3b-4150-92e6-a2ca017a0ef3 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:52:04.278970 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13f130de-4c3b-4150-92e6-a2ca017a0ef3 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:52:04.280541 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13f130de-4c3b-4150-92e6-a2ca017a0ef3 service nova] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:52:04.290141 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 683/1368] 10.222.0.30 () {60 vars in 1365 bytes} [Mon May 23 12:52:04 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 42 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) May 23 12:52:06.741737 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b62d01d6-912f-40e5-8dd3-f79ea7978c1e None None] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:52:06.742332 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b62d01d6-912f-40e5-8dd3-f79ea7978c1e None None] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:52:06.742332 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b62d01d6-912f-40e5-8dd3-f79ea7978c1e None None] PATH_INFO: `/` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:52:06.743687 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 686/1369] 10.222.0.30 () {54 vars in 926 bytes} [Mon May 23 12:52:06 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) May 23 12:52:06.749104 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-873e8af2-2e59-4f98-a322-490e9a2cfb17 None None] REQUEST_METHOD: `POST` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:52:06.749538 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-873e8af2-2e59-4f98-a322-490e9a2cfb17 None None] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:52:06.749665 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-873e8af2-2e59-4f98-a322-490e9a2cfb17 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:52:06.768534 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.auth.core [None req-873e8af2-2e59-4f98-a322-490e9a2cfb17 None None] MFA Rules not processed for user `0f7119f8fa5c4445977671140ded91a4`. Rule list: `[]` (Enabled: `True`). {{(pid=92056) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 12:52:06.769994 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.fernet_utils [None req-873e8af2-2e59-4f98-a322-490e9a2cfb17 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92056) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 12:52:06.783139 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 684/1370] 10.222.0.30 () {58 vars in 1023 bytes} [Mon May 23 12:52:06 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) May 23 12:52:06.822863 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b62d01d6-912f-40e5-8dd3-f79ea7978c1e None None] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:52:06.839155 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b62d01d6-912f-40e5-8dd3-f79ea7978c1e None None] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:52:06.846016 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e73cd24-5fdd-4624-9b8a-a74eb1e1dc0c service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cbe9931d94d44cc7ace0e83a7e5015fc', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9c389ae65aaa4542bb5a5c4fd0bacf5b', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'member', 'reader', 'service', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:52:06.847133 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e73cd24-5fdd-4624-9b8a-a74eb1e1dc0c service nova] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:52:06.847548 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e73cd24-5fdd-4624-9b8a-a74eb1e1dc0c service nova] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:52:06.847931 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e73cd24-5fdd-4624-9b8a-a74eb1e1dc0c service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:52:06.858091 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e73cd24-5fdd-4624-9b8a-a74eb1e1dc0c service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:52:06.859939 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e73cd24-5fdd-4624-9b8a-a74eb1e1dc0c service nova] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:52:06.872125 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 687/1371] 10.222.0.30 () {60 vars in 1365 bytes} [Mon May 23 12:52:06 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 50 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) May 23 12:52:06.909252 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0465f3c3-0986-4247-9b41-7985bb1aad73 None None] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:52:06.909733 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0465f3c3-0986-4247-9b41-7985bb1aad73 None None] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:52:06.910129 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0465f3c3-0986-4247-9b41-7985bb1aad73 None None] PATH_INFO: `/` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:52:06.917082 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 685/1372] 10.222.0.30 () {54 vars in 926 bytes} [Mon May 23 12:52:06 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) May 23 12:52:06.917330 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3066237-b2ba-44b9-90c2-a5b99df037d1 None None] REQUEST_METHOD: `POST` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:52:06.917330 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3066237-b2ba-44b9-90c2-a5b99df037d1 None None] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:52:06.917518 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3066237-b2ba-44b9-90c2-a5b99df037d1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:52:06.935482 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.auth.core [None req-e3066237-b2ba-44b9-90c2-a5b99df037d1 None None] MFA Rules not processed for user `0f7119f8fa5c4445977671140ded91a4`. Rule list: `[]` (Enabled: `True`). {{(pid=92057) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 12:52:06.937353 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.fernet_utils [None req-e3066237-b2ba-44b9-90c2-a5b99df037d1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92057) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 12:52:06.944820 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 688/1373] 10.222.0.30 () {58 vars in 1023 bytes} [Mon May 23 12:52:06 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 29 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) May 23 12:52:07.258176 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a38e3977-1a96-4599-a7f2-0af2e0004497 None None] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:52:07.258454 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a38e3977-1a96-4599-a7f2-0af2e0004497 None None] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:52:07.258725 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a38e3977-1a96-4599-a7f2-0af2e0004497 None None] PATH_INFO: `/` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:52:07.260095 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 686/1374] 10.222.0.30 () {54 vars in 963 bytes} [Mon May 23 12:52:07 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) May 23 12:52:07.267536 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ae3d1eb-1db6-4d3b-9816-456447df6549 None None] REQUEST_METHOD: `POST` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:52:07.268564 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ae3d1eb-1db6-4d3b-9816-456447df6549 None None] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:52:07.268948 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ae3d1eb-1db6-4d3b-9816-456447df6549 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:52:07.322542 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.auth.core [None req-7ae3d1eb-1db6-4d3b-9816-456447df6549 None None] MFA Rules not processed for user `1666097f8ed545c6b0e4809c93e4775a`. Rule list: `[]` (Enabled: `True`). {{(pid=92057) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 12:52:07.323901 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.fernet_utils [None req-7ae3d1eb-1db6-4d3b-9816-456447df6549 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92057) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 12:52:07.394869 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 689/1375] 10.222.0.30 () {58 vars in 1060 bytes} [Mon May 23 12:52:07 2022] POST /identity/v3/auth/tokens => generated 3110 bytes in 129 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) May 23 12:52:07.397462 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a38e3977-1a96-4599-a7f2-0af2e0004497 None None] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:52:07.410054 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a38e3977-1a96-4599-a7f2-0af2e0004497 None None] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:52:07.413713 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09146edc-bf50-4ea4-b7cf-97582b1f083d service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1666097f8ed545c6b0e4809c93e4775a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9c389ae65aaa4542bb5a5c4fd0bacf5b', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:52:07.414392 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09146edc-bf50-4ea4-b7cf-97582b1f083d service cinder] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:52:07.414677 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09146edc-bf50-4ea4-b7cf-97582b1f083d service cinder] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:52:07.414952 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09146edc-bf50-4ea4-b7cf-97582b1f083d service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:52:07.425312 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09146edc-bf50-4ea4-b7cf-97582b1f083d service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:52:07.427058 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09146edc-bf50-4ea4-b7cf-97582b1f083d service cinder] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:52:07.437910 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 687/1376] 10.222.0.30 () {60 vars in 1365 bytes} [Mon May 23 12:52:07 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 41 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) May 23 12:52:09.262667 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-469ab196-ac1b-4493-9172-4b222857561c None None] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:52:09.263238 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-469ab196-ac1b-4493-9172-4b222857561c None None] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:52:09.263238 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-469ab196-ac1b-4493-9172-4b222857561c None None] PATH_INFO: `/` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:52:09.264621 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 690/1377] 10.222.0.30 () {54 vars in 926 bytes} [Mon May 23 12:52:09 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) May 23 12:52:09.270449 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a130378b-6d8e-470d-bb7f-9c37ce659c44 None None] REQUEST_METHOD: `POST` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:52:09.270996 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a130378b-6d8e-470d-bb7f-9c37ce659c44 None None] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:52:09.271093 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a130378b-6d8e-470d-bb7f-9c37ce659c44 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:52:09.290965 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.auth.core [None req-a130378b-6d8e-470d-bb7f-9c37ce659c44 None None] MFA Rules not processed for user `0f7119f8fa5c4445977671140ded91a4`. Rule list: `[]` (Enabled: `True`). {{(pid=92056) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 12:52:09.292192 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.fernet_utils [None req-a130378b-6d8e-470d-bb7f-9c37ce659c44 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92056) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 12:52:09.301090 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 688/1378] 10.222.0.30 () {58 vars in 1023 bytes} [Mon May 23 12:52:09 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) May 23 12:52:09.346754 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-469ab196-ac1b-4493-9172-4b222857561c None None] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:52:09.362173 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-469ab196-ac1b-4493-9172-4b222857561c None None] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:52:09.367980 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f01cd2f5-4ab9-4ae0-928c-44d74798988c service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cbe9931d94d44cc7ace0e83a7e5015fc', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9c389ae65aaa4542bb5a5c4fd0bacf5b', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'member', 'reader', 'service', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:52:09.368677 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f01cd2f5-4ab9-4ae0-928c-44d74798988c service nova] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:52:09.368966 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f01cd2f5-4ab9-4ae0-928c-44d74798988c service nova] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:52:09.369246 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f01cd2f5-4ab9-4ae0-928c-44d74798988c service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:52:09.379012 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f01cd2f5-4ab9-4ae0-928c-44d74798988c service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:52:09.380659 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f01cd2f5-4ab9-4ae0-928c-44d74798988c service nova] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:52:09.385079 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=100 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:52:09.391675 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 691/1379] 10.222.0.30 () {60 vars in 1365 bytes} [Mon May 23 12:52:09 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 46 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) May 23 12:52:11.780245 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6799b66f-e8e6-4c68-8909-fc8e94158458 None None] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:52:11.780778 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6799b66f-e8e6-4c68-8909-fc8e94158458 None None] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:52:11.780778 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6799b66f-e8e6-4c68-8909-fc8e94158458 None None] PATH_INFO: `/` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:52:11.782735 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 689/1380] 10.222.0.30 () {54 vars in 926 bytes} [Mon May 23 12:52:11 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) May 23 12:52:11.788407 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b325354-3a30-4e2d-9d9e-cb5dd6b8fdd8 None None] REQUEST_METHOD: `POST` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:52:11.788853 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b325354-3a30-4e2d-9d9e-cb5dd6b8fdd8 None None] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:52:11.788973 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b325354-3a30-4e2d-9d9e-cb5dd6b8fdd8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:52:11.807963 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.auth.core [None req-0b325354-3a30-4e2d-9d9e-cb5dd6b8fdd8 None None] MFA Rules not processed for user `0f7119f8fa5c4445977671140ded91a4`. Rule list: `[]` (Enabled: `True`). {{(pid=92057) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 12:52:11.809273 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.fernet_utils [None req-0b325354-3a30-4e2d-9d9e-cb5dd6b8fdd8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92057) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 12:52:11.817228 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 692/1381] 10.222.0.30 () {58 vars in 1023 bytes} [Mon May 23 12:52:11 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 30 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) May 23 12:52:11.861987 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6799b66f-e8e6-4c68-8909-fc8e94158458 None None] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:52:11.876918 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6799b66f-e8e6-4c68-8909-fc8e94158458 None None] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:52:11.885455 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63a1a313-2541-4763-9b56-b0751674c198 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cbe9931d94d44cc7ace0e83a7e5015fc', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9c389ae65aaa4542bb5a5c4fd0bacf5b', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'member', 'reader', 'service', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:52:11.886176 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63a1a313-2541-4763-9b56-b0751674c198 service nova] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:52:11.886709 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63a1a313-2541-4763-9b56-b0751674c198 service nova] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:52:11.887231 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63a1a313-2541-4763-9b56-b0751674c198 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:52:11.897403 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63a1a313-2541-4763-9b56-b0751674c198 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:52:11.899267 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63a1a313-2541-4763-9b56-b0751674c198 service nova] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:52:11.911200 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 690/1382] 10.222.0.30 () {60 vars in 1365 bytes} [Mon May 23 12:52:11 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 49 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) May 23 12:52:14.521549 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6aee50c3-5dfa-42a9-b629-a2efbcf9ab7d None None] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:52:14.522646 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6aee50c3-5dfa-42a9-b629-a2efbcf9ab7d None None] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:52:14.523154 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6aee50c3-5dfa-42a9-b629-a2efbcf9ab7d None None] PATH_INFO: `/` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:52:14.525129 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 693/1383] 10.222.0.30 () {54 vars in 926 bytes} [Mon May 23 12:52:14 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) May 23 12:52:14.530750 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd00f66f-5931-4892-9831-a324e5e93954 None None] REQUEST_METHOD: `POST` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:52:14.531178 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd00f66f-5931-4892-9831-a324e5e93954 None None] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:52:14.531314 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd00f66f-5931-4892-9831-a324e5e93954 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:52:14.550760 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.auth.core [None req-fd00f66f-5931-4892-9831-a324e5e93954 None None] MFA Rules not processed for user `0f7119f8fa5c4445977671140ded91a4`. Rule list: `[]` (Enabled: `True`). {{(pid=92056) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 12:52:14.552177 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.fernet_utils [None req-fd00f66f-5931-4892-9831-a324e5e93954 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92056) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 12:52:14.560283 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 691/1384] 10.222.0.30 () {58 vars in 1023 bytes} [Mon May 23 12:52:14 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) May 23 12:52:14.601670 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6aee50c3-5dfa-42a9-b629-a2efbcf9ab7d None None] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:52:14.617027 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6aee50c3-5dfa-42a9-b629-a2efbcf9ab7d None None] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:52:14.622777 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e872b2cf-49ad-4203-a98f-2fab00b63c8b service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cbe9931d94d44cc7ace0e83a7e5015fc', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9c389ae65aaa4542bb5a5c4fd0bacf5b', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'member', 'reader', 'service', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:52:14.623449 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e872b2cf-49ad-4203-a98f-2fab00b63c8b service nova] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:52:14.623736 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e872b2cf-49ad-4203-a98f-2fab00b63c8b service nova] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:52:14.624014 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e872b2cf-49ad-4203-a98f-2fab00b63c8b service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:52:14.633535 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e872b2cf-49ad-4203-a98f-2fab00b63c8b service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:52:14.635168 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e872b2cf-49ad-4203-a98f-2fab00b63c8b service nova] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:52:14.648032 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 694/1385] 10.222.0.30 () {60 vars in 1365 bytes} [Mon May 23 12:52:14 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 47 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) May 23 12:52:16.867784 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0e0d504-42be-49b5-be81-10c9910d0dd7 None None] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:52:16.868338 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0e0d504-42be-49b5-be81-10c9910d0dd7 None None] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:52:16.868338 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0e0d504-42be-49b5-be81-10c9910d0dd7 None None] PATH_INFO: `/` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:52:16.869715 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 692/1386] 10.222.0.30 () {54 vars in 926 bytes} [Mon May 23 12:52:16 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) May 23 12:52:16.875577 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01b1c067-55d7-4ad4-b0fd-3e7faef51727 None None] REQUEST_METHOD: `POST` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:52:16.876099 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01b1c067-55d7-4ad4-b0fd-3e7faef51727 None None] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:52:16.876253 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01b1c067-55d7-4ad4-b0fd-3e7faef51727 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:52:16.894410 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.auth.core [None req-01b1c067-55d7-4ad4-b0fd-3e7faef51727 None None] MFA Rules not processed for user `0f7119f8fa5c4445977671140ded91a4`. Rule list: `[]` (Enabled: `True`). {{(pid=92057) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 12:52:16.895605 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.fernet_utils [None req-01b1c067-55d7-4ad4-b0fd-3e7faef51727 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92057) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 12:52:16.903948 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 695/1387] 10.222.0.30 () {58 vars in 1023 bytes} [Mon May 23 12:52:16 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 30 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) May 23 12:52:16.956175 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0e0d504-42be-49b5-be81-10c9910d0dd7 None None] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:52:16.975404 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0e0d504-42be-49b5-be81-10c9910d0dd7 None None] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:52:16.981070 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e92005bb-0804-45de-a276-be647f9a1dab service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cbe9931d94d44cc7ace0e83a7e5015fc', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9c389ae65aaa4542bb5a5c4fd0bacf5b', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'member', 'reader', 'service', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:52:16.981709 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e92005bb-0804-45de-a276-be647f9a1dab service nova] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:52:16.982026 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e92005bb-0804-45de-a276-be647f9a1dab service nova] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:52:16.982300 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e92005bb-0804-45de-a276-be647f9a1dab service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:52:16.991630 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e92005bb-0804-45de-a276-be647f9a1dab service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:52:16.993206 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e92005bb-0804-45de-a276-be647f9a1dab service nova] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:52:17.002902 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 693/1388] 10.222.0.30 () {60 vars in 1365 bytes} [Mon May 23 12:52:16 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 48 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) May 23 12:52:19.914052 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41833807-a99e-4716-8025-5595e59eb284 None None] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:52:19.914754 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41833807-a99e-4716-8025-5595e59eb284 None None] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:52:19.914754 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41833807-a99e-4716-8025-5595e59eb284 None None] PATH_INFO: `/` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:52:19.916104 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 696/1389] 10.222.0.30 () {54 vars in 926 bytes} [Mon May 23 12:52:19 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) May 23 12:52:19.922499 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d67c703d-0b61-4e0e-9927-62d651170bda None None] REQUEST_METHOD: `POST` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:52:19.922992 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d67c703d-0b61-4e0e-9927-62d651170bda None None] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:52:19.923087 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d67c703d-0b61-4e0e-9927-62d651170bda None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:52:19.941675 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.auth.core [None req-d67c703d-0b61-4e0e-9927-62d651170bda None None] MFA Rules not processed for user `0f7119f8fa5c4445977671140ded91a4`. Rule list: `[]` (Enabled: `True`). {{(pid=92056) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 12:52:19.942926 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.fernet_utils [None req-d67c703d-0b61-4e0e-9927-62d651170bda None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92056) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 12:52:19.952473 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 694/1390] 10.222.0.30 () {58 vars in 1023 bytes} [Mon May 23 12:52:19 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) May 23 12:52:20.009366 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41833807-a99e-4716-8025-5595e59eb284 None None] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:52:20.025464 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41833807-a99e-4716-8025-5595e59eb284 None None] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:52:20.031451 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-020a0fa3-43d0-4dcc-862f-287c595f0592 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cbe9931d94d44cc7ace0e83a7e5015fc', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9c389ae65aaa4542bb5a5c4fd0bacf5b', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'member', 'reader', 'service', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:52:20.032111 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-020a0fa3-43d0-4dcc-862f-287c595f0592 service nova] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:52:20.032415 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-020a0fa3-43d0-4dcc-862f-287c595f0592 service nova] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:52:20.032661 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-020a0fa3-43d0-4dcc-862f-287c595f0592 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:52:20.041527 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-020a0fa3-43d0-4dcc-862f-287c595f0592 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:52:20.043273 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-020a0fa3-43d0-4dcc-862f-287c595f0592 service nova] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:52:20.053524 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 697/1391] 10.222.0.30 () {60 vars in 1365 bytes} [Mon May 23 12:52:20 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 45 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) May 23 12:52:22.604452 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4261b0f3-d839-4fa2-b9da-acfb0e93b1c0 None None] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:52:22.605049 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4261b0f3-d839-4fa2-b9da-acfb0e93b1c0 None None] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:52:22.605161 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4261b0f3-d839-4fa2-b9da-acfb0e93b1c0 None None] PATH_INFO: `/` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:52:22.606516 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 695/1392] 10.222.0.30 () {54 vars in 926 bytes} [Mon May 23 12:52:22 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) May 23 12:52:22.612340 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a1f0122-a2b3-4789-a55a-2ceca6e160f8 None None] REQUEST_METHOD: `POST` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:52:22.612752 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a1f0122-a2b3-4789-a55a-2ceca6e160f8 None None] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:52:22.612902 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a1f0122-a2b3-4789-a55a-2ceca6e160f8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:52:22.631367 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.auth.core [None req-3a1f0122-a2b3-4789-a55a-2ceca6e160f8 None None] MFA Rules not processed for user `0f7119f8fa5c4445977671140ded91a4`. Rule list: `[]` (Enabled: `True`). {{(pid=92057) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 12:52:22.632560 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.fernet_utils [None req-3a1f0122-a2b3-4789-a55a-2ceca6e160f8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92057) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 12:52:22.640231 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 698/1393] 10.222.0.30 () {58 vars in 1023 bytes} [Mon May 23 12:52:22 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 29 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) May 23 12:52:22.684977 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4261b0f3-d839-4fa2-b9da-acfb0e93b1c0 None None] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:52:22.700520 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4261b0f3-d839-4fa2-b9da-acfb0e93b1c0 None None] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:52:22.706627 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72d60ff0-d5a0-45c9-a538-6f9e1cab7c10 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cbe9931d94d44cc7ace0e83a7e5015fc', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9c389ae65aaa4542bb5a5c4fd0bacf5b', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'member', 'reader', 'service', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:52:22.707266 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72d60ff0-d5a0-45c9-a538-6f9e1cab7c10 service nova] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:52:22.707550 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72d60ff0-d5a0-45c9-a538-6f9e1cab7c10 service nova] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:52:22.707827 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72d60ff0-d5a0-45c9-a538-6f9e1cab7c10 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:52:22.716413 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72d60ff0-d5a0-45c9-a538-6f9e1cab7c10 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:52:22.718073 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72d60ff0-d5a0-45c9-a538-6f9e1cab7c10 service nova] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:52:22.728217 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 696/1394] 10.222.0.30 () {60 vars in 1365 bytes} [Mon May 23 12:52:22 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 44 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) May 23 12:52:23.896490 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d608c73-25ad-4c8d-a661-0373ebedb057 None None] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:52:23.897050 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d608c73-25ad-4c8d-a661-0373ebedb057 None None] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:52:23.897050 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d608c73-25ad-4c8d-a661-0373ebedb057 None None] PATH_INFO: `/` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:52:23.898529 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 699/1395] 10.222.0.30 () {54 vars in 926 bytes} [Mon May 23 12:52:23 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) May 23 12:52:23.904090 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d2db8ca-236c-4d42-ba2e-c2611abf8cd6 None None] REQUEST_METHOD: `POST` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:52:23.904506 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d2db8ca-236c-4d42-ba2e-c2611abf8cd6 None None] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:52:23.904768 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d2db8ca-236c-4d42-ba2e-c2611abf8cd6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:52:23.911802 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=52 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:52:23.923804 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.auth.core [None req-4d2db8ca-236c-4d42-ba2e-c2611abf8cd6 None None] MFA Rules not processed for user `0f7119f8fa5c4445977671140ded91a4`. Rule list: `[]` (Enabled: `True`). {{(pid=92056) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 12:52:23.924979 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.fernet_utils [None req-4d2db8ca-236c-4d42-ba2e-c2611abf8cd6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92056) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 12:52:23.932828 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 697/1396] 10.222.0.30 () {58 vars in 1023 bytes} [Mon May 23 12:52:23 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 30 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) May 23 12:52:23.988341 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd2c9f04-46cb-4ea2-a8ca-cca94d22da15 None None] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:52:23.988632 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd2c9f04-46cb-4ea2-a8ca-cca94d22da15 None None] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:52:23.988912 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd2c9f04-46cb-4ea2-a8ca-cca94d22da15 None None] PATH_INFO: `/` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:52:23.990331 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 700/1397] 10.222.0.30 () {54 vars in 962 bytes} [Mon May 23 12:52:23 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) May 23 12:52:23.995257 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51971694-9784-4ce1-bb13-4bcb8cc32a1e None None] REQUEST_METHOD: `POST` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:52:23.995552 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51971694-9784-4ce1-bb13-4bcb8cc32a1e None None] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:52:23.995836 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51971694-9784-4ce1-bb13-4bcb8cc32a1e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:52:24.045109 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.auth.core [None req-51971694-9784-4ce1-bb13-4bcb8cc32a1e None None] MFA Rules not processed for user `5cea0feed4254ec085854986f99acf30`. Rule list: `[]` (Enabled: `True`). {{(pid=92056) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 12:52:24.046338 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.fernet_utils [None req-51971694-9784-4ce1-bb13-4bcb8cc32a1e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92056) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 12:52:24.104351 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 698/1398] 10.222.0.30 () {58 vars in 1059 bytes} [Mon May 23 12:52:23 2022] POST /identity/v3/auth/tokens => generated 3110 bytes in 110 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) May 23 12:52:24.111833 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd2c9f04-46cb-4ea2-a8ca-cca94d22da15 None None] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:52:24.122605 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd2c9f04-46cb-4ea2-a8ca-cca94d22da15 None None] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:52:24.126800 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-450d9b3d-ab5a-46bc-87ca-612478adc4d6 service glance] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '5cea0feed4254ec085854986f99acf30', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9c389ae65aaa4542bb5a5c4fd0bacf5b', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:52:24.127436 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-450d9b3d-ab5a-46bc-87ca-612478adc4d6 service glance] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:52:24.127733 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-450d9b3d-ab5a-46bc-87ca-612478adc4d6 service glance] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:52:24.128012 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-450d9b3d-ab5a-46bc-87ca-612478adc4d6 service glance] PATH_INFO: `/v3/auth/tokens` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:52:24.136975 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-450d9b3d-ab5a-46bc-87ca-612478adc4d6 service glance] RBAC: Authorizing `identity:validate_token()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:52:24.138458 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-450d9b3d-ab5a-46bc-87ca-612478adc4d6 service glance] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:52:24.148991 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 701/1399] 10.222.0.30 () {60 vars in 1365 bytes} [Mon May 23 12:52:24 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 37 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) May 23 12:52:24.850927 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eeceec57-4b32-4db4-a8a4-ce6ba66e6dac None None] REQUEST_METHOD: `POST` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:52:24.851221 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eeceec57-4b32-4db4-a8a4-ce6ba66e6dac None None] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:52:24.851503 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eeceec57-4b32-4db4-a8a4-ce6ba66e6dac None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:52:24.900751 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.auth.core [None req-eeceec57-4b32-4db4-a8a4-ce6ba66e6dac None None] MFA Rules not processed for user `6d21a56126fd4ccd837c33ccc08ab609`. Rule list: `[]` (Enabled: `True`). {{(pid=92056) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 12:52:24.902234 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.fernet_utils [None req-eeceec57-4b32-4db4-a8a4-ce6ba66e6dac None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92056) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 12:52:24.965486 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 699/1400] 10.222.0.30 () {58 vars in 1009 bytes} [Mon May 23 12:52:24 2022] POST /identity/v3/auth/tokens => generated 3185 bytes in 116 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) May 23 12:52:25.007199 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f732387-b663-411b-9648-acecf10367f0 None None] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:52:25.007690 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f732387-b663-411b-9648-acecf10367f0 None None] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:52:25.007879 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f732387-b663-411b-9648-acecf10367f0 None None] PATH_INFO: `/` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:52:25.009324 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 702/1401] 10.222.0.30 () {54 vars in 942 bytes} [Mon May 23 12:52:25 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) May 23 12:52:25.015825 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6f2c3c0-38ba-4b2c-a6b8-3a282f27eb23 None None] REQUEST_METHOD: `POST` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:52:25.016115 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6f2c3c0-38ba-4b2c-a6b8-3a282f27eb23 None None] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:52:25.016395 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6f2c3c0-38ba-4b2c-a6b8-3a282f27eb23 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:52:25.064555 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.auth.core [None req-a6f2c3c0-38ba-4b2c-a6b8-3a282f27eb23 None None] MFA Rules not processed for user `138d91edb99f450e846765f9eef00d62`. Rule list: `[]` (Enabled: `True`). {{(pid=92056) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 12:52:25.065752 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.fernet_utils [None req-a6f2c3c0-38ba-4b2c-a6b8-3a282f27eb23 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92056) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 12:52:25.078733 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=100 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:52:25.138709 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 700/1402] 10.222.0.30 () {58 vars in 1039 bytes} [Mon May 23 12:52:25 2022] POST /identity/v3/auth/tokens => generated 3294 bytes in 124 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) May 23 12:52:25.148613 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f732387-b663-411b-9648-acecf10367f0 None None] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:52:25.164148 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f732387-b663-411b-9648-acecf10367f0 None None] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:52:25.169081 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-672176f8-28bd-41aa-82ff-025b8daa9e06 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '138d91edb99f450e846765f9eef00d62', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9c389ae65aaa4542bb5a5c4fd0bacf5b', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'service', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:52:25.169727 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-672176f8-28bd-41aa-82ff-025b8daa9e06 service swift] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:52:25.170055 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-672176f8-28bd-41aa-82ff-025b8daa9e06 service swift] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:52:25.170335 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-672176f8-28bd-41aa-82ff-025b8daa9e06 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:52:25.179849 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-672176f8-28bd-41aa-82ff-025b8daa9e06 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:52:25.181446 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-672176f8-28bd-41aa-82ff-025b8daa9e06 service swift] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:52:25.189514 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 703/1403] 10.222.0.30 () {60 vars in 1384 bytes} [Mon May 23 12:52:25 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 41 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) May 23 12:52:25.330105 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1af3063a-d8ad-4b16-8c78-85b3359e5d35 None None] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:52:25.330397 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1af3063a-d8ad-4b16-8c78-85b3359e5d35 None None] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:52:25.330678 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1af3063a-d8ad-4b16-8c78-85b3359e5d35 None None] PATH_INFO: `/` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:52:25.332068 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 701/1404] 10.222.0.30 () {54 vars in 926 bytes} [Mon May 23 12:52:25 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) May 23 12:52:25.337407 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae3f94a1-9900-4024-804b-4b065ffd3184 None None] REQUEST_METHOD: `POST` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:52:25.342467 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae3f94a1-9900-4024-804b-4b065ffd3184 None None] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:52:25.342467 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae3f94a1-9900-4024-804b-4b065ffd3184 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:52:25.356592 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.auth.core [None req-ae3f94a1-9900-4024-804b-4b065ffd3184 None None] MFA Rules not processed for user `0f7119f8fa5c4445977671140ded91a4`. Rule list: `[]` (Enabled: `True`). {{(pid=92057) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 12:52:25.357772 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.fernet_utils [None req-ae3f94a1-9900-4024-804b-4b065ffd3184 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92057) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 12:52:25.366141 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 704/1405] 10.222.0.30 () {58 vars in 1023 bytes} [Mon May 23 12:52:25 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 30 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) May 23 12:52:25.411397 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1af3063a-d8ad-4b16-8c78-85b3359e5d35 None None] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:52:25.426102 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1af3063a-d8ad-4b16-8c78-85b3359e5d35 None None] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:52:25.431917 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d78bbbc-a868-469a-b313-d08b6311e56a service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cbe9931d94d44cc7ace0e83a7e5015fc', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9c389ae65aaa4542bb5a5c4fd0bacf5b', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'member', 'reader', 'service', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:52:25.432647 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d78bbbc-a868-469a-b313-d08b6311e56a service nova] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:52:25.432987 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d78bbbc-a868-469a-b313-d08b6311e56a service nova] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:52:25.433266 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d78bbbc-a868-469a-b313-d08b6311e56a service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:52:25.443292 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d78bbbc-a868-469a-b313-d08b6311e56a service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:52:25.444884 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d78bbbc-a868-469a-b313-d08b6311e56a service nova] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:52:25.455134 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 702/1406] 10.222.0.30 () {60 vars in 1365 bytes} [Mon May 23 12:52:25 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 44 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) May 23 12:52:27.869002 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6962c49c-70a1-481b-bec6-efadcb92bd31 None None] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:52:27.869566 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6962c49c-70a1-481b-bec6-efadcb92bd31 None None] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:52:27.869566 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6962c49c-70a1-481b-bec6-efadcb92bd31 None None] PATH_INFO: `/` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:52:27.870958 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 705/1407] 10.222.0.30 () {54 vars in 926 bytes} [Mon May 23 12:52:27 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) May 23 12:52:27.877410 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a97aa2a-8078-4519-981d-8f801ef82b75 None None] REQUEST_METHOD: `POST` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:52:27.877833 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a97aa2a-8078-4519-981d-8f801ef82b75 None None] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:52:27.878035 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a97aa2a-8078-4519-981d-8f801ef82b75 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:52:27.903488 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.auth.core [None req-8a97aa2a-8078-4519-981d-8f801ef82b75 None None] MFA Rules not processed for user `0f7119f8fa5c4445977671140ded91a4`. Rule list: `[]` (Enabled: `True`). {{(pid=92056) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 12:52:27.906532 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.fernet_utils [None req-8a97aa2a-8078-4519-981d-8f801ef82b75 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92056) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 12:52:27.913102 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 703/1408] 10.222.0.30 () {58 vars in 1023 bytes} [Mon May 23 12:52:27 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 37 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) May 23 12:52:27.958184 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6962c49c-70a1-481b-bec6-efadcb92bd31 None None] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:52:27.974087 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6962c49c-70a1-481b-bec6-efadcb92bd31 None None] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:52:27.979845 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d76cb196-51c5-4d01-85cb-0740e3685762 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cbe9931d94d44cc7ace0e83a7e5015fc', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9c389ae65aaa4542bb5a5c4fd0bacf5b', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'member', 'reader', 'service', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:52:27.980621 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d76cb196-51c5-4d01-85cb-0740e3685762 service nova] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:52:27.980911 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d76cb196-51c5-4d01-85cb-0740e3685762 service nova] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:52:27.981446 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d76cb196-51c5-4d01-85cb-0740e3685762 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:52:27.991003 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d76cb196-51c5-4d01-85cb-0740e3685762 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:52:27.992782 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d76cb196-51c5-4d01-85cb-0740e3685762 service nova] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:52:28.003621 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 706/1409] 10.222.0.30 () {60 vars in 1365 bytes} [Mon May 23 12:52:27 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 46 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) May 23 12:52:37.897319 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=33 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:52:38.000638 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=43 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:52:49.554433 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-159df270-26bf-4b25-98b3-d3593e492393 None None] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:52:49.555002 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-159df270-26bf-4b25-98b3-d3593e492393 None None] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:52:49.555002 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-159df270-26bf-4b25-98b3-d3593e492393 None None] PATH_INFO: `/` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:52:49.556358 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 704/1410] 10.222.0.30 () {54 vars in 918 bytes} [Mon May 23 12:52:49 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) May 23 12:52:49.561451 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bc604eb-5048-43d4-b633-00246004f41c None None] REQUEST_METHOD: `POST` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:52:49.561908 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bc604eb-5048-43d4-b633-00246004f41c None None] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:52:49.562065 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bc604eb-5048-43d4-b633-00246004f41c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:52:49.580795 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.auth.core [None req-7bc604eb-5048-43d4-b633-00246004f41c None None] MFA Rules not processed for user `d5fb51f8d0fa46439cb29666970b885b`. Rule list: `[]` (Enabled: `True`). {{(pid=92057) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 12:52:49.582011 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.fernet_utils [None req-7bc604eb-5048-43d4-b633-00246004f41c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92057) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 12:52:49.590486 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 707/1411] 10.222.0.30 () {58 vars in 1015 bytes} [Mon May 23 12:52:49 2022] POST /identity/v3/auth/tokens => generated 3298 bytes in 30 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) May 23 12:52:49.605201 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f3b1e36-53cb-47e6-8e0f-67fb14185c32 None None] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:52:49.605489 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f3b1e36-53cb-47e6-8e0f-67fb14185c32 None None] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:52:49.605761 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f3b1e36-53cb-47e6-8e0f-67fb14185c32 None None] PATH_INFO: `/` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:52:49.607157 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 705/1412] 10.222.0.30 () {54 vars in 918 bytes} [Mon May 23 12:52:49 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) May 23 12:52:49.611668 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c0c0257-00f5-4484-b04d-5e29052e91ca None None] REQUEST_METHOD: `POST` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:52:49.611956 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c0c0257-00f5-4484-b04d-5e29052e91ca None None] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:52:49.612235 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c0c0257-00f5-4484-b04d-5e29052e91ca None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:52:49.629531 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.auth.core [None req-0c0c0257-00f5-4484-b04d-5e29052e91ca None None] MFA Rules not processed for user `cbe9931d94d44cc7ace0e83a7e5015fc`. Rule list: `[]` (Enabled: `True`). {{(pid=92057) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 12:52:49.630735 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.fernet_utils [None req-0c0c0257-00f5-4484-b04d-5e29052e91ca None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92057) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 12:52:49.640205 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 708/1413] 10.222.0.30 () {58 vars in 1015 bytes} [Mon May 23 12:52:49 2022] POST /identity/v3/auth/tokens => generated 3362 bytes in 29 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) May 23 12:52:52.343271 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b29a4470-49ca-47d8-95bf-d81cb091c1bf None None] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:52:52.343808 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b29a4470-49ca-47d8-95bf-d81cb091c1bf None None] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:52:52.343808 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b29a4470-49ca-47d8-95bf-d81cb091c1bf None None] PATH_INFO: `/` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:52:52.345160 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 706/1414] 127.0.0.1 () {54 vars in 916 bytes} [Mon May 23 12:52:52 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) May 23 12:52:52.350534 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-341887f0-0a10-4c52-af70-b849ec1a15d3 None None] REQUEST_METHOD: `POST` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:52:52.350951 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-341887f0-0a10-4c52-af70-b849ec1a15d3 None None] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:52:52.351123 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-341887f0-0a10-4c52-af70-b849ec1a15d3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:52:52.369426 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.auth.core [None req-341887f0-0a10-4c52-af70-b849ec1a15d3 None None] MFA Rules not processed for user `0f7119f8fa5c4445977671140ded91a4`. Rule list: `[]` (Enabled: `True`). {{(pid=92057) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 12:52:52.370630 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.fernet_utils [None req-341887f0-0a10-4c52-af70-b849ec1a15d3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92057) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 12:52:52.379178 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 709/1415] 127.0.0.1 () {58 vars in 1013 bytes} [Mon May 23 12:52:52 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 30 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) May 23 12:52:52.422844 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b29a4470-49ca-47d8-95bf-d81cb091c1bf None None] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:52:52.437364 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b29a4470-49ca-47d8-95bf-d81cb091c1bf None None] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:52:52.442723 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-421b0d99-e579-42da-b8e5-2a12470dcdab service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cbe9931d94d44cc7ace0e83a7e5015fc', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9c389ae65aaa4542bb5a5c4fd0bacf5b', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'member', 'reader', 'service', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:52:52.443352 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-421b0d99-e579-42da-b8e5-2a12470dcdab service nova] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:52:52.443634 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-421b0d99-e579-42da-b8e5-2a12470dcdab service nova] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:52:52.443912 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-421b0d99-e579-42da-b8e5-2a12470dcdab service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:52:52.452124 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-421b0d99-e579-42da-b8e5-2a12470dcdab service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:52:52.453872 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-421b0d99-e579-42da-b8e5-2a12470dcdab service nova] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:52:52.463217 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 707/1416] 10.222.0.30 () {60 vars in 1365 bytes} [Mon May 23 12:52:52 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 41 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) May 23 12:52:53.435458 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-992afc74-5673-4bf8-94f1-b9060e506845 None None] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:52:53.436030 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-992afc74-5673-4bf8-94f1-b9060e506845 None None] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:52:53.436030 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-992afc74-5673-4bf8-94f1-b9060e506845 None None] PATH_INFO: `/` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:52:53.438209 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 710/1417] 127.0.0.1 () {54 vars in 901 bytes} [Mon May 23 12:52:53 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) May 23 12:52:53.443183 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-830abcbf-5d86-4bc1-bbac-4572a656bc2b None None] REQUEST_METHOD: `POST` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:52:53.443616 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-830abcbf-5d86-4bc1-bbac-4572a656bc2b None None] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:52:53.443722 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-830abcbf-5d86-4bc1-bbac-4572a656bc2b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:52:53.461761 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.auth.core [None req-830abcbf-5d86-4bc1-bbac-4572a656bc2b None None] MFA Rules not processed for user `0f7119f8fa5c4445977671140ded91a4`. Rule list: `[]` (Enabled: `True`). {{(pid=92056) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 12:52:53.462960 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.fernet_utils [None req-830abcbf-5d86-4bc1-bbac-4572a656bc2b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92056) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 12:52:53.470714 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 708/1418] 127.0.0.1 () {58 vars in 998 bytes} [Mon May 23 12:52:53 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 29 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) May 23 12:52:53.647985 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-992afc74-5673-4bf8-94f1-b9060e506845 None None] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:52:53.662440 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-992afc74-5673-4bf8-94f1-b9060e506845 None None] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:52:53.667975 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-182d2912-1633-4c84-87d0-00cb1fc221a2 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cbe9931d94d44cc7ace0e83a7e5015fc', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9c389ae65aaa4542bb5a5c4fd0bacf5b', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'member', 'reader', 'service', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:52:53.668645 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-182d2912-1633-4c84-87d0-00cb1fc221a2 service nova] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:52:53.668929 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-182d2912-1633-4c84-87d0-00cb1fc221a2 service nova] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:52:53.669204 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-182d2912-1633-4c84-87d0-00cb1fc221a2 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:52:53.678035 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-182d2912-1633-4c84-87d0-00cb1fc221a2 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:52:53.679661 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-182d2912-1633-4c84-87d0-00cb1fc221a2 service nova] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:52:53.694880 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 711/1419] 10.222.0.30 () {60 vars in 1365 bytes} [Mon May 23 12:52:53 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 42 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) May 23 12:52:54.610680 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44910cfa-92a6-4ad9-b29b-b38ff70fc38a None None] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:52:54.611254 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44910cfa-92a6-4ad9-b29b-b38ff70fc38a None None] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:52:54.611254 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44910cfa-92a6-4ad9-b29b-b38ff70fc38a None None] PATH_INFO: `/` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:52:54.612646 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 709/1420] 127.0.0.1 () {54 vars in 901 bytes} [Mon May 23 12:52:54 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) May 23 12:52:54.617614 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1acbc494-7bf4-448a-b016-d7af583c7bee None None] REQUEST_METHOD: `POST` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:52:54.618037 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1acbc494-7bf4-448a-b016-d7af583c7bee None None] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:52:54.618222 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1acbc494-7bf4-448a-b016-d7af583c7bee None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:52:54.635944 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.auth.core [None req-1acbc494-7bf4-448a-b016-d7af583c7bee None None] MFA Rules not processed for user `0f7119f8fa5c4445977671140ded91a4`. Rule list: `[]` (Enabled: `True`). {{(pid=92057) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 12:52:54.637131 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.fernet_utils [None req-1acbc494-7bf4-448a-b016-d7af583c7bee None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92057) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 12:52:54.645535 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 712/1421] 127.0.0.1 () {58 vars in 998 bytes} [Mon May 23 12:52:54 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 29 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) May 23 12:52:54.798595 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44910cfa-92a6-4ad9-b29b-b38ff70fc38a None None] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:52:54.812750 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44910cfa-92a6-4ad9-b29b-b38ff70fc38a None None] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:52:54.818091 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81df956e-5c1e-482c-8d78-7a68aca3e3de service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cbe9931d94d44cc7ace0e83a7e5015fc', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9c389ae65aaa4542bb5a5c4fd0bacf5b', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'member', 'reader', 'service', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:52:54.820255 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81df956e-5c1e-482c-8d78-7a68aca3e3de service nova] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:52:54.820255 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81df956e-5c1e-482c-8d78-7a68aca3e3de service nova] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:52:54.820255 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81df956e-5c1e-482c-8d78-7a68aca3e3de service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:52:54.828181 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81df956e-5c1e-482c-8d78-7a68aca3e3de service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:52:54.829769 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81df956e-5c1e-482c-8d78-7a68aca3e3de service nova] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:52:54.839722 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 710/1422] 10.222.0.30 () {60 vars in 1365 bytes} [Mon May 23 12:52:54 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 41 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) May 23 12:52:55.772903 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c678bc6f-6886-4198-b3fc-68ffcadee846 None None] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:52:55.773394 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c678bc6f-6886-4198-b3fc-68ffcadee846 None None] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:52:55.773394 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c678bc6f-6886-4198-b3fc-68ffcadee846 None None] PATH_INFO: `/` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:52:55.774782 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 713/1423] 127.0.0.1 () {54 vars in 901 bytes} [Mon May 23 12:52:55 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) May 23 12:52:55.780666 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ef07cc7-d872-494f-b83d-bc7aacac25b8 None None] REQUEST_METHOD: `POST` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:52:55.781098 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ef07cc7-d872-494f-b83d-bc7aacac25b8 None None] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:52:55.781266 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ef07cc7-d872-494f-b83d-bc7aacac25b8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:52:55.800065 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.auth.core [None req-0ef07cc7-d872-494f-b83d-bc7aacac25b8 None None] MFA Rules not processed for user `0f7119f8fa5c4445977671140ded91a4`. Rule list: `[]` (Enabled: `True`). {{(pid=92056) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 12:52:55.801258 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.fernet_utils [None req-0ef07cc7-d872-494f-b83d-bc7aacac25b8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92056) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 12:52:55.809015 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 711/1424] 127.0.0.1 () {58 vars in 998 bytes} [Mon May 23 12:52:55 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 30 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) May 23 12:52:55.961607 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c678bc6f-6886-4198-b3fc-68ffcadee846 None None] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:52:55.976511 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c678bc6f-6886-4198-b3fc-68ffcadee846 None None] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:52:55.981784 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-714e440f-a223-4be2-8b25-bccf393633f5 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cbe9931d94d44cc7ace0e83a7e5015fc', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9c389ae65aaa4542bb5a5c4fd0bacf5b', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'member', 'reader', 'service', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:52:55.982444 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-714e440f-a223-4be2-8b25-bccf393633f5 service nova] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:52:55.982729 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-714e440f-a223-4be2-8b25-bccf393633f5 service nova] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:52:55.983602 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-714e440f-a223-4be2-8b25-bccf393633f5 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:52:55.991573 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-714e440f-a223-4be2-8b25-bccf393633f5 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:52:55.993154 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-714e440f-a223-4be2-8b25-bccf393633f5 service nova] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:52:56.002551 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 714/1425] 10.222.0.30 () {60 vars in 1365 bytes} [Mon May 23 12:52:55 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 41 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) May 23 12:52:56.907731 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c760fc05-c406-4b65-a44c-b66f51d9480a None None] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:52:56.908218 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c760fc05-c406-4b65-a44c-b66f51d9480a None None] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:52:56.908329 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c760fc05-c406-4b65-a44c-b66f51d9480a None None] PATH_INFO: `/` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:52:56.909738 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 712/1426] 127.0.0.1 () {54 vars in 901 bytes} [Mon May 23 12:52:56 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) May 23 12:52:56.914695 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca8d52d6-b552-4559-a73d-2b47220f3519 None None] REQUEST_METHOD: `POST` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:52:56.915116 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca8d52d6-b552-4559-a73d-2b47220f3519 None None] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:52:56.915259 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca8d52d6-b552-4559-a73d-2b47220f3519 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:52:56.933518 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.auth.core [None req-ca8d52d6-b552-4559-a73d-2b47220f3519 None None] MFA Rules not processed for user `0f7119f8fa5c4445977671140ded91a4`. Rule list: `[]` (Enabled: `True`). {{(pid=92057) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 12:52:56.934755 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.fernet_utils [None req-ca8d52d6-b552-4559-a73d-2b47220f3519 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92057) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 12:52:56.943467 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 715/1427] 127.0.0.1 () {58 vars in 998 bytes} [Mon May 23 12:52:56 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 30 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) May 23 12:52:57.098556 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c760fc05-c406-4b65-a44c-b66f51d9480a None None] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:52:57.113442 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c760fc05-c406-4b65-a44c-b66f51d9480a None None] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:52:57.118825 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19c74883-592d-4733-ac10-5c937c4c6a08 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cbe9931d94d44cc7ace0e83a7e5015fc', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9c389ae65aaa4542bb5a5c4fd0bacf5b', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'member', 'reader', 'service', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:52:57.119451 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19c74883-592d-4733-ac10-5c937c4c6a08 service nova] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:52:57.121221 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19c74883-592d-4733-ac10-5c937c4c6a08 service nova] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:52:57.121221 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19c74883-592d-4733-ac10-5c937c4c6a08 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:52:57.129102 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19c74883-592d-4733-ac10-5c937c4c6a08 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:52:57.130726 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19c74883-592d-4733-ac10-5c937c4c6a08 service nova] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:52:57.140150 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 713/1428] 10.222.0.30 () {60 vars in 1365 bytes} [Mon May 23 12:52:57 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 42 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) May 23 12:52:58.103030 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41ff75b8-e224-4d6b-9341-eaf54de528a3 None None] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:52:58.103589 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41ff75b8-e224-4d6b-9341-eaf54de528a3 None None] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:52:58.103589 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41ff75b8-e224-4d6b-9341-eaf54de528a3 None None] PATH_INFO: `/` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:52:58.104978 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 716/1429] 127.0.0.1 () {54 vars in 901 bytes} [Mon May 23 12:52:58 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) May 23 12:52:58.110277 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b2b6317-70fc-4eb4-919f-bf7c89da066b None None] REQUEST_METHOD: `POST` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:52:58.110680 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b2b6317-70fc-4eb4-919f-bf7c89da066b None None] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:52:58.110833 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b2b6317-70fc-4eb4-919f-bf7c89da066b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:52:58.136872 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.auth.core [None req-0b2b6317-70fc-4eb4-919f-bf7c89da066b None None] MFA Rules not processed for user `0f7119f8fa5c4445977671140ded91a4`. Rule list: `[]` (Enabled: `True`). {{(pid=92056) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 12:52:58.138985 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.fernet_utils [None req-0b2b6317-70fc-4eb4-919f-bf7c89da066b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92056) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 12:52:58.149246 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 714/1430] 127.0.0.1 () {58 vars in 998 bytes} [Mon May 23 12:52:58 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 40 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) May 23 12:52:58.299630 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41ff75b8-e224-4d6b-9341-eaf54de528a3 None None] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:52:58.314526 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41ff75b8-e224-4d6b-9341-eaf54de528a3 None None] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:52:58.319781 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40d0b26e-0686-4152-a67b-9a3aec273a6b service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cbe9931d94d44cc7ace0e83a7e5015fc', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9c389ae65aaa4542bb5a5c4fd0bacf5b', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'member', 'reader', 'service', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:52:58.320412 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40d0b26e-0686-4152-a67b-9a3aec273a6b service nova] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:52:58.320697 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40d0b26e-0686-4152-a67b-9a3aec273a6b service nova] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:52:58.320974 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40d0b26e-0686-4152-a67b-9a3aec273a6b service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:52:58.329671 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40d0b26e-0686-4152-a67b-9a3aec273a6b service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:52:58.331291 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40d0b26e-0686-4152-a67b-9a3aec273a6b service nova] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:52:58.340513 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 717/1431] 10.222.0.30 () {60 vars in 1365 bytes} [Mon May 23 12:52:58 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 41 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) May 23 12:52:59.259466 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b34b444b-5dad-4b99-97e5-cf082a3ec31c None None] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:52:59.259998 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b34b444b-5dad-4b99-97e5-cf082a3ec31c None None] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:52:59.259998 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b34b444b-5dad-4b99-97e5-cf082a3ec31c None None] PATH_INFO: `/` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:52:59.261320 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 715/1432] 127.0.0.1 () {54 vars in 901 bytes} [Mon May 23 12:52:59 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) May 23 12:52:59.266954 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc93a869-2622-4e59-a540-25f178aad8e5 None None] REQUEST_METHOD: `POST` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:52:59.267403 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc93a869-2622-4e59-a540-25f178aad8e5 None None] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:52:59.267502 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc93a869-2622-4e59-a540-25f178aad8e5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:52:59.286267 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.auth.core [None req-dc93a869-2622-4e59-a540-25f178aad8e5 None None] MFA Rules not processed for user `0f7119f8fa5c4445977671140ded91a4`. Rule list: `[]` (Enabled: `True`). {{(pid=92057) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 12:52:59.287524 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.fernet_utils [None req-dc93a869-2622-4e59-a540-25f178aad8e5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92057) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 12:52:59.295465 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 718/1433] 127.0.0.1 () {58 vars in 998 bytes} [Mon May 23 12:52:59 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 30 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) May 23 12:52:59.448774 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b34b444b-5dad-4b99-97e5-cf082a3ec31c None None] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:52:59.463982 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b34b444b-5dad-4b99-97e5-cf082a3ec31c None None] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:52:59.469459 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-907826c2-817c-4cf8-a9c1-f713fe260b35 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cbe9931d94d44cc7ace0e83a7e5015fc', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9c389ae65aaa4542bb5a5c4fd0bacf5b', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'member', 'reader', 'service', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:52:59.470114 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-907826c2-817c-4cf8-a9c1-f713fe260b35 service nova] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:52:59.470400 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-907826c2-817c-4cf8-a9c1-f713fe260b35 service nova] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:52:59.470784 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-907826c2-817c-4cf8-a9c1-f713fe260b35 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:52:59.478803 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-907826c2-817c-4cf8-a9c1-f713fe260b35 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:52:59.480393 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-907826c2-817c-4cf8-a9c1-f713fe260b35 service nova] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:52:59.489928 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 716/1434] 10.222.0.30 () {60 vars in 1365 bytes} [Mon May 23 12:52:59 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 42 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) May 23 12:53:00.400899 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43d1756e-9d93-4ef5-acaa-5fb1b0cef4c3 None None] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:53:00.401463 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43d1756e-9d93-4ef5-acaa-5fb1b0cef4c3 None None] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:53:00.401463 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43d1756e-9d93-4ef5-acaa-5fb1b0cef4c3 None None] PATH_INFO: `/` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:53:00.402896 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 719/1435] 127.0.0.1 () {54 vars in 901 bytes} [Mon May 23 12:53:00 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) May 23 12:53:00.408599 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f1996ee-6f80-4c28-9cfd-d43917e9af4b None None] REQUEST_METHOD: `POST` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:53:00.409028 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f1996ee-6f80-4c28-9cfd-d43917e9af4b None None] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:53:00.409188 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f1996ee-6f80-4c28-9cfd-d43917e9af4b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:53:00.427777 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.auth.core [None req-5f1996ee-6f80-4c28-9cfd-d43917e9af4b None None] MFA Rules not processed for user `0f7119f8fa5c4445977671140ded91a4`. Rule list: `[]` (Enabled: `True`). {{(pid=92056) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 12:53:00.428969 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.fernet_utils [None req-5f1996ee-6f80-4c28-9cfd-d43917e9af4b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92056) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 12:53:00.437697 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 717/1436] 127.0.0.1 () {58 vars in 998 bytes} [Mon May 23 12:53:00 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 30 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) May 23 12:53:00.589480 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43d1756e-9d93-4ef5-acaa-5fb1b0cef4c3 None None] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:53:00.604743 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43d1756e-9d93-4ef5-acaa-5fb1b0cef4c3 None None] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:53:00.610394 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99999a30-6847-4250-afb5-9a49466db993 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cbe9931d94d44cc7ace0e83a7e5015fc', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9c389ae65aaa4542bb5a5c4fd0bacf5b', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'member', 'reader', 'service', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:53:00.611030 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99999a30-6847-4250-afb5-9a49466db993 service nova] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:53:00.611318 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99999a30-6847-4250-afb5-9a49466db993 service nova] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:53:00.611594 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99999a30-6847-4250-afb5-9a49466db993 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:53:00.619732 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99999a30-6847-4250-afb5-9a49466db993 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:53:00.621330 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99999a30-6847-4250-afb5-9a49466db993 service nova] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:53:00.630855 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 720/1437] 10.222.0.30 () {60 vars in 1365 bytes} [Mon May 23 12:53:00 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 42 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) May 23 12:53:01.539995 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1553631f-222f-4147-8098-011ae4a85fd8 None None] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:53:01.540564 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1553631f-222f-4147-8098-011ae4a85fd8 None None] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:53:01.540564 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1553631f-222f-4147-8098-011ae4a85fd8 None None] PATH_INFO: `/` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:53:01.541944 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 718/1438] 127.0.0.1 () {54 vars in 901 bytes} [Mon May 23 12:53:01 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) May 23 12:53:01.547459 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cb694b8-70b4-41e1-8c27-b5d2334eccd5 None None] REQUEST_METHOD: `POST` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:53:01.547898 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cb694b8-70b4-41e1-8c27-b5d2334eccd5 None None] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:53:01.547999 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cb694b8-70b4-41e1-8c27-b5d2334eccd5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:53:01.566055 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.auth.core [None req-5cb694b8-70b4-41e1-8c27-b5d2334eccd5 None None] MFA Rules not processed for user `0f7119f8fa5c4445977671140ded91a4`. Rule list: `[]` (Enabled: `True`). {{(pid=92057) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 12:53:01.567239 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.fernet_utils [None req-5cb694b8-70b4-41e1-8c27-b5d2334eccd5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92057) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 12:53:01.574787 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 721/1439] 127.0.0.1 () {58 vars in 998 bytes} [Mon May 23 12:53:01 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 28 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) May 23 12:53:01.728098 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1553631f-222f-4147-8098-011ae4a85fd8 None None] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:53:01.743092 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1553631f-222f-4147-8098-011ae4a85fd8 None None] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:53:01.749752 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62b81eb2-b512-4154-b098-afa8543434b3 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cbe9931d94d44cc7ace0e83a7e5015fc', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9c389ae65aaa4542bb5a5c4fd0bacf5b', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'member', 'reader', 'service', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:53:01.751003 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62b81eb2-b512-4154-b098-afa8543434b3 service nova] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:53:01.751293 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62b81eb2-b512-4154-b098-afa8543434b3 service nova] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:53:01.751575 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62b81eb2-b512-4154-b098-afa8543434b3 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:53:01.760133 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-62b81eb2-b512-4154-b098-afa8543434b3 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:53:01.761723 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-62b81eb2-b512-4154-b098-afa8543434b3 service nova] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:53:01.771867 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 719/1440] 10.222.0.30 () {60 vars in 1365 bytes} [Mon May 23 12:53:01 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 44 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) May 23 12:53:03.842485 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5948c78a-0a06-4aea-8bd9-df6af735f2b0 None None] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:53:03.844234 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5948c78a-0a06-4aea-8bd9-df6af735f2b0 None None] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:53:03.844234 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5948c78a-0a06-4aea-8bd9-df6af735f2b0 None None] PATH_INFO: `/` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:53:03.844572 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 722/1441] 127.0.0.1 () {54 vars in 916 bytes} [Mon May 23 12:53:03 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) May 23 12:53:03.850102 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7792d68a-f17e-4f66-a14b-500c3e0160cb None None] REQUEST_METHOD: `POST` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:53:03.850507 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7792d68a-f17e-4f66-a14b-500c3e0160cb None None] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:53:03.850738 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7792d68a-f17e-4f66-a14b-500c3e0160cb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:53:03.868915 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.auth.core [None req-7792d68a-f17e-4f66-a14b-500c3e0160cb None None] MFA Rules not processed for user `0f7119f8fa5c4445977671140ded91a4`. Rule list: `[]` (Enabled: `True`). {{(pid=92056) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 12:53:03.870168 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.fernet_utils [None req-7792d68a-f17e-4f66-a14b-500c3e0160cb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92056) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 12:53:03.879095 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 720/1442] 127.0.0.1 () {58 vars in 1013 bytes} [Mon May 23 12:53:03 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 30 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) May 23 12:53:03.916998 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5948c78a-0a06-4aea-8bd9-df6af735f2b0 None None] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:53:03.928662 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5948c78a-0a06-4aea-8bd9-df6af735f2b0 None None] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:53:03.932382 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3de15a00-411c-458f-9a08-fc481f83bd6c service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2eb83ca38e79449095f3a2f947bbb06a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9c389ae65aaa4542bb5a5c4fd0bacf5b', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:53:03.933013 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3de15a00-411c-458f-9a08-fc481f83bd6c service neutron] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:53:03.933299 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3de15a00-411c-458f-9a08-fc481f83bd6c service neutron] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:53:03.933576 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3de15a00-411c-458f-9a08-fc481f83bd6c service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:53:03.943340 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3de15a00-411c-458f-9a08-fc481f83bd6c service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:53:03.944851 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3de15a00-411c-458f-9a08-fc481f83bd6c service neutron] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:53:03.954226 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 723/1443] 10.222.0.30 () {60 vars in 1365 bytes} [Mon May 23 12:53:03 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 38 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) May 23 12:53:06.291503 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-260b8991-6f62-4a16-8526-fe9e00f4cd94 None None] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:53:06.292014 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-260b8991-6f62-4a16-8526-fe9e00f4cd94 None None] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:53:06.292014 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-260b8991-6f62-4a16-8526-fe9e00f4cd94 None None] PATH_INFO: `/` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:53:06.293339 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 721/1444] 127.0.0.1 () {54 vars in 916 bytes} [Mon May 23 12:53:06 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) May 23 12:53:06.299167 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-233877e2-e011-461c-963f-a4c2872a9c06 None None] REQUEST_METHOD: `POST` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:53:06.299601 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-233877e2-e011-461c-963f-a4c2872a9c06 None None] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:53:06.299701 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-233877e2-e011-461c-963f-a4c2872a9c06 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:53:06.316965 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.auth.core [None req-233877e2-e011-461c-963f-a4c2872a9c06 None None] MFA Rules not processed for user `0f7119f8fa5c4445977671140ded91a4`. Rule list: `[]` (Enabled: `True`). {{(pid=92057) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 12:53:06.318187 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.fernet_utils [None req-233877e2-e011-461c-963f-a4c2872a9c06 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92057) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 12:53:06.325808 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 724/1445] 127.0.0.1 () {58 vars in 1013 bytes} [Mon May 23 12:53:06 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 28 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) May 23 12:53:06.363764 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-260b8991-6f62-4a16-8526-fe9e00f4cd94 None None] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:53:06.374363 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-260b8991-6f62-4a16-8526-fe9e00f4cd94 None None] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:53:06.377782 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ddd2cbc-5b69-4efc-a466-932dfbf3a36e service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2eb83ca38e79449095f3a2f947bbb06a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9c389ae65aaa4542bb5a5c4fd0bacf5b', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:53:06.378432 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ddd2cbc-5b69-4efc-a466-932dfbf3a36e service neutron] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:53:06.378715 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ddd2cbc-5b69-4efc-a466-932dfbf3a36e service neutron] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:53:06.378991 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ddd2cbc-5b69-4efc-a466-932dfbf3a36e service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:53:06.387632 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ddd2cbc-5b69-4efc-a466-932dfbf3a36e service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:53:06.389084 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ddd2cbc-5b69-4efc-a466-932dfbf3a36e service neutron] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:53:06.398178 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 722/1446] 10.222.0.30 () {60 vars in 1365 bytes} [Mon May 23 12:53:06 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 35 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) May 23 12:53:08.954273 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1941c78d-e6b7-41f7-abd5-da045ae058ff None None] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:53:08.954793 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1941c78d-e6b7-41f7-abd5-da045ae058ff None None] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:53:08.954793 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1941c78d-e6b7-41f7-abd5-da045ae058ff None None] PATH_INFO: `/` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:53:08.956179 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 725/1447] 127.0.0.1 () {54 vars in 916 bytes} [Mon May 23 12:53:08 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) May 23 12:53:08.961545 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4223e79-0284-4fc7-8a51-3be6258b4ab1 None None] REQUEST_METHOD: `POST` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:53:08.961986 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4223e79-0284-4fc7-8a51-3be6258b4ab1 None None] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:53:08.962151 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4223e79-0284-4fc7-8a51-3be6258b4ab1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:53:08.980418 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.auth.core [None req-d4223e79-0284-4fc7-8a51-3be6258b4ab1 None None] MFA Rules not processed for user `0f7119f8fa5c4445977671140ded91a4`. Rule list: `[]` (Enabled: `True`). {{(pid=92056) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 12:53:08.981593 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.fernet_utils [None req-d4223e79-0284-4fc7-8a51-3be6258b4ab1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92056) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 12:53:08.989889 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 723/1448] 127.0.0.1 () {58 vars in 1013 bytes} [Mon May 23 12:53:08 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 29 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) May 23 12:53:09.029071 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1941c78d-e6b7-41f7-abd5-da045ae058ff None None] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:53:09.039950 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1941c78d-e6b7-41f7-abd5-da045ae058ff None None] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:53:09.043492 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-246238ba-3c3b-4581-8da0-44f1834e6c91 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2eb83ca38e79449095f3a2f947bbb06a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9c389ae65aaa4542bb5a5c4fd0bacf5b', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:53:09.044123 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-246238ba-3c3b-4581-8da0-44f1834e6c91 service neutron] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:53:09.044411 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-246238ba-3c3b-4581-8da0-44f1834e6c91 service neutron] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:53:09.045202 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-246238ba-3c3b-4581-8da0-44f1834e6c91 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:53:09.054522 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-246238ba-3c3b-4581-8da0-44f1834e6c91 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:53:09.055959 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-246238ba-3c3b-4581-8da0-44f1834e6c91 service neutron] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:53:09.066448 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 726/1449] 10.222.0.30 () {60 vars in 1365 bytes} [Mon May 23 12:53:09 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 38 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) May 23 12:53:11.379284 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbfa431d-0e5b-4d24-88e9-ebd993cfff72 None None] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:53:11.379910 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbfa431d-0e5b-4d24-88e9-ebd993cfff72 None None] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:53:11.379910 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbfa431d-0e5b-4d24-88e9-ebd993cfff72 None None] PATH_INFO: `/` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:53:11.381199 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 724/1450] 127.0.0.1 () {54 vars in 916 bytes} [Mon May 23 12:53:11 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) May 23 12:53:11.386404 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80909e8f-4754-49f8-bec8-db3dce68cb9e None None] REQUEST_METHOD: `POST` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:53:11.386863 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80909e8f-4754-49f8-bec8-db3dce68cb9e None None] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:53:11.386975 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80909e8f-4754-49f8-bec8-db3dce68cb9e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:53:11.405870 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.auth.core [None req-80909e8f-4754-49f8-bec8-db3dce68cb9e None None] MFA Rules not processed for user `0f7119f8fa5c4445977671140ded91a4`. Rule list: `[]` (Enabled: `True`). {{(pid=92057) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 12:53:11.407073 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.fernet_utils [None req-80909e8f-4754-49f8-bec8-db3dce68cb9e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92057) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 12:53:11.414700 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 727/1451] 127.0.0.1 () {58 vars in 1013 bytes} [Mon May 23 12:53:11 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 29 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) May 23 12:53:11.453102 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cbfa431d-0e5b-4d24-88e9-ebd993cfff72 None None] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:53:11.463921 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cbfa431d-0e5b-4d24-88e9-ebd993cfff72 None None] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:53:11.467589 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-659cf153-5085-47df-968e-35778f7fb306 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2eb83ca38e79449095f3a2f947bbb06a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9c389ae65aaa4542bb5a5c4fd0bacf5b', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:53:11.468215 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-659cf153-5085-47df-968e-35778f7fb306 service neutron] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:53:11.468494 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-659cf153-5085-47df-968e-35778f7fb306 service neutron] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:53:11.468766 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-659cf153-5085-47df-968e-35778f7fb306 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:53:11.477352 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-659cf153-5085-47df-968e-35778f7fb306 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:53:11.478787 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-659cf153-5085-47df-968e-35778f7fb306 service neutron] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:53:11.488053 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 725/1452] 10.222.0.30 () {60 vars in 1365 bytes} [Mon May 23 12:53:11 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 35 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) May 23 12:53:13.785177 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fb9f67b-846b-4045-a6a8-f9775efb2652 None None] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:53:13.785749 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fb9f67b-846b-4045-a6a8-f9775efb2652 None None] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:53:13.785749 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fb9f67b-846b-4045-a6a8-f9775efb2652 None None] PATH_INFO: `/` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:53:13.787174 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 728/1453] 127.0.0.1 () {54 vars in 916 bytes} [Mon May 23 12:53:13 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) May 23 12:53:13.792331 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84081a9b-d9d1-44e8-9926-fd254ac83e47 None None] REQUEST_METHOD: `POST` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:53:13.792733 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84081a9b-d9d1-44e8-9926-fd254ac83e47 None None] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:53:13.792877 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84081a9b-d9d1-44e8-9926-fd254ac83e47 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:53:13.811610 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.auth.core [None req-84081a9b-d9d1-44e8-9926-fd254ac83e47 None None] MFA Rules not processed for user `0f7119f8fa5c4445977671140ded91a4`. Rule list: `[]` (Enabled: `True`). {{(pid=92056) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 12:53:13.812761 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.fernet_utils [None req-84081a9b-d9d1-44e8-9926-fd254ac83e47 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92056) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 12:53:13.820359 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 726/1454] 127.0.0.1 () {58 vars in 1013 bytes} [Mon May 23 12:53:13 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 29 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) May 23 12:53:13.858542 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0fb9f67b-846b-4045-a6a8-f9775efb2652 None None] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:53:13.860221 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.fernet_utils [None req-0fb9f67b-846b-4045-a6a8-f9775efb2652 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92057) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 12:53:13.873550 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0fb9f67b-846b-4045-a6a8-f9775efb2652 None None] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:53:13.877040 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-449903c4-af88-4893-a7a8-076fbb4ed5b8 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2eb83ca38e79449095f3a2f947bbb06a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9c389ae65aaa4542bb5a5c4fd0bacf5b', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:53:13.877670 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-449903c4-af88-4893-a7a8-076fbb4ed5b8 service neutron] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:53:13.878003 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-449903c4-af88-4893-a7a8-076fbb4ed5b8 service neutron] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:53:13.878279 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-449903c4-af88-4893-a7a8-076fbb4ed5b8 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:53:13.887237 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-449903c4-af88-4893-a7a8-076fbb4ed5b8 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:53:13.890789 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-449903c4-af88-4893-a7a8-076fbb4ed5b8 service neutron] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:53:13.898216 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 729/1455] 10.222.0.30 () {60 vars in 1365 bytes} [Mon May 23 12:53:13 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 41 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) May 23 12:53:16.146803 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-822e6e5e-642c-408c-b699-6b9d76392d5d None None] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:53:16.147360 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-822e6e5e-642c-408c-b699-6b9d76392d5d None None] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:53:16.147360 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-822e6e5e-642c-408c-b699-6b9d76392d5d None None] PATH_INFO: `/` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:53:16.148765 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 727/1456] 127.0.0.1 () {54 vars in 916 bytes} [Mon May 23 12:53:16 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) May 23 12:53:16.154872 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed641dea-b20f-4bac-9847-85464ab03f55 None None] REQUEST_METHOD: `POST` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:53:16.155284 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed641dea-b20f-4bac-9847-85464ab03f55 None None] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:53:16.155431 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed641dea-b20f-4bac-9847-85464ab03f55 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:53:16.173200 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.auth.core [None req-ed641dea-b20f-4bac-9847-85464ab03f55 None None] MFA Rules not processed for user `0f7119f8fa5c4445977671140ded91a4`. Rule list: `[]` (Enabled: `True`). {{(pid=92057) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 12:53:16.174424 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.fernet_utils [None req-ed641dea-b20f-4bac-9847-85464ab03f55 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92057) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 12:53:16.182371 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 730/1457] 127.0.0.1 () {58 vars in 1013 bytes} [Mon May 23 12:53:16 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 29 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) May 23 12:53:16.220182 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-822e6e5e-642c-408c-b699-6b9d76392d5d None None] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:53:16.230911 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-822e6e5e-642c-408c-b699-6b9d76392d5d None None] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:53:16.235209 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb3cfe9d-e576-463b-98e4-cde1a65ef744 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2eb83ca38e79449095f3a2f947bbb06a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9c389ae65aaa4542bb5a5c4fd0bacf5b', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:53:16.235843 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb3cfe9d-e576-463b-98e4-cde1a65ef744 service neutron] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:53:16.236122 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb3cfe9d-e576-463b-98e4-cde1a65ef744 service neutron] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:53:16.236395 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb3cfe9d-e576-463b-98e4-cde1a65ef744 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:53:16.245327 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb3cfe9d-e576-463b-98e4-cde1a65ef744 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:53:16.246788 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb3cfe9d-e576-463b-98e4-cde1a65ef744 service neutron] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:53:16.255926 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 728/1458] 10.222.0.30 () {60 vars in 1365 bytes} [Mon May 23 12:53:16 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 36 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) May 23 12:53:18.516188 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-417feef4-bd34-4493-9672-e04e423aa4e4 None None] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:53:18.516796 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-417feef4-bd34-4493-9672-e04e423aa4e4 None None] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:53:18.516796 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-417feef4-bd34-4493-9672-e04e423aa4e4 None None] PATH_INFO: `/` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:53:18.518146 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 731/1459] 127.0.0.1 () {54 vars in 916 bytes} [Mon May 23 12:53:18 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) May 23 12:53:18.523379 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ce63505-e692-45a8-b018-dabd03be0d84 None None] REQUEST_METHOD: `POST` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:53:18.523817 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ce63505-e692-45a8-b018-dabd03be0d84 None None] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:53:18.523970 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ce63505-e692-45a8-b018-dabd03be0d84 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:53:18.531010 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=70 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:53:18.542647 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.auth.core [None req-5ce63505-e692-45a8-b018-dabd03be0d84 None None] MFA Rules not processed for user `0f7119f8fa5c4445977671140ded91a4`. Rule list: `[]` (Enabled: `True`). {{(pid=92056) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 12:53:18.543826 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.fernet_utils [None req-5ce63505-e692-45a8-b018-dabd03be0d84 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92056) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 12:53:18.552428 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 729/1460] 127.0.0.1 () {58 vars in 1013 bytes} [Mon May 23 12:53:18 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 30 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) May 23 12:53:18.588990 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-417feef4-bd34-4493-9672-e04e423aa4e4 None None] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:53:18.600388 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-417feef4-bd34-4493-9672-e04e423aa4e4 None None] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:53:18.604078 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a6c2a61-bd39-4951-8c98-77256ebc4c10 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2eb83ca38e79449095f3a2f947bbb06a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9c389ae65aaa4542bb5a5c4fd0bacf5b', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:53:18.604707 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a6c2a61-bd39-4951-8c98-77256ebc4c10 service neutron] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:53:18.604986 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a6c2a61-bd39-4951-8c98-77256ebc4c10 service neutron] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:53:18.605260 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a6c2a61-bd39-4951-8c98-77256ebc4c10 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:53:18.611218 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=75 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:53:18.614725 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a6c2a61-bd39-4951-8c98-77256ebc4c10 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:53:18.616242 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a6c2a61-bd39-4951-8c98-77256ebc4c10 service neutron] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:53:18.625356 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 732/1461] 10.222.0.30 () {60 vars in 1365 bytes} [Mon May 23 12:53:18 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 37 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) May 23 12:53:20.885153 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffb37258-1c8d-495b-8e2c-bff4c065c27c None None] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:53:20.885750 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffb37258-1c8d-495b-8e2c-bff4c065c27c None None] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:53:20.885750 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffb37258-1c8d-495b-8e2c-bff4c065c27c None None] PATH_INFO: `/` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:53:20.887110 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 730/1462] 127.0.0.1 () {54 vars in 916 bytes} [Mon May 23 12:53:20 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) May 23 12:53:20.892711 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1470c6c-34fe-4ec0-9dfb-98bfe309a478 None None] REQUEST_METHOD: `POST` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:53:20.893134 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1470c6c-34fe-4ec0-9dfb-98bfe309a478 None None] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:53:20.893271 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1470c6c-34fe-4ec0-9dfb-98bfe309a478 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:53:20.911288 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.auth.core [None req-c1470c6c-34fe-4ec0-9dfb-98bfe309a478 None None] MFA Rules not processed for user `0f7119f8fa5c4445977671140ded91a4`. Rule list: `[]` (Enabled: `True`). {{(pid=92057) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 12:53:20.912475 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.fernet_utils [None req-c1470c6c-34fe-4ec0-9dfb-98bfe309a478 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92057) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 12:53:20.920700 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 733/1463] 127.0.0.1 () {58 vars in 1013 bytes} [Mon May 23 12:53:20 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 29 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) May 23 12:53:20.957007 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ffb37258-1c8d-495b-8e2c-bff4c065c27c None None] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:53:20.967595 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ffb37258-1c8d-495b-8e2c-bff4c065c27c None None] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:53:20.971057 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b3a0269-4467-4639-93c1-dc04a3a1689d service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2eb83ca38e79449095f3a2f947bbb06a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9c389ae65aaa4542bb5a5c4fd0bacf5b', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:53:20.971683 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b3a0269-4467-4639-93c1-dc04a3a1689d service neutron] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:53:20.971965 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b3a0269-4467-4639-93c1-dc04a3a1689d service neutron] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:53:20.972241 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b3a0269-4467-4639-93c1-dc04a3a1689d service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:53:20.980913 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b3a0269-4467-4639-93c1-dc04a3a1689d service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:53:20.983046 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b3a0269-4467-4639-93c1-dc04a3a1689d service neutron] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:53:20.992823 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 731/1464] 10.222.0.30 () {60 vars in 1365 bytes} [Mon May 23 12:53:20 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 36 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) May 23 12:53:23.196236 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-309838c6-b753-4f98-86c1-6169abec3e4c None None] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:53:23.196959 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-309838c6-b753-4f98-86c1-6169abec3e4c None None] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:53:23.196959 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-309838c6-b753-4f98-86c1-6169abec3e4c None None] PATH_INFO: `/` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:53:23.198189 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 734/1465] 127.0.0.1 () {54 vars in 916 bytes} [Mon May 23 12:53:23 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) May 23 12:53:23.203678 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68030d04-db68-464b-b014-e2a951f134e6 None None] REQUEST_METHOD: `POST` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:53:23.204140 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68030d04-db68-464b-b014-e2a951f134e6 None None] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:53:23.204246 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68030d04-db68-464b-b014-e2a951f134e6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:53:23.225047 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.auth.core [None req-68030d04-db68-464b-b014-e2a951f134e6 None None] MFA Rules not processed for user `0f7119f8fa5c4445977671140ded91a4`. Rule list: `[]` (Enabled: `True`). {{(pid=92056) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 12:53:23.226797 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.fernet_utils [None req-68030d04-db68-464b-b014-e2a951f134e6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92056) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 12:53:23.235985 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 732/1466] 127.0.0.1 () {58 vars in 1013 bytes} [Mon May 23 12:53:23 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 33 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) May 23 12:53:23.272557 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-309838c6-b753-4f98-86c1-6169abec3e4c None None] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:53:23.283756 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-309838c6-b753-4f98-86c1-6169abec3e4c None None] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:53:23.287276 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7223f1b4-6264-44d7-8595-a407bb05001c service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2eb83ca38e79449095f3a2f947bbb06a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9c389ae65aaa4542bb5a5c4fd0bacf5b', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:53:23.287909 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7223f1b4-6264-44d7-8595-a407bb05001c service neutron] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:53:23.288194 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7223f1b4-6264-44d7-8595-a407bb05001c service neutron] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:53:23.288471 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7223f1b4-6264-44d7-8595-a407bb05001c service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:53:23.297827 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7223f1b4-6264-44d7-8595-a407bb05001c service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:53:23.299282 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7223f1b4-6264-44d7-8595-a407bb05001c service neutron] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:53:23.308567 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 735/1467] 10.222.0.30 () {60 vars in 1365 bytes} [Mon May 23 12:53:23 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 36 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) May 23 12:53:25.512200 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b67bd882-a287-494a-8b36-b6c1373c17da None None] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:53:25.512831 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b67bd882-a287-494a-8b36-b6c1373c17da None None] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:53:25.512968 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b67bd882-a287-494a-8b36-b6c1373c17da None None] PATH_INFO: `/` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:53:25.514443 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 733/1468] 127.0.0.1 () {54 vars in 916 bytes} [Mon May 23 12:53:25 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) May 23 12:53:25.519743 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb3872de-2813-463c-9231-9ce623f83880 None None] REQUEST_METHOD: `POST` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:53:25.520206 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb3872de-2813-463c-9231-9ce623f83880 None None] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:53:25.520314 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb3872de-2813-463c-9231-9ce623f83880 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:53:25.537907 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.auth.core [None req-eb3872de-2813-463c-9231-9ce623f83880 None None] MFA Rules not processed for user `0f7119f8fa5c4445977671140ded91a4`. Rule list: `[]` (Enabled: `True`). {{(pid=92057) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 12:53:25.539097 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.fernet_utils [None req-eb3872de-2813-463c-9231-9ce623f83880 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92057) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 12:53:25.547400 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 736/1469] 127.0.0.1 () {58 vars in 1013 bytes} [Mon May 23 12:53:25 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 29 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) May 23 12:53:25.582963 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b67bd882-a287-494a-8b36-b6c1373c17da None None] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:53:25.593746 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b67bd882-a287-494a-8b36-b6c1373c17da None None] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:53:25.597237 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c0819d4-472e-4d58-9919-fc929b4d68ca service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2eb83ca38e79449095f3a2f947bbb06a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9c389ae65aaa4542bb5a5c4fd0bacf5b', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:53:25.597921 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c0819d4-472e-4d58-9919-fc929b4d68ca service neutron] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:53:25.598208 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c0819d4-472e-4d58-9919-fc929b4d68ca service neutron] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:53:25.598483 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c0819d4-472e-4d58-9919-fc929b4d68ca service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:53:25.607952 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c0819d4-472e-4d58-9919-fc929b4d68ca service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:53:25.609381 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c0819d4-472e-4d58-9919-fc929b4d68ca service neutron] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:53:25.618620 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 734/1470] 10.222.0.30 () {60 vars in 1365 bytes} [Mon May 23 12:53:25 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 36 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) May 23 12:53:35.532841 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=15 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:53:35.614534 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=17 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:53:58.871542 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de3c108d-b08c-4159-8a09-d0405c5f4819 None None] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:53:58.872175 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de3c108d-b08c-4159-8a09-d0405c5f4819 None None] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:53:58.872175 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de3c108d-b08c-4159-8a09-d0405c5f4819 None None] PATH_INFO: `/` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:53:58.873553 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 737/1471] 10.222.0.30 () {54 vars in 926 bytes} [Mon May 23 12:53:58 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) May 23 12:53:58.878995 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3319e362-7721-4ed1-bf07-ddada91596aa None None] REQUEST_METHOD: `POST` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:53:58.879437 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3319e362-7721-4ed1-bf07-ddada91596aa None None] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:53:58.879584 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3319e362-7721-4ed1-bf07-ddada91596aa None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:53:58.897161 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.auth.core [None req-3319e362-7721-4ed1-bf07-ddada91596aa None None] MFA Rules not processed for user `0f7119f8fa5c4445977671140ded91a4`. Rule list: `[]` (Enabled: `True`). {{(pid=92056) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 12:53:58.898385 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.fernet_utils [None req-3319e362-7721-4ed1-bf07-ddada91596aa None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92056) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 12:53:58.907109 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 735/1472] 10.222.0.30 () {58 vars in 1023 bytes} [Mon May 23 12:53:58 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 29 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) May 23 12:53:58.953504 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbfa8307-0611-44d8-b993-0e8b80b70826 None None] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:53:58.953789 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbfa8307-0611-44d8-b993-0e8b80b70826 None None] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:53:58.954092 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbfa8307-0611-44d8-b993-0e8b80b70826 None None] PATH_INFO: `/` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:53:58.955453 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 738/1473] 10.222.0.30 () {54 vars in 962 bytes} [Mon May 23 12:53:58 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) May 23 12:53:58.960296 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89f372fa-da52-4e77-88f3-a3fb972ac6bd None None] REQUEST_METHOD: `POST` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:53:58.960589 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89f372fa-da52-4e77-88f3-a3fb972ac6bd None None] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:53:58.960901 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89f372fa-da52-4e77-88f3-a3fb972ac6bd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:53:58.979802 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.auth.core [None req-89f372fa-da52-4e77-88f3-a3fb972ac6bd None None] MFA Rules not processed for user `5cea0feed4254ec085854986f99acf30`. Rule list: `[]` (Enabled: `True`). {{(pid=92056) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 12:53:58.981000 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.fernet_utils [None req-89f372fa-da52-4e77-88f3-a3fb972ac6bd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92056) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 12:53:58.987488 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 736/1474] 10.222.0.30 () {58 vars in 1059 bytes} [Mon May 23 12:53:58 2022] POST /identity/v3/auth/tokens => generated 3110 bytes in 28 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) May 23 12:53:58.994492 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dbfa8307-0611-44d8-b993-0e8b80b70826 None None] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:53:59.005639 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dbfa8307-0611-44d8-b993-0e8b80b70826 None None] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:53:59.009210 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34c923f5-cff5-47cd-98a5-57f9d741dd9f service glance] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '5cea0feed4254ec085854986f99acf30', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9c389ae65aaa4542bb5a5c4fd0bacf5b', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:53:59.009848 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34c923f5-cff5-47cd-98a5-57f9d741dd9f service glance] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:53:59.010199 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34c923f5-cff5-47cd-98a5-57f9d741dd9f service glance] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:53:59.010524 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34c923f5-cff5-47cd-98a5-57f9d741dd9f service glance] PATH_INFO: `/v3/auth/tokens` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:53:59.020607 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34c923f5-cff5-47cd-98a5-57f9d741dd9f service glance] RBAC: Authorizing `identity:validate_token()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:53:59.022086 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34c923f5-cff5-47cd-98a5-57f9d741dd9f service glance] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:53:59.032780 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 739/1475] 10.222.0.30 () {60 vars in 1365 bytes} [Mon May 23 12:53:58 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 39 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) May 23 12:54:01.358417 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-169f2971-7416-4b47-82a1-8629d0a21e0d None None] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:54:01.359046 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-169f2971-7416-4b47-82a1-8629d0a21e0d None None] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:54:01.359046 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-169f2971-7416-4b47-82a1-8629d0a21e0d None None] PATH_INFO: `/` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:54:01.360383 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 737/1476] 10.222.0.30 () {54 vars in 926 bytes} [Mon May 23 12:54:01 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) May 23 12:54:01.365552 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7543745e-84a8-4e55-ad47-98c6b5d9efff None None] REQUEST_METHOD: `POST` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:54:01.365982 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7543745e-84a8-4e55-ad47-98c6b5d9efff None None] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:54:01.366144 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7543745e-84a8-4e55-ad47-98c6b5d9efff None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:54:01.384766 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.auth.core [None req-7543745e-84a8-4e55-ad47-98c6b5d9efff None None] MFA Rules not processed for user `0f7119f8fa5c4445977671140ded91a4`. Rule list: `[]` (Enabled: `True`). {{(pid=92057) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 12:54:01.386191 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.fernet_utils [None req-7543745e-84a8-4e55-ad47-98c6b5d9efff None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92057) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 12:54:01.393769 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 740/1477] 10.222.0.30 () {58 vars in 1023 bytes} [Mon May 23 12:54:01 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 29 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) May 23 12:54:01.417047 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-169f2971-7416-4b47-82a1-8629d0a21e0d None None] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:54:01.429462 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-169f2971-7416-4b47-82a1-8629d0a21e0d None None] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:54:01.434063 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68fa4a3a-ea8b-48cc-824a-e3880a779165 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:54:01.434701 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68fa4a3a-ea8b-48cc-824a-e3880a779165 admin admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:54:01.434986 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68fa4a3a-ea8b-48cc-824a-e3880a779165 admin admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:54:01.435260 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68fa4a3a-ea8b-48cc-824a-e3880a779165 admin admin] PATH_INFO: `/v3/projects` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:54:01.435982 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68fa4a3a-ea8b-48cc-824a-e3880a779165 admin admin] RBAC: Authorizing `identity:list_projects()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:54:01.437768 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: WARNING py.warnings [None req-68fa4a3a-ea8b-48cc-824a-e3880a779165 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:list_projects": "(role:reader and system_scope:all) or (role:reader and domain_id:%(target.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required May 23 12:54:01.437768 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: warnings.warn(msg) May 23 12:54:01.437768 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: May 23 12:54:01.438935 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68fa4a3a-ea8b-48cc-824a-e3880a779165 admin admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:54:01.458227 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 738/1478] 10.222.0.30 () {56 vars in 1108 bytes} [Mon May 23 12:54:01 2022] GET /identity/v3/projects => generated 2553 bytes in 42 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) May 23 12:54:03.674116 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5a55292-8654-46a1-8507-792db17ca5c7 None None] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:54:03.674629 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5a55292-8654-46a1-8507-792db17ca5c7 None None] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:54:03.674629 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5a55292-8654-46a1-8507-792db17ca5c7 None None] PATH_INFO: `/` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:54:03.675942 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 741/1479] 10.222.0.30 () {54 vars in 926 bytes} [Mon May 23 12:54:03 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) May 23 12:54:03.681437 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7979c7bc-02ad-4922-be2f-210595a3ddfe None None] REQUEST_METHOD: `POST` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:54:03.682007 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7979c7bc-02ad-4922-be2f-210595a3ddfe None None] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:54:03.682698 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7979c7bc-02ad-4922-be2f-210595a3ddfe None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:54:03.700400 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.auth.core [None req-7979c7bc-02ad-4922-be2f-210595a3ddfe None None] MFA Rules not processed for user `0f7119f8fa5c4445977671140ded91a4`. Rule list: `[]` (Enabled: `True`). {{(pid=92056) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 12:54:03.701576 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.fernet_utils [None req-7979c7bc-02ad-4922-be2f-210595a3ddfe None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92056) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 12:54:03.709239 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 739/1480] 10.222.0.30 () {58 vars in 1023 bytes} [Mon May 23 12:54:03 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 29 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) May 23 12:54:03.752866 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5a55292-8654-46a1-8507-792db17ca5c7 None None] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:54:03.767309 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5a55292-8654-46a1-8507-792db17ca5c7 None None] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:54:03.772595 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6be2d9e-99a1-4c0b-8ffb-07d1f33a39ef service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cbe9931d94d44cc7ace0e83a7e5015fc', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9c389ae65aaa4542bb5a5c4fd0bacf5b', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'member', 'reader', 'service', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:54:03.773224 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6be2d9e-99a1-4c0b-8ffb-07d1f33a39ef service nova] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:54:03.773508 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6be2d9e-99a1-4c0b-8ffb-07d1f33a39ef service nova] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:54:03.773784 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6be2d9e-99a1-4c0b-8ffb-07d1f33a39ef service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:54:03.785057 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6be2d9e-99a1-4c0b-8ffb-07d1f33a39ef service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:54:03.786838 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6be2d9e-99a1-4c0b-8ffb-07d1f33a39ef service nova] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:54:03.797149 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 742/1481] 10.222.0.30 () {60 vars in 1365 bytes} [Mon May 23 12:54:03 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 45 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) May 23 12:54:06.154647 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-478c885d-e74b-4913-9df8-a2c3e884c071 None None] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:54:06.155177 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-478c885d-e74b-4913-9df8-a2c3e884c071 None None] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:54:06.155177 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-478c885d-e74b-4913-9df8-a2c3e884c071 None None] PATH_INFO: `/` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:54:06.156564 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 740/1482] 10.222.0.30 () {54 vars in 926 bytes} [Mon May 23 12:54:06 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) May 23 12:54:06.161706 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-922bce31-96a9-4c4d-a623-2325639d60c4 None None] REQUEST_METHOD: `POST` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:54:06.162173 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-922bce31-96a9-4c4d-a623-2325639d60c4 None None] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:54:06.162318 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-922bce31-96a9-4c4d-a623-2325639d60c4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:54:06.180461 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.auth.core [None req-922bce31-96a9-4c4d-a623-2325639d60c4 None None] MFA Rules not processed for user `0f7119f8fa5c4445977671140ded91a4`. Rule list: `[]` (Enabled: `True`). {{(pid=92057) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 12:54:06.181663 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.fernet_utils [None req-922bce31-96a9-4c4d-a623-2325639d60c4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92057) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 12:54:06.189975 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 743/1483] 10.222.0.30 () {58 vars in 1023 bytes} [Mon May 23 12:54:06 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 29 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) May 23 12:54:06.226177 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-478c885d-e74b-4913-9df8-a2c3e884c071 None None] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:54:06.237309 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-478c885d-e74b-4913-9df8-a2c3e884c071 None None] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:54:06.240791 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7deb2ebd-182e-4f31-a204-95e34a6b6259 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2eb83ca38e79449095f3a2f947bbb06a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9c389ae65aaa4542bb5a5c4fd0bacf5b', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:54:06.241421 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7deb2ebd-182e-4f31-a204-95e34a6b6259 service neutron] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:54:06.241708 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7deb2ebd-182e-4f31-a204-95e34a6b6259 service neutron] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:54:06.242015 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7deb2ebd-182e-4f31-a204-95e34a6b6259 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:54:06.251001 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7deb2ebd-182e-4f31-a204-95e34a6b6259 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:54:06.252436 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7deb2ebd-182e-4f31-a204-95e34a6b6259 service neutron] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:54:06.261870 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 741/1484] 10.222.0.30 () {60 vars in 1365 bytes} [Mon May 23 12:54:06 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 36 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) May 23 12:54:08.516283 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53a4b782-5723-4b6c-a3d1-cc8d17a90519 None None] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:54:08.516817 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53a4b782-5723-4b6c-a3d1-cc8d17a90519 None None] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:54:08.521632 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53a4b782-5723-4b6c-a3d1-cc8d17a90519 None None] PATH_INFO: `/` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:54:08.521632 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 744/1485] 10.222.0.30 () {54 vars in 926 bytes} [Mon May 23 12:54:08 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) May 23 12:54:08.523508 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76684150-7018-4259-9f54-a13b6b281c8d None None] REQUEST_METHOD: `POST` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:54:08.524201 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76684150-7018-4259-9f54-a13b6b281c8d None None] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:54:08.524384 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76684150-7018-4259-9f54-a13b6b281c8d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:54:08.542304 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.auth.core [None req-76684150-7018-4259-9f54-a13b6b281c8d None None] MFA Rules not processed for user `0f7119f8fa5c4445977671140ded91a4`. Rule list: `[]` (Enabled: `True`). {{(pid=92056) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 12:54:08.543537 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.fernet_utils [None req-76684150-7018-4259-9f54-a13b6b281c8d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92056) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 12:54:08.551499 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 742/1486] 10.222.0.30 () {58 vars in 1023 bytes} [Mon May 23 12:54:08 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 29 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) May 23 12:54:08.590355 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53a4b782-5723-4b6c-a3d1-cc8d17a90519 None None] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:54:08.600788 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53a4b782-5723-4b6c-a3d1-cc8d17a90519 None None] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:54:08.604430 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28463b2b-3528-438a-a988-e65ba25832d0 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2eb83ca38e79449095f3a2f947bbb06a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9c389ae65aaa4542bb5a5c4fd0bacf5b', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:54:08.605055 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28463b2b-3528-438a-a988-e65ba25832d0 service neutron] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:54:08.605335 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28463b2b-3528-438a-a988-e65ba25832d0 service neutron] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:54:08.605609 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28463b2b-3528-438a-a988-e65ba25832d0 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:54:08.614340 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28463b2b-3528-438a-a988-e65ba25832d0 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:54:08.615759 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28463b2b-3528-438a-a988-e65ba25832d0 service neutron] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:54:08.626004 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 745/1487] 10.222.0.30 () {60 vars in 1365 bytes} [Mon May 23 12:54:08 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 36 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) May 23 12:54:10.995148 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4854d32-5b81-41d9-a7c1-412898d36664 None None] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:54:10.995659 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4854d32-5b81-41d9-a7c1-412898d36664 None None] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:54:10.995659 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4854d32-5b81-41d9-a7c1-412898d36664 None None] PATH_INFO: `/` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:54:10.997022 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 743/1488] 10.222.0.30 () {54 vars in 926 bytes} [Mon May 23 12:54:10 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) May 23 12:54:11.002426 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d04bbb53-ad57-4595-90b5-8aafb158488b None None] REQUEST_METHOD: `POST` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:54:11.002864 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d04bbb53-ad57-4595-90b5-8aafb158488b None None] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:54:11.002987 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d04bbb53-ad57-4595-90b5-8aafb158488b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:54:11.021124 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.auth.core [None req-d04bbb53-ad57-4595-90b5-8aafb158488b None None] MFA Rules not processed for user `0f7119f8fa5c4445977671140ded91a4`. Rule list: `[]` (Enabled: `True`). {{(pid=92057) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 12:54:11.022334 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.fernet_utils [None req-d04bbb53-ad57-4595-90b5-8aafb158488b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92057) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 12:54:11.030445 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 746/1489] 10.222.0.30 () {58 vars in 1023 bytes} [Mon May 23 12:54:11 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 29 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) May 23 12:54:11.071240 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4854d32-5b81-41d9-a7c1-412898d36664 None None] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:54:11.083135 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4854d32-5b81-41d9-a7c1-412898d36664 None None] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:54:11.086575 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-466007e9-4694-45f3-a87a-b896b2905a30 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2eb83ca38e79449095f3a2f947bbb06a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9c389ae65aaa4542bb5a5c4fd0bacf5b', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:54:11.087203 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-466007e9-4694-45f3-a87a-b896b2905a30 service neutron] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:54:11.087483 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-466007e9-4694-45f3-a87a-b896b2905a30 service neutron] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:54:11.087760 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-466007e9-4694-45f3-a87a-b896b2905a30 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:54:11.096684 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-466007e9-4694-45f3-a87a-b896b2905a30 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:54:11.098135 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-466007e9-4694-45f3-a87a-b896b2905a30 service neutron] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:54:11.108020 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 744/1490] 10.222.0.30 () {60 vars in 1365 bytes} [Mon May 23 12:54:11 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 37 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) May 23 12:54:13.773338 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a976b18-d42c-4894-bfce-8a1c98e46883 None None] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:54:13.774212 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a976b18-d42c-4894-bfce-8a1c98e46883 None None] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:54:13.774212 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a976b18-d42c-4894-bfce-8a1c98e46883 None None] PATH_INFO: `/` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:54:13.775685 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 747/1491] 10.222.0.30 () {54 vars in 926 bytes} [Mon May 23 12:54:13 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) May 23 12:54:13.780995 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-815bff89-a0e2-4e60-8bce-0bba63edbe4d None None] REQUEST_METHOD: `POST` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:54:13.781436 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-815bff89-a0e2-4e60-8bce-0bba63edbe4d None None] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:54:13.781546 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-815bff89-a0e2-4e60-8bce-0bba63edbe4d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:54:13.799085 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.auth.core [None req-815bff89-a0e2-4e60-8bce-0bba63edbe4d None None] MFA Rules not processed for user `0f7119f8fa5c4445977671140ded91a4`. Rule list: `[]` (Enabled: `True`). {{(pid=92056) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 12:54:13.800273 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.fernet_utils [None req-815bff89-a0e2-4e60-8bce-0bba63edbe4d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92056) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 12:54:13.808643 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 745/1492] 10.222.0.30 () {58 vars in 1023 bytes} [Mon May 23 12:54:13 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 29 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) May 23 12:54:13.847268 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a976b18-d42c-4894-bfce-8a1c98e46883 None None] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:54:13.858050 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a976b18-d42c-4894-bfce-8a1c98e46883 None None] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:54:13.861571 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7bf124d9-4612-4198-b4f1-4133409a787f service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2eb83ca38e79449095f3a2f947bbb06a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9c389ae65aaa4542bb5a5c4fd0bacf5b', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:54:13.862234 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bf124d9-4612-4198-b4f1-4133409a787f service neutron] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:54:13.862515 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bf124d9-4612-4198-b4f1-4133409a787f service neutron] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:54:13.862792 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bf124d9-4612-4198-b4f1-4133409a787f service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:54:13.872320 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7bf124d9-4612-4198-b4f1-4133409a787f service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:54:13.877596 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7bf124d9-4612-4198-b4f1-4133409a787f service neutron] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:54:13.883167 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 748/1493] 10.222.0.30 () {60 vars in 1365 bytes} [Mon May 23 12:54:13 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 36 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) May 23 12:54:23.793977 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=33 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:54:23.879032 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=33 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:56:02.664046 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf18782a-64b5-4a70-a3ae-43590208dd95 None None] REQUEST_METHOD: `POST` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:56:02.664653 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf18782a-64b5-4a70-a3ae-43590208dd95 None None] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:56:02.664653 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf18782a-64b5-4a70-a3ae-43590208dd95 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:56:02.682591 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.auth.core [None req-bf18782a-64b5-4a70-a3ae-43590208dd95 None None] MFA Rules not processed for user `0f7119f8fa5c4445977671140ded91a4`. Rule list: `[]` (Enabled: `True`). {{(pid=92056) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 12:56:02.683791 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.fernet_utils [None req-bf18782a-64b5-4a70-a3ae-43590208dd95 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92056) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 12:56:02.691760 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 746/1494] 10.222.0.30 () {58 vars in 957 bytes} [Mon May 23 12:56:02 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 29 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) May 23 12:56:02.701060 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cf49625-cdc2-4c57-af83-b3bb3be12150 None None] REQUEST_METHOD: `POST` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:56:02.701537 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cf49625-cdc2-4c57-af83-b3bb3be12150 None None] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:56:02.701634 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cf49625-cdc2-4c57-af83-b3bb3be12150 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:56:02.719825 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.auth.core [None req-5cf49625-cdc2-4c57-af83-b3bb3be12150 None None] MFA Rules not processed for user `0f7119f8fa5c4445977671140ded91a4`. Rule list: `[]` (Enabled: `True`). {{(pid=92057) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 12:56:02.720992 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.fernet_utils [None req-5cf49625-cdc2-4c57-af83-b3bb3be12150 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92057) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 12:56:02.729272 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 749/1495] 10.222.0.30 () {58 vars in 957 bytes} [Mon May 23 12:56:02 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 29 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) May 23 12:56:02.733776 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf18782a-64b5-4a70-a3ae-43590208dd95 None None] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:56:02.743410 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf18782a-64b5-4a70-a3ae-43590208dd95 None None] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:56:02.747849 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57cac01e-a40f-4e7b-a645-fb21916ffd67 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:56:02.748492 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57cac01e-a40f-4e7b-a645-fb21916ffd67 admin admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:56:02.748775 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57cac01e-a40f-4e7b-a645-fb21916ffd67 admin admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:56:02.749054 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57cac01e-a40f-4e7b-a645-fb21916ffd67 admin admin] PATH_INFO: `/v3/domains` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:56:02.749770 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57cac01e-a40f-4e7b-a645-fb21916ffd67 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=92056) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} May 23 12:56:02.750112 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57cac01e-a40f-4e7b-a645-fb21916ffd67 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:56:02.751117 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: WARNING py.warnings [None req-57cac01e-a40f-4e7b-a645-fb21916ffd67 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:list_domains": "role:reader and system_scope:all" failed scope check. The token used to make the request was project scoped but the policy requires ['system'] scope. This behavior may change in the future where using the intended scope is required May 23 12:56:02.751117 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: warnings.warn(msg) May 23 12:56:02.751117 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: May 23 12:56:02.751995 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57cac01e-a40f-4e7b-a645-fb21916ffd67 admin admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:56:02.763621 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 747/1496] 10.222.0.30 () {58 vars in 1152 bytes} [Mon May 23 12:56:02 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) May 23 12:56:02.772462 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5cf49625-cdc2-4c57-af83-b3bb3be12150 None None] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:56:02.781652 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5cf49625-cdc2-4c57-af83-b3bb3be12150 None None] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:56:02.787094 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f72dedf-5f8b-4df5-9c8b-9d641799eee4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:56:02.787720 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f72dedf-5f8b-4df5-9c8b-9d641799eee4 admin admin] REQUEST_METHOD: `POST` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:56:02.788012 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f72dedf-5f8b-4df5-9c8b-9d641799eee4 admin admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:56:02.788289 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f72dedf-5f8b-4df5-9c8b-9d641799eee4 admin admin] PATH_INFO: `/v3/projects` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:56:02.788981 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f72dedf-5f8b-4df5-9c8b-9d641799eee4 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:56:02.790799 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: WARNING py.warnings [None req-5f72dedf-5f8b-4df5-9c8b-9d641799eee4 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:create_project": "(role:admin and system_scope:all) or (role:admin and domain_id:%(target.project.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required May 23 12:56:02.790799 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: warnings.warn(msg) May 23 12:56:02.790799 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: May 23 12:56:02.791981 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f72dedf-5f8b-4df5-9c8b-9d641799eee4 admin admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:56:02.931177 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 750/1497] 10.222.0.30 () {60 vars in 1152 bytes} [Mon May 23 12:56:02 2022] POST /identity/v3/projects => generated 379 bytes in 159 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) May 23 12:56:02.935324 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57cac01e-a40f-4e7b-a645-fb21916ffd67 admin admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:56:02.946497 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57cac01e-a40f-4e7b-a645-fb21916ffd67 admin admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:56:03.000791 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5da3bde0-aa91-48d3-a48c-6cd9e9d879a8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:56:03.001455 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5da3bde0-aa91-48d3-a48c-6cd9e9d879a8 admin admin] REQUEST_METHOD: `POST` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:56:03.001736 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5da3bde0-aa91-48d3-a48c-6cd9e9d879a8 admin admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:56:03.002042 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5da3bde0-aa91-48d3-a48c-6cd9e9d879a8 admin admin] PATH_INFO: `/v3/users` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:56:03.002726 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5da3bde0-aa91-48d3-a48c-6cd9e9d879a8 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:56:03.004251 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: WARNING py.warnings [None req-5da3bde0-aa91-48d3-a48c-6cd9e9d879a8 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:create_user": "(role:admin and system_scope:all) or (role:admin and token.domain.id:%(target.user.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required May 23 12:56:03.004251 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: warnings.warn(msg) May 23 12:56:03.004251 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: May 23 12:56:03.005500 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5da3bde0-aa91-48d3-a48c-6cd9e9d879a8 admin admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:56:03.072970 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 748/1498] 10.222.0.30 () {60 vars in 1143 bytes} [Mon May 23 12:56:02 2022] POST /identity/v3/users => generated 290 bytes in 138 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) May 23 12:56:03.077205 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f72dedf-5f8b-4df5-9c8b-9d641799eee4 admin admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:56:03.087979 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f72dedf-5f8b-4df5-9c8b-9d641799eee4 admin admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:56:03.092524 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7905fb61-d62c-4896-b378-e52b89b91642 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:56:03.093210 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7905fb61-d62c-4896-b378-e52b89b91642 admin admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:56:03.093494 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7905fb61-d62c-4896-b378-e52b89b91642 admin admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:56:03.093765 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7905fb61-d62c-4896-b378-e52b89b91642 admin admin] PATH_INFO: `/v3/roles` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:56:03.094507 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7905fb61-d62c-4896-b378-e52b89b91642 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:56:03.095757 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: WARNING py.warnings [None req-7905fb61-d62c-4896-b378-e52b89b91642 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:list_roles": "role:reader and system_scope:all" failed scope check. The token used to make the request was project scoped but the policy requires ['system'] scope. This behavior may change in the future where using the intended scope is required May 23 12:56:03.095757 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: warnings.warn(msg) May 23 12:56:03.095757 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: May 23 12:56:03.096430 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7905fb61-d62c-4896-b378-e52b89b91642 admin admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:56:03.109260 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 751/1499] 10.222.0.30 () {58 vars in 1121 bytes} [Mon May 23 12:56:03 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) May 23 12:56:03.113445 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5da3bde0-aa91-48d3-a48c-6cd9e9d879a8 admin admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:56:03.123191 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5da3bde0-aa91-48d3-a48c-6cd9e9d879a8 admin admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:56:03.127827 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ed138e5-43b6-484b-a6dc-26b28143f8b8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:56:03.129012 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ed138e5-43b6-484b-a6dc-26b28143f8b8 admin admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:56:03.129740 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ed138e5-43b6-484b-a6dc-26b28143f8b8 admin admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:56:03.130055 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ed138e5-43b6-484b-a6dc-26b28143f8b8 admin admin] PATH_INFO: `/v3/roles` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:56:03.130785 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ed138e5-43b6-484b-a6dc-26b28143f8b8 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:56:03.131792 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: WARNING py.warnings [None req-0ed138e5-43b6-484b-a6dc-26b28143f8b8 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:list_roles": "role:reader and system_scope:all" failed scope check. The token used to make the request was project scoped but the policy requires ['system'] scope. This behavior may change in the future where using the intended scope is required May 23 12:56:03.131792 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: warnings.warn(msg) May 23 12:56:03.131792 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: May 23 12:56:03.132686 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ed138e5-43b6-484b-a6dc-26b28143f8b8 admin admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:56:03.147324 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 749/1500] 10.222.0.30 () {58 vars in 1121 bytes} [Mon May 23 12:56:03 2022] GET /identity/v3/roles => generated 1420 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) May 23 12:56:03.151324 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7905fb61-d62c-4896-b378-e52b89b91642 admin admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:56:03.161309 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7905fb61-d62c-4896-b378-e52b89b91642 admin admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:56:03.166025 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6244c718-06ee-47c7-9510-c14853f31231 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:56:03.166902 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6244c718-06ee-47c7-9510-c14853f31231 admin admin] REQUEST_METHOD: `PUT` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:56:03.167183 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6244c718-06ee-47c7-9510-c14853f31231 admin admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:56:03.167457 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6244c718-06ee-47c7-9510-c14853f31231 admin admin] PATH_INFO: `/v3/projects/534bd14413ed45b2a5612013d3d8dc2c/users/560c0e83f5a842a189194e2b5e40eeca/roles/80fb22507c9a4ff09d44d8f12f6fff1a` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:56:03.182924 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6244c718-06ee-47c7-9510-c14853f31231 admin admin] RBAC: Authorizing `identity:create_grant(project_id=534bd14413ed45b2a5612013d3d8dc2c, user_id=560c0e83f5a842a189194e2b5e40eeca, role_id=80fb22507c9a4ff09d44d8f12f6fff1a)` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:56:03.183914 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: WARNING py.warnings [None req-6244c718-06ee-47c7-9510-c14853f31231 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:create_grant": "(role:admin and system_scope:all) or ((role:admin and domain_id:%(target.user.domain_id)s and domain_id:%(target.project.domain_id)s) or (role:admin and domain_id:%(target.user.domain_id)s and domain_id:%(target.domain.id)s) or (role:admin and domain_id:%(target.group.domain_id)s and domain_id:%(target.project.domain_id)s) or (role:admin and domain_id:%(target.group.domain_id)s and domain_id:%(target.domain.id)s)) and (domain_id:%(target.role.domain_id)s or None:%(target.role.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required May 23 12:56:03.183914 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: warnings.warn(msg) May 23 12:56:03.183914 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: May 23 12:56:03.185829 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6244c718-06ee-47c7-9510-c14853f31231 admin admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:56:03.284245 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 752/1501] 10.222.0.30 () {60 vars in 1482 bytes} [Mon May 23 12:56:03 2022] PUT /identity/v3/projects/534bd14413ed45b2a5612013d3d8dc2c/users/560c0e83f5a842a189194e2b5e40eeca/roles/80fb22507c9a4ff09d44d8f12f6fff1a => generated 0 bytes in 133 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) May 23 12:56:03.295438 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f419a05-8e3c-4efd-9a87-1cc96eff4c92 None None] REQUEST_METHOD: `POST` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:56:03.295728 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f419a05-8e3c-4efd-9a87-1cc96eff4c92 None None] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:56:03.296009 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f419a05-8e3c-4efd-9a87-1cc96eff4c92 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:56:03.323573 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.auth.core [None req-3f419a05-8e3c-4efd-9a87-1cc96eff4c92 None None] MFA Rules not processed for user `560c0e83f5a842a189194e2b5e40eeca`. Rule list: `[]` (Enabled: `True`). {{(pid=92056) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 12:56:03.324759 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.fernet_utils [None req-3f419a05-8e3c-4efd-9a87-1cc96eff4c92 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92056) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 12:56:03.387793 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 750/1502] 10.222.0.30 () {58 vars in 957 bytes} [Mon May 23 12:56:03 2022] POST /identity/v3/auth/tokens => generated 3246 bytes in 93 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) May 23 12:56:03.402557 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38ecc841-17a5-4786-87f2-fbc7bb1197f9 None None] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:56:03.402841 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38ecc841-17a5-4786-87f2-fbc7bb1197f9 None None] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:56:03.403117 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38ecc841-17a5-4786-87f2-fbc7bb1197f9 None None] PATH_INFO: `/` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:56:03.404545 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 753/1503] 10.222.0.30 () {54 vars in 963 bytes} [Mon May 23 12:56:03 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) May 23 12:56:03.409760 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7736ce7-5daf-4cf3-a1ef-eba684c8c95f None None] REQUEST_METHOD: `POST` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:56:03.410195 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7736ce7-5daf-4cf3-a1ef-eba684c8c95f None None] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:56:03.410479 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7736ce7-5daf-4cf3-a1ef-eba684c8c95f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:56:03.428524 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.auth.core [None req-b7736ce7-5daf-4cf3-a1ef-eba684c8c95f None None] MFA Rules not processed for user `1666097f8ed545c6b0e4809c93e4775a`. Rule list: `[]` (Enabled: `True`). {{(pid=92056) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 12:56:03.429695 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.fernet_utils [None req-b7736ce7-5daf-4cf3-a1ef-eba684c8c95f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92056) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 12:56:03.472304 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:56:03.479977 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 751/1504] 10.222.0.30 () {58 vars in 1060 bytes} [Mon May 23 12:56:03 2022] POST /identity/v3/auth/tokens => generated 3110 bytes in 71 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) May 23 12:56:03.487596 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38ecc841-17a5-4786-87f2-fbc7bb1197f9 None None] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:56:03.498819 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38ecc841-17a5-4786-87f2-fbc7bb1197f9 None None] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:56:03.502362 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a3819b4-11f3-455b-8aab-980e3222173a service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1666097f8ed545c6b0e4809c93e4775a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9c389ae65aaa4542bb5a5c4fd0bacf5b', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:56:03.502987 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a3819b4-11f3-455b-8aab-980e3222173a service cinder] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:56:03.503271 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a3819b4-11f3-455b-8aab-980e3222173a service cinder] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:56:03.504226 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a3819b4-11f3-455b-8aab-980e3222173a service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:56:03.512663 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a3819b4-11f3-455b-8aab-980e3222173a service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:56:03.514132 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a3819b4-11f3-455b-8aab-980e3222173a service cinder] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:56:03.523603 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 754/1505] 10.222.0.30 () {60 vars in 1365 bytes} [Mon May 23 12:56:03 2022] GET /identity/v3/auth/tokens => generated 3246 bytes in 36 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) May 23 12:56:03.566909 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a2c97be-2e39-4274-b626-47f0a4314e94 None None] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:56:03.567204 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a2c97be-2e39-4274-b626-47f0a4314e94 None None] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:56:03.567491 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a2c97be-2e39-4274-b626-47f0a4314e94 None None] PATH_INFO: `/` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:56:03.568801 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 752/1506] 10.222.0.30 () {52 vars in 829 bytes} [Mon May 23 12:56:03 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) May 23 12:56:03.572879 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a3819b4-11f3-455b-8aab-980e3222173a service cinder] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:56:03.643581 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a3819b4-11f3-455b-8aab-980e3222173a service cinder] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:56:03.649120 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81c32bd4-9f8d-406e-9a2f-1b1254bef92f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:56:03.650157 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81c32bd4-9f8d-406e-9a2f-1b1254bef92f admin admin] REQUEST_METHOD: `DELETE` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:56:03.650447 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81c32bd4-9f8d-406e-9a2f-1b1254bef92f admin admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:56:03.650728 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81c32bd4-9f8d-406e-9a2f-1b1254bef92f admin admin] PATH_INFO: `/v3/users/560c0e83f5a842a189194e2b5e40eeca` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:56:03.651976 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81c32bd4-9f8d-406e-9a2f-1b1254bef92f admin admin] RBAC: Authorizing `identity:delete_user(user_id=560c0e83f5a842a189194e2b5e40eeca)` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:56:03.653026 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: WARNING py.warnings [None req-81c32bd4-9f8d-406e-9a2f-1b1254bef92f admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:delete_user": "(role:admin and system_scope:all) or (role:admin and token.domain.id:%(target.user.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required May 23 12:56:03.653026 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: warnings.warn(msg) May 23 12:56:03.653026 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: May 23 12:56:03.654457 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81c32bd4-9f8d-406e-9a2f-1b1254bef92f admin admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:56:03.978051 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [None req-81c32bd4-9f8d-406e-9a2f-1b1254bef92f admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '560c0e83f5a842a189194e2b5e40eeca'} {{(pid=92057) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 12:56:04.037971 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [None req-81c32bd4-9f8d-406e-9a2f-1b1254bef92f admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '560c0e83f5a842a189194e2b5e40eeca'} {{(pid=92057) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 12:56:04.050253 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [None req-81c32bd4-9f8d-406e-9a2f-1b1254bef92f admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '560c0e83f5a842a189194e2b5e40eeca'} {{(pid=92057) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 12:56:04.050992 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [None req-81c32bd4-9f8d-406e-9a2f-1b1254bef92f admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '560c0e83f5a842a189194e2b5e40eeca'} {{(pid=92057) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 12:56:04.066911 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=88,keystone:INSERT=3,keystone:DELETE=9 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:56:04.069357 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [None req-81c32bd4-9f8d-406e-9a2f-1b1254bef92f admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '560c0e83f5a842a189194e2b5e40eeca'} {{(pid=92057) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 12:56:04.075227 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 755/1507] 10.222.0.30 () {58 vars in 1223 bytes} [Mon May 23 12:56:03 2022] DELETE /identity/v3/users/560c0e83f5a842a189194e2b5e40eeca => generated 0 bytes in 503 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) May 23 12:56:04.079701 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90478409-17a0-4d1c-ab4a-e85e35b09848 None None] REQUEST_METHOD: `POST` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:56:04.080180 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90478409-17a0-4d1c-ab4a-e85e35b09848 None None] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:56:04.080289 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90478409-17a0-4d1c-ab4a-e85e35b09848 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:56:04.098841 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.auth.core [None req-90478409-17a0-4d1c-ab4a-e85e35b09848 None None] MFA Rules not processed for user `0f7119f8fa5c4445977671140ded91a4`. Rule list: `[]` (Enabled: `True`). {{(pid=92056) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 12:56:04.100013 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.fernet_utils [None req-90478409-17a0-4d1c-ab4a-e85e35b09848 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92056) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 12:56:04.158482 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 753/1508] 10.222.0.30 () {58 vars in 957 bytes} [Mon May 23 12:56:04 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 80 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) May 23 12:56:04.168727 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81c32bd4-9f8d-406e-9a2f-1b1254bef92f admin admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:56:04.170251 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.fernet_utils [None req-81c32bd4-9f8d-406e-9a2f-1b1254bef92f admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92057) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 12:56:04.229715 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81c32bd4-9f8d-406e-9a2f-1b1254bef92f admin admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:56:04.233316 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7090502d-f109-45ed-95e1-b214e4723ff1 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2eb83ca38e79449095f3a2f947bbb06a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9c389ae65aaa4542bb5a5c4fd0bacf5b', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:56:04.234003 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7090502d-f109-45ed-95e1-b214e4723ff1 service neutron] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:56:04.234280 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7090502d-f109-45ed-95e1-b214e4723ff1 service neutron] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:56:04.234556 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7090502d-f109-45ed-95e1-b214e4723ff1 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:56:04.244685 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7090502d-f109-45ed-95e1-b214e4723ff1 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:56:04.246525 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7090502d-f109-45ed-95e1-b214e4723ff1 service neutron] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:56:04.256955 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 756/1509] 10.222.0.30 () {60 vars in 1365 bytes} [Mon May 23 12:56:04 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 89 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) May 23 12:56:04.720021 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90478409-17a0-4d1c-ab4a-e85e35b09848 None None] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:56:04.721763 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.fernet_utils [None req-90478409-17a0-4d1c-ab4a-e85e35b09848 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92056) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 12:56:04.733388 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90478409-17a0-4d1c-ab4a-e85e35b09848 None None] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:56:04.740065 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3be59663-4a1b-4c22-a359-6a4956f52024 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:56:04.741117 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3be59663-4a1b-4c22-a359-6a4956f52024 admin admin] REQUEST_METHOD: `DELETE` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:56:04.741423 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3be59663-4a1b-4c22-a359-6a4956f52024 admin admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:56:04.741702 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3be59663-4a1b-4c22-a359-6a4956f52024 admin admin] PATH_INFO: `/v3/projects/534bd14413ed45b2a5612013d3d8dc2c` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:56:04.742980 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3be59663-4a1b-4c22-a359-6a4956f52024 admin admin] RBAC: Authorizing `identity:delete_project(project_id=534bd14413ed45b2a5612013d3d8dc2c)` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:56:04.744045 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: WARNING py.warnings [None req-3be59663-4a1b-4c22-a359-6a4956f52024 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:delete_project": "(role:admin and system_scope:all) or (role:admin and domain_id:%(target.project.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required May 23 12:56:04.744045 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: warnings.warn(msg) May 23 12:56:04.744045 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: May 23 12:56:04.746335 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3be59663-4a1b-4c22-a359-6a4956f52024 admin admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:56:04.843364 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-3be59663-4a1b-4c22-a359-6a4956f52024 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '534bd14413ed45b2a5612013d3d8dc2c', 'request_id': 'req-3be59663-4a1b-4c22-a359-6a4956f52024'} {{(pid=92056) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 12:56:04.901006 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-3be59663-4a1b-4c22-a359-6a4956f52024 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '534bd14413ed45b2a5612013d3d8dc2c', 'request_id': 'req-3be59663-4a1b-4c22-a359-6a4956f52024'} {{(pid=92056) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 12:56:04.915140 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-3be59663-4a1b-4c22-a359-6a4956f52024 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '534bd14413ed45b2a5612013d3d8dc2c', 'request_id': 'req-3be59663-4a1b-4c22-a359-6a4956f52024'} {{(pid=92056) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 12:56:04.926579 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-3be59663-4a1b-4c22-a359-6a4956f52024 admin admin] The token cache is being invalidate because project 534bd14413ed45b2a5612013d3d8dc2c was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=92056) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} May 23 12:56:04.926863 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-3be59663-4a1b-4c22-a359-6a4956f52024 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=92056) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 12:56:04.927568 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-3be59663-4a1b-4c22-a359-6a4956f52024 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=92056) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 12:56:04.929225 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 754/1510] 10.222.0.30 () {58 vars in 1232 bytes} [Mon May 23 12:56:04 2022] DELETE /identity/v3/projects/534bd14413ed45b2a5612013d3d8dc2c => generated 0 bytes in 210 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) May 23 12:56:14.251993 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:DELETE=1,keystone:SELECT=28 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:56:14.923093 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=55,keystone:DELETE=7,keystone:INSERT=1 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:57:48.594825 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54ed23d1-76a2-4d13-a37e-6b373236d41d None None] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:57:48.595414 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54ed23d1-76a2-4d13-a37e-6b373236d41d None None] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:57:48.595414 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54ed23d1-76a2-4d13-a37e-6b373236d41d None None] PATH_INFO: `/` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:57:48.596879 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 757/1511] 127.0.0.1 () {54 vars in 901 bytes} [Mon May 23 12:57:48 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) May 23 12:57:48.601895 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c96cc05-2670-44f3-bcc5-7bf64568b8fd None None] REQUEST_METHOD: `POST` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:57:48.602322 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c96cc05-2670-44f3-bcc5-7bf64568b8fd None None] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:57:48.602466 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c96cc05-2670-44f3-bcc5-7bf64568b8fd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:57:48.620928 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.auth.core [None req-0c96cc05-2670-44f3-bcc5-7bf64568b8fd None None] MFA Rules not processed for user `0f7119f8fa5c4445977671140ded91a4`. Rule list: `[]` (Enabled: `True`). {{(pid=92056) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 12:57:48.622153 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.fernet_utils [None req-0c96cc05-2670-44f3-bcc5-7bf64568b8fd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92056) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 12:57:48.681032 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 755/1512] 127.0.0.1 () {58 vars in 998 bytes} [Mon May 23 12:57:48 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 80 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) May 23 12:57:48.848775 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54ed23d1-76a2-4d13-a37e-6b373236d41d None None] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:57:48.850569 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.fernet_utils [None req-54ed23d1-76a2-4d13-a37e-6b373236d41d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92057) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 12:57:48.925064 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54ed23d1-76a2-4d13-a37e-6b373236d41d None None] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:57:48.930451 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-933e6f4c-2e2e-44a7-b924-ffdcc22c97de service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cbe9931d94d44cc7ace0e83a7e5015fc', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9c389ae65aaa4542bb5a5c4fd0bacf5b', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'member', 'reader', 'service', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:57:48.931093 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-933e6f4c-2e2e-44a7-b924-ffdcc22c97de service nova] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:57:48.931375 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-933e6f4c-2e2e-44a7-b924-ffdcc22c97de service nova] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:57:48.931651 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-933e6f4c-2e2e-44a7-b924-ffdcc22c97de service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:57:48.939744 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-933e6f4c-2e2e-44a7-b924-ffdcc22c97de service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:57:48.941345 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-933e6f4c-2e2e-44a7-b924-ffdcc22c97de service nova] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:57:48.951241 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 758/1513] 10.222.0.30 () {60 vars in 1365 bytes} [Mon May 23 12:57:48 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 103 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) May 23 12:57:49.579055 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-136d79a8-6aa8-413d-b683-4a674ff9c136 None None] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:57:49.579463 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-136d79a8-6aa8-413d-b683-4a674ff9c136 None None] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:57:49.579742 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-136d79a8-6aa8-413d-b683-4a674ff9c136 None None] PATH_INFO: `/v3` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:57:49.581079 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 756/1514] 10.222.0.49 () {54 vars in 925 bytes} [Mon May 23 12:57:49 2022] GET /identity/v3 => generated 255 bytes in 3 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) May 23 12:57:49.587631 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fb704cc-ff5f-4fb6-a55c-0d45825eacbe None None] REQUEST_METHOD: `POST` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:57:49.587924 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fb704cc-ff5f-4fb6-a55c-0d45825eacbe None None] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:57:49.588204 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fb704cc-ff5f-4fb6-a55c-0d45825eacbe None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:57:49.608274 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.auth.core [None req-7fb704cc-ff5f-4fb6-a55c-0d45825eacbe None None] MFA Rules not processed for user `d5fb51f8d0fa46439cb29666970b885b`. Rule list: `[]` (Enabled: `True`). {{(pid=92057) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 12:57:49.609520 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.fernet_utils [None req-7fb704cc-ff5f-4fb6-a55c-0d45825eacbe None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92057) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 12:57:49.674549 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 759/1515] 10.222.0.49 () {58 vars in 1015 bytes} [Mon May 23 12:57:49 2022] POST /identity/v3/auth/tokens => generated 3298 bytes in 88 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) May 23 12:57:49.824642 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2dc57849-5c68-4446-bf12-dddbcc7634e9 None None] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:57:49.825201 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2dc57849-5c68-4446-bf12-dddbcc7634e9 None None] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:57:49.825201 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2dc57849-5c68-4446-bf12-dddbcc7634e9 None None] PATH_INFO: `/` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:57:49.827301 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 757/1516] 127.0.0.1 () {54 vars in 901 bytes} [Mon May 23 12:57:49 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) May 23 12:57:49.832161 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bfa7147-f6dc-4760-a7e9-05ba67f5cff0 None None] REQUEST_METHOD: `POST` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:57:49.832401 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bfa7147-f6dc-4760-a7e9-05ba67f5cff0 None None] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:57:49.832685 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bfa7147-f6dc-4760-a7e9-05ba67f5cff0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:57:49.851583 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.auth.core [None req-1bfa7147-f6dc-4760-a7e9-05ba67f5cff0 None None] MFA Rules not processed for user `0f7119f8fa5c4445977671140ded91a4`. Rule list: `[]` (Enabled: `True`). {{(pid=92057) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 12:57:49.852772 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.fernet_utils [None req-1bfa7147-f6dc-4760-a7e9-05ba67f5cff0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92057) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 12:57:49.864850 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 760/1517] 127.0.0.1 () {58 vars in 998 bytes} [Mon May 23 12:57:49 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) May 23 12:57:50.016486 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2dc57849-5c68-4446-bf12-dddbcc7634e9 None None] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:57:50.031082 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2dc57849-5c68-4446-bf12-dddbcc7634e9 None None] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:57:50.036379 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53e45e98-1e16-41d4-b35e-9f4df0956e03 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cbe9931d94d44cc7ace0e83a7e5015fc', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9c389ae65aaa4542bb5a5c4fd0bacf5b', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'member', 'reader', 'service', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:57:50.037007 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53e45e98-1e16-41d4-b35e-9f4df0956e03 service nova] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:57:50.037285 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53e45e98-1e16-41d4-b35e-9f4df0956e03 service nova] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:57:50.037560 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53e45e98-1e16-41d4-b35e-9f4df0956e03 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:57:50.045988 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53e45e98-1e16-41d4-b35e-9f4df0956e03 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:57:50.047591 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53e45e98-1e16-41d4-b35e-9f4df0956e03 service nova] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:57:50.056935 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 758/1518] 10.222.0.30 () {60 vars in 1365 bytes} [Mon May 23 12:57:50 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 41 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) May 23 12:57:59.268738 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f015362b-5f11-44f4-a831-7ba7674370e6 None None] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:57:59.269261 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f015362b-5f11-44f4-a831-7ba7674370e6 None None] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:57:59.269261 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f015362b-5f11-44f4-a831-7ba7674370e6 None None] PATH_INFO: `/` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:57:59.270680 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 761/1519] 10.222.0.30 () {54 vars in 960 bytes} [Mon May 23 12:57:59 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) May 23 12:57:59.275715 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b4d74c7-3541-4726-85fc-a1dae5297628 None None] REQUEST_METHOD: `POST` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:57:59.276156 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b4d74c7-3541-4726-85fc-a1dae5297628 None None] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:57:59.276264 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b4d74c7-3541-4726-85fc-a1dae5297628 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:57:59.293695 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.auth.core [None req-2b4d74c7-3541-4726-85fc-a1dae5297628 None None] MFA Rules not processed for user `d5fb51f8d0fa46439cb29666970b885b`. Rule list: `[]` (Enabled: `True`). {{(pid=92056) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 12:57:59.294966 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.fernet_utils [None req-2b4d74c7-3541-4726-85fc-a1dae5297628 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92056) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 12:57:59.303149 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 759/1520] 10.222.0.30 () {58 vars in 1057 bytes} [Mon May 23 12:57:59 2022] POST /identity/v3/auth/tokens => generated 3298 bytes in 28 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) May 23 12:57:59.311444 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f015362b-5f11-44f4-a831-7ba7674370e6 None None] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:57:59.324490 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f015362b-5f11-44f4-a831-7ba7674370e6 None None] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:57:59.330552 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53b35f34-5c5c-4f62-9b08-cbc9ea88381b service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd5fb51f8d0fa46439cb29666970b885b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9c389ae65aaa4542bb5a5c4fd0bacf5b', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'service', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:57:59.331174 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53b35f34-5c5c-4f62-9b08-cbc9ea88381b service placement] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:57:59.331459 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53b35f34-5c5c-4f62-9b08-cbc9ea88381b service placement] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:57:59.331734 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53b35f34-5c5c-4f62-9b08-cbc9ea88381b service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:57:59.340183 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53b35f34-5c5c-4f62-9b08-cbc9ea88381b service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:57:59.342219 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53b35f34-5c5c-4f62-9b08-cbc9ea88381b service placement] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:57:59.351738 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 762/1521] 10.222.0.30 () {60 vars in 1365 bytes} [Mon May 23 12:57:59 2022] GET /identity/v3/auth/tokens => generated 3298 bytes in 42 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) May 23 12:58:01.363921 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-419bd49d-3361-47f5-a78e-607d41da57d8 None None] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:58:01.364490 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-419bd49d-3361-47f5-a78e-607d41da57d8 None None] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:58:01.364490 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-419bd49d-3361-47f5-a78e-607d41da57d8 None None] PATH_INFO: `/` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:58:01.365837 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 760/1522] 127.0.0.1 () {54 vars in 901 bytes} [Mon May 23 12:58:01 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) May 23 12:58:01.370566 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0e68cf9-5914-40f6-a5df-4cf394d4c8b1 None None] REQUEST_METHOD: `POST` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:58:01.371013 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0e68cf9-5914-40f6-a5df-4cf394d4c8b1 None None] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:58:01.371119 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0e68cf9-5914-40f6-a5df-4cf394d4c8b1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:58:01.389900 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.auth.core [None req-b0e68cf9-5914-40f6-a5df-4cf394d4c8b1 None None] MFA Rules not processed for user `0f7119f8fa5c4445977671140ded91a4`. Rule list: `[]` (Enabled: `True`). {{(pid=92057) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 12:58:01.391098 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.fernet_utils [None req-b0e68cf9-5914-40f6-a5df-4cf394d4c8b1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92057) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 12:58:01.399077 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 763/1523] 127.0.0.1 () {58 vars in 998 bytes} [Mon May 23 12:58:01 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 30 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) May 23 12:58:01.561466 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-419bd49d-3361-47f5-a78e-607d41da57d8 None None] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:58:01.575581 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-419bd49d-3361-47f5-a78e-607d41da57d8 None None] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:58:01.581638 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d9b4d01-984b-4c83-9200-adf80b00a4b9 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cbe9931d94d44cc7ace0e83a7e5015fc', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9c389ae65aaa4542bb5a5c4fd0bacf5b', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'member', 'reader', 'service', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:58:01.582301 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d9b4d01-984b-4c83-9200-adf80b00a4b9 service nova] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:58:01.582584 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d9b4d01-984b-4c83-9200-adf80b00a4b9 service nova] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:58:01.582925 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d9b4d01-984b-4c83-9200-adf80b00a4b9 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:58:01.591125 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d9b4d01-984b-4c83-9200-adf80b00a4b9 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:58:01.593562 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d9b4d01-984b-4c83-9200-adf80b00a4b9 service nova] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:58:01.603478 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 761/1524] 10.222.0.30 () {60 vars in 1365 bytes} [Mon May 23 12:58:01 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 43 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) May 23 12:58:02.474426 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc7d7cb9-d56c-4302-b5ba-3792f86ad24c None None] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:58:02.474964 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc7d7cb9-d56c-4302-b5ba-3792f86ad24c None None] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:58:02.474964 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc7d7cb9-d56c-4302-b5ba-3792f86ad24c None None] PATH_INFO: `/` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:58:02.476299 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 764/1525] 127.0.0.1 () {54 vars in 901 bytes} [Mon May 23 12:58:02 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) May 23 12:58:02.481093 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c7ee3c9-b578-4c53-ad4e-1f0b3717e44a None None] REQUEST_METHOD: `POST` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:58:02.481534 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c7ee3c9-b578-4c53-ad4e-1f0b3717e44a None None] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:58:02.481655 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c7ee3c9-b578-4c53-ad4e-1f0b3717e44a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:58:02.502110 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.auth.core [None req-8c7ee3c9-b578-4c53-ad4e-1f0b3717e44a None None] MFA Rules not processed for user `0f7119f8fa5c4445977671140ded91a4`. Rule list: `[]` (Enabled: `True`). {{(pid=92056) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 12:58:02.503309 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.fernet_utils [None req-8c7ee3c9-b578-4c53-ad4e-1f0b3717e44a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92056) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 12:58:02.511100 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 762/1526] 127.0.0.1 () {58 vars in 998 bytes} [Mon May 23 12:58:02 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) May 23 12:58:02.661615 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc7d7cb9-d56c-4302-b5ba-3792f86ad24c None None] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:58:02.676423 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc7d7cb9-d56c-4302-b5ba-3792f86ad24c None None] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:58:02.681711 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0a5b947-b2cc-4ee3-9fe7-55965c740ef6 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cbe9931d94d44cc7ace0e83a7e5015fc', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9c389ae65aaa4542bb5a5c4fd0bacf5b', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'member', 'reader', 'service', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:58:02.682374 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0a5b947-b2cc-4ee3-9fe7-55965c740ef6 service nova] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:58:02.682656 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0a5b947-b2cc-4ee3-9fe7-55965c740ef6 service nova] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:58:02.682930 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0a5b947-b2cc-4ee3-9fe7-55965c740ef6 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:58:02.691994 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0a5b947-b2cc-4ee3-9fe7-55965c740ef6 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:58:02.693570 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0a5b947-b2cc-4ee3-9fe7-55965c740ef6 service nova] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:58:02.703420 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 765/1527] 10.222.0.30 () {60 vars in 1365 bytes} [Mon May 23 12:58:02 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 42 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) May 23 12:58:07.152077 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a3eaf19-ac74-4ff5-baca-d52af85078be None None] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:58:07.152711 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a3eaf19-ac74-4ff5-baca-d52af85078be None None] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:58:07.152711 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a3eaf19-ac74-4ff5-baca-d52af85078be None None] PATH_INFO: `/` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:58:07.154069 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 763/1528] 127.0.0.1 () {54 vars in 916 bytes} [Mon May 23 12:58:07 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) May 23 12:58:07.159934 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9907ba1-f2cd-4b70-b31f-a7f7b9da26dc None None] REQUEST_METHOD: `POST` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:58:07.160387 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9907ba1-f2cd-4b70-b31f-a7f7b9da26dc None None] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:58:07.160492 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9907ba1-f2cd-4b70-b31f-a7f7b9da26dc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:58:07.179199 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.auth.core [None req-e9907ba1-f2cd-4b70-b31f-a7f7b9da26dc None None] MFA Rules not processed for user `0f7119f8fa5c4445977671140ded91a4`. Rule list: `[]` (Enabled: `True`). {{(pid=92057) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 12:58:07.180381 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.fernet_utils [None req-e9907ba1-f2cd-4b70-b31f-a7f7b9da26dc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92057) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 12:58:07.188731 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 766/1529] 127.0.0.1 () {58 vars in 1013 bytes} [Mon May 23 12:58:07 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 30 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) May 23 12:58:07.225311 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a3eaf19-ac74-4ff5-baca-d52af85078be None None] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:58:07.227028 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.fernet_utils [None req-6a3eaf19-ac74-4ff5-baca-d52af85078be None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92056) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 12:58:07.285531 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a3eaf19-ac74-4ff5-baca-d52af85078be None None] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:58:07.289064 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b580a03-b29f-47a2-b14c-77ef1d86e830 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2eb83ca38e79449095f3a2f947bbb06a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9c389ae65aaa4542bb5a5c4fd0bacf5b', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:58:07.289696 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b580a03-b29f-47a2-b14c-77ef1d86e830 service neutron] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:58:07.290010 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b580a03-b29f-47a2-b14c-77ef1d86e830 service neutron] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:58:07.290296 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b580a03-b29f-47a2-b14c-77ef1d86e830 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:58:07.300195 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b580a03-b29f-47a2-b14c-77ef1d86e830 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:58:07.301639 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b580a03-b29f-47a2-b14c-77ef1d86e830 service neutron] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:58:07.311172 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 764/1530] 10.222.0.30 () {60 vars in 1365 bytes} [Mon May 23 12:58:07 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 86 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) May 23 12:58:09.493470 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f3a474d-1b00-4dca-be04-3ac3e6c36ad6 None None] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:58:09.494094 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f3a474d-1b00-4dca-be04-3ac3e6c36ad6 None None] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:58:09.494094 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f3a474d-1b00-4dca-be04-3ac3e6c36ad6 None None] PATH_INFO: `/` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:58:09.495461 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 767/1531] 127.0.0.1 () {54 vars in 916 bytes} [Mon May 23 12:58:09 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) May 23 12:58:09.500667 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a881398d-d307-4506-b20b-66f845035651 None None] REQUEST_METHOD: `POST` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:58:09.501125 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a881398d-d307-4506-b20b-66f845035651 None None] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:58:09.501233 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a881398d-d307-4506-b20b-66f845035651 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:58:09.519095 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.auth.core [None req-a881398d-d307-4506-b20b-66f845035651 None None] MFA Rules not processed for user `0f7119f8fa5c4445977671140ded91a4`. Rule list: `[]` (Enabled: `True`). {{(pid=92056) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 12:58:09.520268 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.fernet_utils [None req-a881398d-d307-4506-b20b-66f845035651 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92056) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 12:58:09.527840 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 765/1532] 127.0.0.1 () {58 vars in 1013 bytes} [Mon May 23 12:58:09 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 28 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) May 23 12:58:09.564559 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f3a474d-1b00-4dca-be04-3ac3e6c36ad6 None None] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:58:09.566406 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.fernet_utils [None req-1f3a474d-1b00-4dca-be04-3ac3e6c36ad6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92057) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 12:58:09.579088 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f3a474d-1b00-4dca-be04-3ac3e6c36ad6 None None] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:58:09.582565 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bde4de4a-b9cf-4d5c-a5ab-66da9e6f0214 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2eb83ca38e79449095f3a2f947bbb06a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9c389ae65aaa4542bb5a5c4fd0bacf5b', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:58:09.583195 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bde4de4a-b9cf-4d5c-a5ab-66da9e6f0214 service neutron] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:58:09.583477 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bde4de4a-b9cf-4d5c-a5ab-66da9e6f0214 service neutron] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:58:09.583754 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bde4de4a-b9cf-4d5c-a5ab-66da9e6f0214 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:58:09.592992 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bde4de4a-b9cf-4d5c-a5ab-66da9e6f0214 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:58:09.594537 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bde4de4a-b9cf-4d5c-a5ab-66da9e6f0214 service neutron] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:58:09.603734 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 768/1533] 10.222.0.30 () {60 vars in 1365 bytes} [Mon May 23 12:58:09 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 40 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) May 23 12:58:13.693064 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60d0d6d3-d99b-4255-837c-d61d426fe2cd None None] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:58:13.693645 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60d0d6d3-d99b-4255-837c-d61d426fe2cd None None] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:58:13.693753 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60d0d6d3-d99b-4255-837c-d61d426fe2cd None None] PATH_INFO: `/` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:58:13.695822 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 766/1534] 127.0.0.1 () {54 vars in 916 bytes} [Mon May 23 12:58:13 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) May 23 12:58:13.701602 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a037773c-e687-445b-8e6f-7ac9632148b0 None None] REQUEST_METHOD: `POST` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:58:13.703800 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a037773c-e687-445b-8e6f-7ac9632148b0 None None] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:58:13.703800 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a037773c-e687-445b-8e6f-7ac9632148b0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:58:13.720013 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.auth.core [None req-a037773c-e687-445b-8e6f-7ac9632148b0 None None] MFA Rules not processed for user `0f7119f8fa5c4445977671140ded91a4`. Rule list: `[]` (Enabled: `True`). {{(pid=92057) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 12:58:13.721227 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.fernet_utils [None req-a037773c-e687-445b-8e6f-7ac9632148b0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92057) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 12:58:13.729093 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 769/1535] 127.0.0.1 () {58 vars in 1013 bytes} [Mon May 23 12:58:13 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 29 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) May 23 12:58:14.005029 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60d0d6d3-d99b-4255-837c-d61d426fe2cd None None] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:58:14.006750 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.fernet_utils [None req-60d0d6d3-d99b-4255-837c-d61d426fe2cd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92056) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 12:58:14.065726 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60d0d6d3-d99b-4255-837c-d61d426fe2cd None None] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:58:14.069310 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-651706c5-55e6-426f-89a1-d8c313d12ee7 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1666097f8ed545c6b0e4809c93e4775a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9c389ae65aaa4542bb5a5c4fd0bacf5b', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:58:14.069980 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-651706c5-55e6-426f-89a1-d8c313d12ee7 service cinder] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:58:14.070259 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-651706c5-55e6-426f-89a1-d8c313d12ee7 service cinder] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:58:14.070529 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-651706c5-55e6-426f-89a1-d8c313d12ee7 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:58:14.079635 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-651706c5-55e6-426f-89a1-d8c313d12ee7 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:58:14.081203 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-651706c5-55e6-426f-89a1-d8c313d12ee7 service cinder] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:58:14.086282 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=100 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:58:14.090985 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 767/1536] 10.222.0.30 () {60 vars in 1365 bytes} [Mon May 23 12:58:14 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 86 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) May 23 12:58:16.392873 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f53e44a5-13ca-49c1-bc16-5a0acf920e25 None None] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:58:16.393408 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f53e44a5-13ca-49c1-bc16-5a0acf920e25 None None] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:58:16.393408 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f53e44a5-13ca-49c1-bc16-5a0acf920e25 None None] PATH_INFO: `/` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:58:16.394841 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 770/1537] 127.0.0.1 () {54 vars in 916 bytes} [Mon May 23 12:58:16 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) May 23 12:58:16.400263 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a401d7ac-3dd5-4bed-94e1-0905308ac194 None None] REQUEST_METHOD: `POST` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:58:16.400652 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a401d7ac-3dd5-4bed-94e1-0905308ac194 None None] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:58:16.400825 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a401d7ac-3dd5-4bed-94e1-0905308ac194 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:58:16.419611 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.auth.core [None req-a401d7ac-3dd5-4bed-94e1-0905308ac194 None None] MFA Rules not processed for user `0f7119f8fa5c4445977671140ded91a4`. Rule list: `[]` (Enabled: `True`). {{(pid=92056) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 12:58:16.421517 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.fernet_utils [None req-a401d7ac-3dd5-4bed-94e1-0905308ac194 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92056) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 12:58:16.429268 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 768/1538] 127.0.0.1 () {58 vars in 1013 bytes} [Mon May 23 12:58:16 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 30 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) May 23 12:58:16.704565 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f53e44a5-13ca-49c1-bc16-5a0acf920e25 None None] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:58:16.706284 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.fernet_utils [None req-f53e44a5-13ca-49c1-bc16-5a0acf920e25 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92057) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 12:58:16.714100 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=94 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:58:16.720953 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f53e44a5-13ca-49c1-bc16-5a0acf920e25 None None] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:58:16.724574 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf847e44-36c9-41df-b09f-d073437677a6 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1666097f8ed545c6b0e4809c93e4775a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9c389ae65aaa4542bb5a5c4fd0bacf5b', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:58:16.725330 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf847e44-36c9-41df-b09f-d073437677a6 service cinder] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:58:16.725616 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf847e44-36c9-41df-b09f-d073437677a6 service cinder] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:58:16.725927 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf847e44-36c9-41df-b09f-d073437677a6 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:58:16.735060 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf847e44-36c9-41df-b09f-d073437677a6 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:58:16.736525 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf847e44-36c9-41df-b09f-d073437677a6 service cinder] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:58:16.746552 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 771/1539] 10.222.0.30 () {60 vars in 1365 bytes} [Mon May 23 12:58:16 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 42 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) May 23 12:58:26.414339 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=3 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:58:26.746801 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=5 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:59:50.196561 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f373424e-c41b-45f9-96d7-8306e808cc05 None None] REQUEST_METHOD: `POST` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:59:50.197716 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f373424e-c41b-45f9-96d7-8306e808cc05 None None] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:59:50.198310 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f373424e-c41b-45f9-96d7-8306e808cc05 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:59:50.230677 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.auth.core [None req-f373424e-c41b-45f9-96d7-8306e808cc05 None None] MFA Rules not processed for user `0f7119f8fa5c4445977671140ded91a4`. Rule list: `[]` (Enabled: `True`). {{(pid=92056) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 12:59:50.232511 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.fernet_utils [None req-f373424e-c41b-45f9-96d7-8306e808cc05 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92056) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 12:59:50.241293 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 769/1540] 10.222.0.30 () {58 vars in 957 bytes} [Mon May 23 12:59:50 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 46 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) May 23 12:59:50.251560 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4de24fe-a788-43b4-aa05-2230ae552c75 None None] REQUEST_METHOD: `POST` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:59:50.257145 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4de24fe-a788-43b4-aa05-2230ae552c75 None None] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:59:50.257665 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4de24fe-a788-43b4-aa05-2230ae552c75 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:59:50.284974 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.auth.core [None req-d4de24fe-a788-43b4-aa05-2230ae552c75 None None] MFA Rules not processed for user `0f7119f8fa5c4445977671140ded91a4`. Rule list: `[]` (Enabled: `True`). {{(pid=92057) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 12:59:50.286222 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.fernet_utils [None req-d4de24fe-a788-43b4-aa05-2230ae552c75 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92057) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 12:59:50.305237 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 772/1541] 10.222.0.30 () {58 vars in 957 bytes} [Mon May 23 12:59:50 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 55 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) May 23 12:59:50.310814 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f373424e-c41b-45f9-96d7-8306e808cc05 None None] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:59:50.321968 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f373424e-c41b-45f9-96d7-8306e808cc05 None None] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:59:50.334538 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-403200b4-1518-4810-9c7d-983f69be7bd8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:59:50.335625 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-403200b4-1518-4810-9c7d-983f69be7bd8 admin admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:59:50.336139 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-403200b4-1518-4810-9c7d-983f69be7bd8 admin admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:59:50.336643 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-403200b4-1518-4810-9c7d-983f69be7bd8 admin admin] PATH_INFO: `/v3/domains` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:59:50.337667 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-403200b4-1518-4810-9c7d-983f69be7bd8 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=92056) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} May 23 12:59:50.338232 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-403200b4-1518-4810-9c7d-983f69be7bd8 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:59:50.340723 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-403200b4-1518-4810-9c7d-983f69be7bd8 admin admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:59:50.361280 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 770/1542] 10.222.0.30 () {58 vars in 1152 bytes} [Mon May 23 12:59:50 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 51 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) May 23 12:59:50.370582 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4de24fe-a788-43b4-aa05-2230ae552c75 None None] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:59:50.385299 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4de24fe-a788-43b4-aa05-2230ae552c75 None None] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:59:50.399396 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1fdc4dc2-2314-4704-8cf6-822953516a7e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:59:50.400085 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fdc4dc2-2314-4704-8cf6-822953516a7e admin admin] REQUEST_METHOD: `POST` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:59:50.400366 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fdc4dc2-2314-4704-8cf6-822953516a7e admin admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:59:50.400641 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fdc4dc2-2314-4704-8cf6-822953516a7e admin admin] PATH_INFO: `/v3/projects` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:59:50.401958 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1fdc4dc2-2314-4704-8cf6-822953516a7e admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:59:50.404384 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1fdc4dc2-2314-4704-8cf6-822953516a7e admin admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:59:50.512978 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 773/1543] 10.222.0.30 () {60 vars in 1152 bytes} [Mon May 23 12:59:50 2022] POST /identity/v3/projects => generated 371 bytes in 143 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) May 23 12:59:50.517909 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-403200b4-1518-4810-9c7d-983f69be7bd8 admin admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:59:50.532493 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-403200b4-1518-4810-9c7d-983f69be7bd8 admin admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:59:50.588537 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25ab131c-2ce8-4dab-9589-8260dbf8ebdd None None] REQUEST_METHOD: `POST` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:59:50.588537 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25ab131c-2ce8-4dab-9589-8260dbf8ebdd None None] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:59:50.588881 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25ab131c-2ce8-4dab-9589-8260dbf8ebdd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:59:50.628753 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07d8995e-0e39-4165-a333-0fa1f93b4c37 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:59:50.633690 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.auth.core [None req-25ab131c-2ce8-4dab-9589-8260dbf8ebdd None None] MFA Rules not processed for user `0f7119f8fa5c4445977671140ded91a4`. Rule list: `[]` (Enabled: `True`). {{(pid=92057) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 12:59:50.633835 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07d8995e-0e39-4165-a333-0fa1f93b4c37 admin admin] REQUEST_METHOD: `POST` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:59:50.633969 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.fernet_utils [None req-25ab131c-2ce8-4dab-9589-8260dbf8ebdd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92057) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 12:59:50.634056 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07d8995e-0e39-4165-a333-0fa1f93b4c37 admin admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:59:50.634056 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07d8995e-0e39-4165-a333-0fa1f93b4c37 admin admin] PATH_INFO: `/v3/users` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:59:50.634377 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07d8995e-0e39-4165-a333-0fa1f93b4c37 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:59:50.636748 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07d8995e-0e39-4165-a333-0fa1f93b4c37 admin admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:59:50.656267 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 774/1544] 10.222.0.30 () {58 vars in 957 bytes} [Mon May 23 12:59:50 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 70 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) May 23 12:59:50.657806 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-495af703-9724-4ae4-9c82-87caab1e3879 None None] REQUEST_METHOD: `POST` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:59:50.658136 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-495af703-9724-4ae4-9c82-87caab1e3879 None None] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:59:50.658934 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-495af703-9724-4ae4-9c82-87caab1e3879 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:59:50.698105 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.auth.core [None req-495af703-9724-4ae4-9c82-87caab1e3879 None None] MFA Rules not processed for user `0f7119f8fa5c4445977671140ded91a4`. Rule list: `[]` (Enabled: `True`). {{(pid=92057) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 12:59:50.699416 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.fernet_utils [None req-495af703-9724-4ae4-9c82-87caab1e3879 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92057) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 12:59:50.722833 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 775/1545] 10.222.0.30 () {58 vars in 957 bytes} [Mon May 23 12:59:50 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 66 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) May 23 12:59:50.725731 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b123003d-f63c-4d13-9e35-e67da6c7fbfc None None] REQUEST_METHOD: `POST` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:59:50.725731 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b123003d-f63c-4d13-9e35-e67da6c7fbfc None None] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:59:50.725731 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b123003d-f63c-4d13-9e35-e67da6c7fbfc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:59:50.740111 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 771/1546] 10.222.0.30 () {60 vars in 1143 bytes} [Mon May 23 12:59:50 2022] POST /identity/v3/users => generated 286 bytes in 223 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) May 23 12:59:50.742627 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d64bec5-ae86-4e4a-8967-3a4bf90637e9 None None] REQUEST_METHOD: `POST` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:59:50.743170 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d64bec5-ae86-4e4a-8967-3a4bf90637e9 None None] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:59:50.746468 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d64bec5-ae86-4e4a-8967-3a4bf90637e9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:59:50.758421 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.auth.core [None req-b123003d-f63c-4d13-9e35-e67da6c7fbfc None None] MFA Rules not processed for user `0f7119f8fa5c4445977671140ded91a4`. Rule list: `[]` (Enabled: `True`). {{(pid=92057) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 12:59:50.760113 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.fernet_utils [None req-b123003d-f63c-4d13-9e35-e67da6c7fbfc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92057) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 12:59:50.780411 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 776/1547] 10.222.0.30 () {58 vars in 957 bytes} [Mon May 23 12:59:50 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 57 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) May 23 12:59:50.783473 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b123003d-f63c-4d13-9e35-e67da6c7fbfc None None] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:59:50.790116 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.auth.core [None req-5d64bec5-ae86-4e4a-8967-3a4bf90637e9 None None] MFA Rules not processed for user `0f7119f8fa5c4445977671140ded91a4`. Rule list: `[]` (Enabled: `True`). {{(pid=92056) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 12:59:50.791549 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.fernet_utils [None req-5d64bec5-ae86-4e4a-8967-3a4bf90637e9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92056) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 12:59:50.800270 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b123003d-f63c-4d13-9e35-e67da6c7fbfc None None] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:59:50.806264 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 772/1548] 10.222.0.30 () {58 vars in 957 bytes} [Mon May 23 12:59:50 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 64 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) May 23 12:59:50.808227 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6d80c82-4fd1-4707-8cb7-c2241241dba2 None None] REQUEST_METHOD: `POST` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:59:50.808516 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6d80c82-4fd1-4707-8cb7-c2241241dba2 None None] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:59:50.808789 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6d80c82-4fd1-4707-8cb7-c2241241dba2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:59:50.811822 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c8902ea-4ffe-4d04-b2cf-7397aedaf0ac admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:59:50.812513 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c8902ea-4ffe-4d04-b2cf-7397aedaf0ac admin admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:59:50.812834 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c8902ea-4ffe-4d04-b2cf-7397aedaf0ac admin admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:59:50.813082 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c8902ea-4ffe-4d04-b2cf-7397aedaf0ac admin admin] PATH_INFO: `/v3/roles` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:59:50.816685 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c8902ea-4ffe-4d04-b2cf-7397aedaf0ac admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:59:50.816685 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c8902ea-4ffe-4d04-b2cf-7397aedaf0ac admin admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:59:50.829646 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 777/1549] 10.222.0.30 () {58 vars in 1121 bytes} [Mon May 23 12:59:50 2022] GET /identity/v3/roles => generated 1420 bytes in 49 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) May 23 12:59:50.836518 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2dd1df5-6493-462a-a85b-193ffd81a5c4 None None] REQUEST_METHOD: `POST` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:59:50.836518 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2dd1df5-6493-462a-a85b-193ffd81a5c4 None None] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:59:50.836518 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2dd1df5-6493-462a-a85b-193ffd81a5c4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:59:50.839668 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.auth.core [None req-e6d80c82-4fd1-4707-8cb7-c2241241dba2 None None] MFA Rules not processed for user `0f7119f8fa5c4445977671140ded91a4`. Rule list: `[]` (Enabled: `True`). {{(pid=92056) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 12:59:50.840856 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.fernet_utils [None req-e6d80c82-4fd1-4707-8cb7-c2241241dba2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92056) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 12:59:50.848877 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 773/1550] 10.222.0.30 () {58 vars in 957 bytes} [Mon May 23 12:59:50 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 42 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) May 23 12:59:50.854057 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6d80c82-4fd1-4707-8cb7-c2241241dba2 None None] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:59:50.856851 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.auth.core [None req-d2dd1df5-6493-462a-a85b-193ffd81a5c4 None None] MFA Rules not processed for user `0f7119f8fa5c4445977671140ded91a4`. Rule list: `[]` (Enabled: `True`). {{(pid=92057) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 12:59:50.858674 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.fernet_utils [None req-d2dd1df5-6493-462a-a85b-193ffd81a5c4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92057) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 12:59:50.861387 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6d80c82-4fd1-4707-8cb7-c2241241dba2 None None] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:59:50.866430 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ad2f16c-dc9d-4e66-a107-a3ae0dcf296a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:59:50.866595 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 778/1551] 10.222.0.30 () {58 vars in 957 bytes} [Mon May 23 12:59:50 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 36 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) May 23 12:59:50.867083 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ad2f16c-dc9d-4e66-a107-a3ae0dcf296a admin admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:59:50.867363 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ad2f16c-dc9d-4e66-a107-a3ae0dcf296a admin admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:59:50.867635 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ad2f16c-dc9d-4e66-a107-a3ae0dcf296a admin admin] PATH_INFO: `/v3/domains` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:59:50.867738 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3383798-a467-4167-8afd-4558fe89f5e9 None None] REQUEST_METHOD: `POST` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:59:50.868017 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3383798-a467-4167-8afd-4558fe89f5e9 None None] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:59:50.868291 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3383798-a467-4167-8afd-4558fe89f5e9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:59:50.868401 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ad2f16c-dc9d-4e66-a107-a3ae0dcf296a admin admin] RBAC: Adding query filter params (name=Default) {{(pid=92056) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} May 23 12:59:50.868643 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ad2f16c-dc9d-4e66-a107-a3ae0dcf296a admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:59:50.870433 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ad2f16c-dc9d-4e66-a107-a3ae0dcf296a admin admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:59:50.885982 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 774/1552] 10.222.0.30 () {58 vars in 1152 bytes} [Mon May 23 12:59:50 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) May 23 12:59:50.885982 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ad2f16c-dc9d-4e66-a107-a3ae0dcf296a admin admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:59:50.889349 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.auth.core [None req-b3383798-a467-4167-8afd-4558fe89f5e9 None None] MFA Rules not processed for user `0f7119f8fa5c4445977671140ded91a4`. Rule list: `[]` (Enabled: `True`). {{(pid=92057) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 12:59:50.889349 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.fernet_utils [None req-b3383798-a467-4167-8afd-4558fe89f5e9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92057) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 12:59:50.895506 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ad2f16c-dc9d-4e66-a107-a3ae0dcf296a admin admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:59:50.897815 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 779/1553] 10.222.0.30 () {58 vars in 957 bytes} [Mon May 23 12:59:50 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) May 23 12:59:50.899055 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4cacc498-872a-404b-9d3c-e5356c619fc1 None None] REQUEST_METHOD: `POST` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:59:50.899342 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4cacc498-872a-404b-9d3c-e5356c619fc1 None None] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:59:50.899617 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4cacc498-872a-404b-9d3c-e5356c619fc1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:59:50.905835 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26230245-c7ec-406b-b4d6-2dae11b7d286 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:59:50.907471 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26230245-c7ec-406b-b4d6-2dae11b7d286 admin admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:59:50.907986 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26230245-c7ec-406b-b4d6-2dae11b7d286 admin admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:59:50.908480 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26230245-c7ec-406b-b4d6-2dae11b7d286 admin admin] PATH_INFO: `/v3/domains` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:59:50.909789 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26230245-c7ec-406b-b4d6-2dae11b7d286 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=92056) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} May 23 12:59:50.910474 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26230245-c7ec-406b-b4d6-2dae11b7d286 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:59:50.912236 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26230245-c7ec-406b-b4d6-2dae11b7d286 admin admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:59:50.919212 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.auth.core [None req-4cacc498-872a-404b-9d3c-e5356c619fc1 None None] MFA Rules not processed for user `0f7119f8fa5c4445977671140ded91a4`. Rule list: `[]` (Enabled: `True`). {{(pid=92057) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 12:59:50.920389 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.fernet_utils [None req-4cacc498-872a-404b-9d3c-e5356c619fc1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92057) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 12:59:50.925135 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 775/1554] 10.222.0.30 () {58 vars in 1152 bytes} [Mon May 23 12:59:50 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 41 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) May 23 12:59:50.929807 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26230245-c7ec-406b-b4d6-2dae11b7d286 admin admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:59:50.930059 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 780/1555] 10.222.0.30 () {58 vars in 957 bytes} [Mon May 23 12:59:50 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 30 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) May 23 12:59:50.930059 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11eac204-b21c-44c7-8e38-953efc2b14df None None] REQUEST_METHOD: `POST` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:59:50.930320 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11eac204-b21c-44c7-8e38-953efc2b14df None None] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:59:50.930598 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11eac204-b21c-44c7-8e38-953efc2b14df None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:59:50.942133 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26230245-c7ec-406b-b4d6-2dae11b7d286 admin admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:59:50.946714 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8fb0f5ab-5b37-478d-9752-e5ab659c783a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:59:50.947370 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fb0f5ab-5b37-478d-9752-e5ab659c783a admin admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:59:50.947650 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fb0f5ab-5b37-478d-9752-e5ab659c783a admin admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:59:50.947968 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fb0f5ab-5b37-478d-9752-e5ab659c783a admin admin] PATH_INFO: `/v3/roles` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:59:50.948693 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8fb0f5ab-5b37-478d-9752-e5ab659c783a admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:59:50.950329 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8fb0f5ab-5b37-478d-9752-e5ab659c783a admin admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:59:50.953821 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.auth.core [None req-11eac204-b21c-44c7-8e38-953efc2b14df None None] MFA Rules not processed for user `0f7119f8fa5c4445977671140ded91a4`. Rule list: `[]` (Enabled: `True`). {{(pid=92057) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 12:59:50.955027 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.fernet_utils [None req-11eac204-b21c-44c7-8e38-953efc2b14df None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92057) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 12:59:50.960432 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 776/1556] 10.222.0.30 () {58 vars in 1121 bytes} [Mon May 23 12:59:50 2022] GET /identity/v3/roles => generated 1420 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) May 23 12:59:50.961748 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db169b13-19e9-4874-8971-438c8d8dc2f2 None None] REQUEST_METHOD: `POST` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:59:50.962133 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db169b13-19e9-4874-8971-438c8d8dc2f2 None None] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:59:50.962497 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db169b13-19e9-4874-8971-438c8d8dc2f2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:59:50.963105 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 781/1557] 10.222.0.30 () {58 vars in 957 bytes} [Mon May 23 12:59:50 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 34 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) May 23 12:59:50.966975 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a22c24bf-7afe-4717-ab11-4acbb97ee823 None None] REQUEST_METHOD: `POST` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:59:50.967521 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a22c24bf-7afe-4717-ab11-4acbb97ee823 None None] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:59:50.967807 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a22c24bf-7afe-4717-ab11-4acbb97ee823 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:59:50.982190 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.auth.core [None req-db169b13-19e9-4874-8971-438c8d8dc2f2 None None] MFA Rules not processed for user `0f7119f8fa5c4445977671140ded91a4`. Rule list: `[]` (Enabled: `True`). {{(pid=92056) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 12:59:50.983386 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.fernet_utils [None req-db169b13-19e9-4874-8971-438c8d8dc2f2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92056) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 12:59:50.987438 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.auth.core [None req-a22c24bf-7afe-4717-ab11-4acbb97ee823 None None] MFA Rules not processed for user `0f7119f8fa5c4445977671140ded91a4`. Rule list: `[]` (Enabled: `True`). {{(pid=92057) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 12:59:50.988595 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.fernet_utils [None req-a22c24bf-7afe-4717-ab11-4acbb97ee823 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92057) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 12:59:50.991147 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 777/1558] 10.222.0.30 () {58 vars in 957 bytes} [Mon May 23 12:59:50 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 30 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) May 23 12:59:50.992668 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db169b13-19e9-4874-8971-438c8d8dc2f2 None None] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:59:50.996425 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 782/1559] 10.222.0.30 () {58 vars in 957 bytes} [Mon May 23 12:59:50 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 33 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) May 23 12:59:50.997650 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49d43292-ebd0-4808-b636-a051ac05bbd0 None None] REQUEST_METHOD: `POST` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:59:50.997978 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49d43292-ebd0-4808-b636-a051ac05bbd0 None None] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:59:50.998266 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49d43292-ebd0-4808-b636-a051ac05bbd0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:59:51.003185 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db169b13-19e9-4874-8971-438c8d8dc2f2 None None] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:59:51.008512 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-facea75c-404e-4b8b-a529-630f4f4e8a16 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:59:51.009193 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-facea75c-404e-4b8b-a529-630f4f4e8a16 admin admin] REQUEST_METHOD: `POST` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:59:51.009477 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-facea75c-404e-4b8b-a529-630f4f4e8a16 admin admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:59:51.009853 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-facea75c-404e-4b8b-a529-630f4f4e8a16 admin admin] PATH_INFO: `/v3/projects` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:59:51.011197 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-facea75c-404e-4b8b-a529-630f4f4e8a16 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:59:51.012221 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: WARNING py.warnings [None req-facea75c-404e-4b8b-a529-630f4f4e8a16 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:create_project": "(role:admin and system_scope:all) or (role:admin and domain_id:%(target.project.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required May 23 12:59:51.012221 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: warnings.warn(msg) May 23 12:59:51.012221 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: May 23 12:59:51.013413 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-facea75c-404e-4b8b-a529-630f4f4e8a16 admin admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:59:51.016287 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.auth.core [None req-49d43292-ebd0-4808-b636-a051ac05bbd0 None None] MFA Rules not processed for user `0f7119f8fa5c4445977671140ded91a4`. Rule list: `[]` (Enabled: `True`). {{(pid=92057) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 12:59:51.017476 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.fernet_utils [None req-49d43292-ebd0-4808-b636-a051ac05bbd0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92057) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 12:59:51.025335 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 783/1560] 10.222.0.30 () {58 vars in 957 bytes} [Mon May 23 12:59:50 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 28 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) May 23 12:59:51.026777 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49d43292-ebd0-4808-b636-a051ac05bbd0 None None] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:59:51.040174 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49d43292-ebd0-4808-b636-a051ac05bbd0 None None] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:59:51.040604 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a89456a2-1810-4827-8678-9432141ef418 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:59:51.041237 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a89456a2-1810-4827-8678-9432141ef418 admin admin] REQUEST_METHOD: `POST` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:59:51.041523 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a89456a2-1810-4827-8678-9432141ef418 admin admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:59:51.041833 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a89456a2-1810-4827-8678-9432141ef418 admin admin] PATH_INFO: `/v3/projects` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:59:51.042551 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a89456a2-1810-4827-8678-9432141ef418 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:59:51.044469 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a89456a2-1810-4827-8678-9432141ef418 admin admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:59:51.106832 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 778/1561] 10.222.0.30 () {60 vars in 1152 bytes} [Mon May 23 12:59:50 2022] POST /identity/v3/projects => generated 379 bytes in 111 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) May 23 12:59:51.106832 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a295b43a-d62a-4154-825a-db0416926c22 None None] REQUEST_METHOD: `POST` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:59:51.106832 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a295b43a-d62a-4154-825a-db0416926c22 None None] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:59:51.106832 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a295b43a-d62a-4154-825a-db0416926c22 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:59:51.123815 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.auth.core [None req-a295b43a-d62a-4154-825a-db0416926c22 None None] MFA Rules not processed for user `0f7119f8fa5c4445977671140ded91a4`. Rule list: `[]` (Enabled: `True`). {{(pid=92056) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 12:59:51.125023 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.fernet_utils [None req-a295b43a-d62a-4154-825a-db0416926c22 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92056) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 12:59:51.132246 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 784/1562] 10.222.0.30 () {60 vars in 1152 bytes} [Mon May 23 12:59:51 2022] POST /identity/v3/projects => generated 403 bytes in 106 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) May 23 12:59:51.136093 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a89456a2-1810-4827-8678-9432141ef418 admin admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:59:51.179952 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:59:51.194348 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 779/1563] 10.222.0.30 () {58 vars in 957 bytes} [Mon May 23 12:59:51 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 91 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) May 23 12:59:51.198655 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93aa9b68-2766-41b4-a43a-b52b34181820 None None] REQUEST_METHOD: `POST` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:59:51.198655 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93aa9b68-2766-41b4-a43a-b52b34181820 None None] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:59:51.198655 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93aa9b68-2766-41b4-a43a-b52b34181820 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:59:51.199623 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a89456a2-1810-4827-8678-9432141ef418 admin admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:59:51.204060 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b81fc2a-d6d3-41e4-ba8e-f412be8a4165 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:59:51.204938 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b81fc2a-d6d3-41e4-ba8e-f412be8a4165 admin admin] REQUEST_METHOD: `PUT` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:59:51.205221 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b81fc2a-d6d3-41e4-ba8e-f412be8a4165 admin admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:59:51.205497 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b81fc2a-d6d3-41e4-ba8e-f412be8a4165 admin admin] PATH_INFO: `/v3/projects/e8e72a1080a44ad3a0ef7918563a2a82/users/bf73cd653d094781b1d2fa4e6d1ad8e9/roles/80fb22507c9a4ff09d44d8f12f6fff1a` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:59:51.216066 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.auth.core [None req-93aa9b68-2766-41b4-a43a-b52b34181820 None None] MFA Rules not processed for user `0f7119f8fa5c4445977671140ded91a4`. Rule list: `[]` (Enabled: `True`). {{(pid=92056) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 12:59:51.217334 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.fernet_utils [None req-93aa9b68-2766-41b4-a43a-b52b34181820 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92056) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 12:59:51.220937 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b81fc2a-d6d3-41e4-ba8e-f412be8a4165 admin admin] RBAC: Authorizing `identity:create_grant(project_id=e8e72a1080a44ad3a0ef7918563a2a82, user_id=bf73cd653d094781b1d2fa4e6d1ad8e9, role_id=80fb22507c9a4ff09d44d8f12f6fff1a)` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:59:51.230428 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b81fc2a-d6d3-41e4-ba8e-f412be8a4165 admin admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:59:51.230563 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 780/1564] 10.222.0.30 () {58 vars in 957 bytes} [Mon May 23 12:59:51 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) May 23 12:59:51.230563 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93aa9b68-2766-41b4-a43a-b52b34181820 None None] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:59:51.238396 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93aa9b68-2766-41b4-a43a-b52b34181820 None None] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:59:51.242987 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb1b2d96-819a-41de-9954-4efea9cb5ae9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:59:51.243652 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb1b2d96-819a-41de-9954-4efea9cb5ae9 admin admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:59:51.243941 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb1b2d96-819a-41de-9954-4efea9cb5ae9 admin admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:59:51.244222 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb1b2d96-819a-41de-9954-4efea9cb5ae9 admin admin] PATH_INFO: `/v3/domains` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:59:51.245078 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb1b2d96-819a-41de-9954-4efea9cb5ae9 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=92056) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} May 23 12:59:51.245386 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb1b2d96-819a-41de-9954-4efea9cb5ae9 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:59:51.247024 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb1b2d96-819a-41de-9954-4efea9cb5ae9 admin admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:59:51.258362 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 781/1565] 10.222.0.30 () {58 vars in 1152 bytes} [Mon May 23 12:59:51 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) May 23 12:59:51.267645 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb1b2d96-819a-41de-9954-4efea9cb5ae9 admin admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:59:51.269313 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb1b2d96-819a-41de-9954-4efea9cb5ae9 admin admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:59:51.273895 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9887e9d-9f04-40ef-ba08-67454ae39c88 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:59:51.274505 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9887e9d-9f04-40ef-ba08-67454ae39c88 admin admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:59:51.274792 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9887e9d-9f04-40ef-ba08-67454ae39c88 admin admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:59:51.275070 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9887e9d-9f04-40ef-ba08-67454ae39c88 admin admin] PATH_INFO: `/v3/domains` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:59:51.275763 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9887e9d-9f04-40ef-ba08-67454ae39c88 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=92056) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} May 23 12:59:51.276696 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9887e9d-9f04-40ef-ba08-67454ae39c88 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:59:51.278311 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9887e9d-9f04-40ef-ba08-67454ae39c88 admin admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:59:51.289805 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 782/1566] 10.222.0.30 () {58 vars in 1152 bytes} [Mon May 23 12:59:51 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) May 23 12:59:51.291063 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9887e9d-9f04-40ef-ba08-67454ae39c88 admin admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:59:51.300086 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9887e9d-9f04-40ef-ba08-67454ae39c88 admin admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:59:51.304595 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4780d12-e79d-41be-8f92-5121668b4a7a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:59:51.305292 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4780d12-e79d-41be-8f92-5121668b4a7a admin admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:59:51.305581 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4780d12-e79d-41be-8f92-5121668b4a7a admin admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:59:51.305858 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4780d12-e79d-41be-8f92-5121668b4a7a admin admin] PATH_INFO: `/v3/domains` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:59:51.306581 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4780d12-e79d-41be-8f92-5121668b4a7a admin admin] RBAC: Adding query filter params (name=Default) {{(pid=92056) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} May 23 12:59:51.306888 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4780d12-e79d-41be-8f92-5121668b4a7a admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:59:51.308476 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4780d12-e79d-41be-8f92-5121668b4a7a admin admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:59:51.319370 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 785/1567] 10.222.0.30 () {60 vars in 1482 bytes} [Mon May 23 12:59:51 2022] PUT /identity/v3/projects/e8e72a1080a44ad3a0ef7918563a2a82/users/bf73cd653d094781b1d2fa4e6d1ad8e9/roles/80fb22507c9a4ff09d44d8f12f6fff1a => generated 0 bytes in 187 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) May 23 12:59:51.332477 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b81fc2a-d6d3-41e4-ba8e-f412be8a4165 admin admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:59:51.332608 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 783/1568] 10.222.0.30 () {58 vars in 1152 bytes} [Mon May 23 12:59:51 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) May 23 12:59:51.332608 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4780d12-e79d-41be-8f92-5121668b4a7a admin admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:59:51.332608 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4780d12-e79d-41be-8f92-5121668b4a7a admin admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:59:51.357515 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:59:51.388370 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-148fb375-e8f5-4bb6-ae83-68539317c140 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:59:51.389297 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-148fb375-e8f5-4bb6-ae83-68539317c140 admin admin] REQUEST_METHOD: `POST` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:59:51.389696 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-148fb375-e8f5-4bb6-ae83-68539317c140 admin admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:59:51.390213 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-148fb375-e8f5-4bb6-ae83-68539317c140 admin admin] PATH_INFO: `/v3/users` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:59:51.391167 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-148fb375-e8f5-4bb6-ae83-68539317c140 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:59:51.393821 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-148fb375-e8f5-4bb6-ae83-68539317c140 admin admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:59:51.394539 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b81fc2a-d6d3-41e4-ba8e-f412be8a4165 admin admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:59:51.399589 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f288171-1593-44dd-94f4-8ca6932f00d7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:59:51.400277 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f288171-1593-44dd-94f4-8ca6932f00d7 admin admin] REQUEST_METHOD: `POST` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:59:51.400560 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f288171-1593-44dd-94f4-8ca6932f00d7 admin admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:59:51.401255 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f288171-1593-44dd-94f4-8ca6932f00d7 admin admin] PATH_INFO: `/v3/users` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:59:51.401983 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f288171-1593-44dd-94f4-8ca6932f00d7 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:59:51.403863 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: WARNING py.warnings [None req-6f288171-1593-44dd-94f4-8ca6932f00d7 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:create_user": "(role:admin and system_scope:all) or (role:admin and token.domain.id:%(target.user.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required May 23 12:59:51.403863 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: warnings.warn(msg) May 23 12:59:51.403863 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: May 23 12:59:51.405401 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f288171-1593-44dd-94f4-8ca6932f00d7 admin admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:59:51.525130 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 784/1569] 10.222.0.30 () {60 vars in 1143 bytes} [Mon May 23 12:59:51 2022] POST /identity/v3/users => generated 302 bytes in 204 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) May 23 12:59:51.526220 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-148fb375-e8f5-4bb6-ae83-68539317c140 admin admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:59:51.535757 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-148fb375-e8f5-4bb6-ae83-68539317c140 admin admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:59:51.540524 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6cf0121-6354-4121-8695-45860ad8b6b0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:59:51.541220 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6cf0121-6354-4121-8695-45860ad8b6b0 admin admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:59:51.541460 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6cf0121-6354-4121-8695-45860ad8b6b0 admin admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:59:51.541888 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6cf0121-6354-4121-8695-45860ad8b6b0 admin admin] PATH_INFO: `/v3/domains` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:59:51.542805 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 786/1570] 10.222.0.30 () {60 vars in 1143 bytes} [Mon May 23 12:59:51 2022] POST /identity/v3/users => generated 290 bytes in 223 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) May 23 12:59:51.542927 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6cf0121-6354-4121-8695-45860ad8b6b0 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=92056) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} May 23 12:59:51.542927 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6cf0121-6354-4121-8695-45860ad8b6b0 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:59:51.544530 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6cf0121-6354-4121-8695-45860ad8b6b0 admin admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:59:51.546330 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f288171-1593-44dd-94f4-8ca6932f00d7 admin admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:59:51.556887 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f288171-1593-44dd-94f4-8ca6932f00d7 admin admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:59:51.557347 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 785/1571] 10.222.0.30 () {58 vars in 1152 bytes} [Mon May 23 12:59:51 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) May 23 12:59:51.558037 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6cf0121-6354-4121-8695-45860ad8b6b0 admin admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:59:51.563737 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9dea5df3-7438-4a29-8a81-28aa86156ddc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:59:51.572759 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9dea5df3-7438-4a29-8a81-28aa86156ddc admin admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:59:51.572759 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9dea5df3-7438-4a29-8a81-28aa86156ddc admin admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:59:51.572759 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9dea5df3-7438-4a29-8a81-28aa86156ddc admin admin] PATH_INFO: `/v3/domains` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:59:51.572759 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9dea5df3-7438-4a29-8a81-28aa86156ddc admin admin] RBAC: Adding query filter params (name=Default) {{(pid=92057) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} May 23 12:59:51.572759 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9dea5df3-7438-4a29-8a81-28aa86156ddc admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:59:51.573250 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: WARNING py.warnings [None req-9dea5df3-7438-4a29-8a81-28aa86156ddc admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:list_domains": "role:reader and system_scope:all" failed scope check. The token used to make the request was project scoped but the policy requires ['system'] scope. This behavior may change in the future where using the intended scope is required May 23 12:59:51.573250 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: warnings.warn(msg) May 23 12:59:51.573250 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: May 23 12:59:51.573250 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9dea5df3-7438-4a29-8a81-28aa86156ddc admin admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:59:51.573565 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6cf0121-6354-4121-8695-45860ad8b6b0 admin admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:59:51.573565 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9213ebe-10ff-42c4-b416-e69c86db4b5c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:59:51.574122 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9213ebe-10ff-42c4-b416-e69c86db4b5c admin admin] REQUEST_METHOD: `POST` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:59:51.574500 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9213ebe-10ff-42c4-b416-e69c86db4b5c admin admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:59:51.575492 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9213ebe-10ff-42c4-b416-e69c86db4b5c admin admin] PATH_INFO: `/v3/projects` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:59:51.576187 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9213ebe-10ff-42c4-b416-e69c86db4b5c admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:59:51.578173 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9213ebe-10ff-42c4-b416-e69c86db4b5c admin admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:59:51.582219 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 787/1572] 10.222.0.30 () {58 vars in 1152 bytes} [Mon May 23 12:59:51 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 39 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) May 23 12:59:51.583018 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9dea5df3-7438-4a29-8a81-28aa86156ddc admin admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:59:51.592801 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9dea5df3-7438-4a29-8a81-28aa86156ddc admin admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:59:51.597502 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dfd8fd8a-f0c6-4359-ac30-e64fd98612e4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:59:51.598184 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfd8fd8a-f0c6-4359-ac30-e64fd98612e4 admin admin] REQUEST_METHOD: `POST` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:59:51.598469 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfd8fd8a-f0c6-4359-ac30-e64fd98612e4 admin admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:59:51.598745 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfd8fd8a-f0c6-4359-ac30-e64fd98612e4 admin admin] PATH_INFO: `/v3/projects` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:59:51.599423 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dfd8fd8a-f0c6-4359-ac30-e64fd98612e4 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:59:51.601360 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dfd8fd8a-f0c6-4359-ac30-e64fd98612e4 admin admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:59:51.692406 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 786/1573] 10.222.0.30 () {60 vars in 1152 bytes} [Mon May 23 12:59:51 2022] POST /identity/v3/projects => generated 385 bytes in 135 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) May 23 12:59:51.693401 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9213ebe-10ff-42c4-b416-e69c86db4b5c admin admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:59:51.712579 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 788/1574] 10.222.0.30 () {60 vars in 1152 bytes} [Mon May 23 12:59:51 2022] POST /identity/v3/projects => generated 379 bytes in 127 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) May 23 12:59:51.712579 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2f14dd1-7123-4ae6-98b0-a2ec674ed592 None None] REQUEST_METHOD: `POST` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:59:51.712579 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2f14dd1-7123-4ae6-98b0-a2ec674ed592 None None] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:59:51.712579 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2f14dd1-7123-4ae6-98b0-a2ec674ed592 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:59:51.737680 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.auth.core [None req-b2f14dd1-7123-4ae6-98b0-a2ec674ed592 None None] MFA Rules not processed for user `bf73cd653d094781b1d2fa4e6d1ad8e9`. Rule list: `[]` (Enabled: `True`). {{(pid=92057) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 12:59:51.738942 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.fernet_utils [None req-b2f14dd1-7123-4ae6-98b0-a2ec674ed592 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92057) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 12:59:51.760484 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9213ebe-10ff-42c4-b416-e69c86db4b5c admin admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:59:51.764965 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:59:51.806395 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 789/1575] 10.222.0.30 () {58 vars in 957 bytes} [Mon May 23 12:59:51 2022] POST /identity/v3/auth/tokens => generated 3238 bytes in 96 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) May 23 12:59:51.807433 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2f14dd1-7123-4ae6-98b0-a2ec674ed592 None None] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:59:51.818138 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4c5dfff-30cf-4345-af03-a6b2599f98e9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:59:51.818796 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4c5dfff-30cf-4345-af03-a6b2599f98e9 admin admin] REQUEST_METHOD: `POST` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:59:51.819081 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4c5dfff-30cf-4345-af03-a6b2599f98e9 admin admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:59:51.819358 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4c5dfff-30cf-4345-af03-a6b2599f98e9 admin admin] PATH_INFO: `/v3/projects` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:59:51.820046 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f4c5dfff-30cf-4345-af03-a6b2599f98e9 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:59:51.821984 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f4c5dfff-30cf-4345-af03-a6b2599f98e9 admin admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:59:51.870010 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2f14dd1-7123-4ae6-98b0-a2ec674ed592 None None] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:59:51.875068 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3023e7e-d656-4810-9670-2e566a70ff82 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:59:51.878217 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3023e7e-d656-4810-9670-2e566a70ff82 admin admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:59:51.878217 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3023e7e-d656-4810-9670-2e566a70ff82 admin admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:59:51.878217 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3023e7e-d656-4810-9670-2e566a70ff82 admin admin] PATH_INFO: `/v3/roles` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:59:51.878217 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a3023e7e-d656-4810-9670-2e566a70ff82 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:59:51.878598 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a3023e7e-d656-4810-9670-2e566a70ff82 admin admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:59:51.888137 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 790/1576] 10.222.0.30 () {58 vars in 1121 bytes} [Mon May 23 12:59:51 2022] GET /identity/v3/roles => generated 1420 bytes in 81 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) May 23 12:59:51.889162 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3023e7e-d656-4810-9670-2e566a70ff82 admin admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:59:51.894161 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:59:51.899097 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3023e7e-d656-4810-9670-2e566a70ff82 admin admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:59:51.903717 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba7c7a51-705b-4dbe-8972-14dafd33faf1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:59:51.904388 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba7c7a51-705b-4dbe-8972-14dafd33faf1 admin admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:59:51.904680 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba7c7a51-705b-4dbe-8972-14dafd33faf1 admin admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:59:51.904963 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba7c7a51-705b-4dbe-8972-14dafd33faf1 admin admin] PATH_INFO: `/v3/roles` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:59:51.905676 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba7c7a51-705b-4dbe-8972-14dafd33faf1 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:59:51.907298 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba7c7a51-705b-4dbe-8972-14dafd33faf1 admin admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:59:51.907415 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 787/1577] 10.222.0.30 () {60 vars in 1152 bytes} [Mon May 23 12:59:51 2022] POST /identity/v3/projects => generated 391 bytes in 214 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) May 23 12:59:51.908102 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4c5dfff-30cf-4345-af03-a6b2599f98e9 admin admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:59:51.920482 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 791/1578] 10.222.0.30 () {58 vars in 1121 bytes} [Mon May 23 12:59:51 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) May 23 12:59:51.920482 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba7c7a51-705b-4dbe-8972-14dafd33faf1 admin admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:59:51.929404 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba7c7a51-705b-4dbe-8972-14dafd33faf1 admin admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:59:51.974732 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4c5dfff-30cf-4345-af03-a6b2599f98e9 admin admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:59:51.979665 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5edac981-031b-4932-9d1f-21442170c7df admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:59:51.980396 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5edac981-031b-4932-9d1f-21442170c7df admin admin] REQUEST_METHOD: `POST` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:59:51.981322 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5edac981-031b-4932-9d1f-21442170c7df admin admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:59:51.981970 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5edac981-031b-4932-9d1f-21442170c7df admin admin] PATH_INFO: `/v3/projects` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:59:51.982553 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5edac981-031b-4932-9d1f-21442170c7df admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:59:51.984485 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5edac981-031b-4932-9d1f-21442170c7df admin admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:59:51.986987 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8b20e9f-2bb7-476a-ae6b-cb10f7c513a2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:59:51.987709 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8b20e9f-2bb7-476a-ae6b-cb10f7c513a2 admin admin] REQUEST_METHOD: `POST` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:59:51.987992 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8b20e9f-2bb7-476a-ae6b-cb10f7c513a2 admin admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:59:51.988266 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8b20e9f-2bb7-476a-ae6b-cb10f7c513a2 admin admin] PATH_INFO: `/v3/projects` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:59:51.988935 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8b20e9f-2bb7-476a-ae6b-cb10f7c513a2 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:59:51.990851 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8b20e9f-2bb7-476a-ae6b-cb10f7c513a2 admin admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:59:52.064116 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 788/1579] 10.222.0.30 () {60 vars in 1152 bytes} [Mon May 23 12:59:51 2022] POST /identity/v3/projects => generated 367 bytes in 156 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) May 23 12:59:52.064868 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5edac981-031b-4932-9d1f-21442170c7df admin admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:59:52.083236 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 792/1580] 10.222.0.30 () {60 vars in 1152 bytes} [Mon May 23 12:59:51 2022] POST /identity/v3/projects => generated 383 bytes in 162 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) May 23 12:59:52.083236 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8b20e9f-2bb7-476a-ae6b-cb10f7c513a2 admin admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:59:52.126672 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5edac981-031b-4932-9d1f-21442170c7df admin admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:59:52.146846 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:59:52.147492 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8b20e9f-2bb7-476a-ae6b-cb10f7c513a2 admin admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:59:52.152038 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-948aead3-04d4-440f-83a9-dd839a191081 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:59:52.153824 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-948aead3-04d4-440f-83a9-dd839a191081 admin admin] REQUEST_METHOD: `POST` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:59:52.159099 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-948aead3-04d4-440f-83a9-dd839a191081 admin admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:59:52.159099 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-948aead3-04d4-440f-83a9-dd839a191081 admin admin] PATH_INFO: `/v3/users` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:59:52.159099 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-948aead3-04d4-440f-83a9-dd839a191081 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:59:52.159099 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-948aead3-04d4-440f-83a9-dd839a191081 admin admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:59:52.183705 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0516616f-d9a2-498d-aeea-edb71b035c7a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:59:52.184371 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0516616f-d9a2-498d-aeea-edb71b035c7a admin admin] REQUEST_METHOD: `POST` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:59:52.184651 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0516616f-d9a2-498d-aeea-edb71b035c7a admin admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:59:52.184925 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0516616f-d9a2-498d-aeea-edb71b035c7a admin admin] PATH_INFO: `/v3/users` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:59:52.185609 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0516616f-d9a2-498d-aeea-edb71b035c7a admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:59:52.187556 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0516616f-d9a2-498d-aeea-edb71b035c7a admin admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:59:52.263817 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 793/1581] 10.222.0.30 () {60 vars in 1143 bytes} [Mon May 23 12:59:52 2022] POST /identity/v3/users => generated 290 bytes in 183 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) May 23 12:59:52.269437 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-948aead3-04d4-440f-83a9-dd839a191081 admin admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:59:52.274899 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-948aead3-04d4-440f-83a9-dd839a191081 admin admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:59:52.280765 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f79da7af-cda8-4d87-91fe-c29b7db2e07a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:59:52.281411 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f79da7af-cda8-4d87-91fe-c29b7db2e07a admin admin] REQUEST_METHOD: `POST` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:59:52.281701 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f79da7af-cda8-4d87-91fe-c29b7db2e07a admin admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:59:52.282013 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f79da7af-cda8-4d87-91fe-c29b7db2e07a admin admin] PATH_INFO: `/v3/projects` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:59:52.282696 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f79da7af-cda8-4d87-91fe-c29b7db2e07a admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:59:52.284625 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f79da7af-cda8-4d87-91fe-c29b7db2e07a admin admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:59:52.295126 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 789/1582] 10.222.0.30 () {60 vars in 1143 bytes} [Mon May 23 12:59:52 2022] POST /identity/v3/users => generated 293 bytes in 231 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) May 23 12:59:52.296037 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0516616f-d9a2-498d-aeea-edb71b035c7a admin admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:59:52.305815 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0516616f-d9a2-498d-aeea-edb71b035c7a admin admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:59:52.310629 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2392f67d-ecd2-4be5-97e4-47f45c3a78b5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:59:52.311290 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2392f67d-ecd2-4be5-97e4-47f45c3a78b5 admin admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:59:52.311590 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2392f67d-ecd2-4be5-97e4-47f45c3a78b5 admin admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:59:52.311866 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2392f67d-ecd2-4be5-97e4-47f45c3a78b5 admin admin] PATH_INFO: `/v3/roles` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:59:52.312588 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2392f67d-ecd2-4be5-97e4-47f45c3a78b5 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:59:52.314298 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2392f67d-ecd2-4be5-97e4-47f45c3a78b5 admin admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:59:52.324555 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 790/1583] 10.222.0.30 () {58 vars in 1121 bytes} [Mon May 23 12:59:52 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) May 23 12:59:52.325307 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2392f67d-ecd2-4be5-97e4-47f45c3a78b5 admin admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:59:52.334439 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2392f67d-ecd2-4be5-97e4-47f45c3a78b5 admin admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:59:52.338927 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-932fdf23-cf5c-4e1c-8c1d-bf911207156a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:59:52.339526 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-932fdf23-cf5c-4e1c-8c1d-bf911207156a admin admin] REQUEST_METHOD: `POST` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:59:52.339808 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-932fdf23-cf5c-4e1c-8c1d-bf911207156a admin admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:59:52.340663 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-932fdf23-cf5c-4e1c-8c1d-bf911207156a admin admin] PATH_INFO: `/v3/users` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:59:52.341480 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-932fdf23-cf5c-4e1c-8c1d-bf911207156a admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:59:52.343352 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-932fdf23-cf5c-4e1c-8c1d-bf911207156a admin admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:59:52.387085 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 794/1584] 10.222.0.30 () {60 vars in 1152 bytes} [Mon May 23 12:59:52 2022] POST /identity/v3/projects => generated 373 bytes in 122 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) May 23 12:59:52.387871 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f79da7af-cda8-4d87-91fe-c29b7db2e07a admin admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:59:52.433353 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:59:52.442885 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 791/1585] 10.222.0.30 () {60 vars in 1143 bytes} [Mon May 23 12:59:52 2022] POST /identity/v3/users => generated 296 bytes in 118 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) May 23 12:59:52.443752 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-932fdf23-cf5c-4e1c-8c1d-bf911207156a admin admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:59:52.449379 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f79da7af-cda8-4d87-91fe-c29b7db2e07a admin admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:59:52.454323 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-932fdf23-cf5c-4e1c-8c1d-bf911207156a admin admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:59:52.455006 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eec5efa9-fae3-4d99-92d4-596f314151ee admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:59:52.455683 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eec5efa9-fae3-4d99-92d4-596f314151ee admin admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:59:52.455966 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eec5efa9-fae3-4d99-92d4-596f314151ee admin admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:59:52.456254 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eec5efa9-fae3-4d99-92d4-596f314151ee admin admin] PATH_INFO: `/v3/roles` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:59:52.456969 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eec5efa9-fae3-4d99-92d4-596f314151ee admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:59:52.458625 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eec5efa9-fae3-4d99-92d4-596f314151ee admin admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:59:52.459194 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4e2605d-fc1c-402d-acad-4355e971b012 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:59:52.459849 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4e2605d-fc1c-402d-acad-4355e971b012 admin admin] REQUEST_METHOD: `POST` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:59:52.460131 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4e2605d-fc1c-402d-acad-4355e971b012 admin admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:59:52.460404 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4e2605d-fc1c-402d-acad-4355e971b012 admin admin] PATH_INFO: `/v3/users` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:59:52.461078 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f4e2605d-fc1c-402d-acad-4355e971b012 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:59:52.463009 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f4e2605d-fc1c-402d-acad-4355e971b012 admin admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:59:52.469259 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 795/1586] 10.222.0.30 () {58 vars in 1121 bytes} [Mon May 23 12:59:52 2022] GET /identity/v3/roles => generated 1420 bytes in 82 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) May 23 12:59:52.470095 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eec5efa9-fae3-4d99-92d4-596f314151ee admin admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:59:52.479710 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eec5efa9-fae3-4d99-92d4-596f314151ee admin admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:59:52.484954 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76a76e13-8d2a-40ce-92ff-8d7894cc50c9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:59:52.487503 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76a76e13-8d2a-40ce-92ff-8d7894cc50c9 admin admin] REQUEST_METHOD: `POST` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:59:52.487503 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76a76e13-8d2a-40ce-92ff-8d7894cc50c9 admin admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:59:52.487503 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76a76e13-8d2a-40ce-92ff-8d7894cc50c9 admin admin] PATH_INFO: `/v3/users` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:59:52.487503 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76a76e13-8d2a-40ce-92ff-8d7894cc50c9 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:59:52.488726 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76a76e13-8d2a-40ce-92ff-8d7894cc50c9 admin admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:59:52.541514 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 792/1587] 10.222.0.30 () {60 vars in 1143 bytes} [Mon May 23 12:59:52 2022] POST /identity/v3/users => generated 284 bytes in 98 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) May 23 12:59:52.542577 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4e2605d-fc1c-402d-acad-4355e971b012 admin admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:59:52.553063 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4e2605d-fc1c-402d-acad-4355e971b012 admin admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:59:52.557572 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cea3344a-96b8-4ce0-bcd3-6de994205a4b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:59:52.558245 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cea3344a-96b8-4ce0-bcd3-6de994205a4b admin admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:59:52.558529 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cea3344a-96b8-4ce0-bcd3-6de994205a4b admin admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:59:52.558966 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cea3344a-96b8-4ce0-bcd3-6de994205a4b admin admin] PATH_INFO: `/v3/roles` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:59:52.559979 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cea3344a-96b8-4ce0-bcd3-6de994205a4b admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:59:52.561577 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cea3344a-96b8-4ce0-bcd3-6de994205a4b admin admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:59:52.585447 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 796/1588] 10.222.0.30 () {60 vars in 1143 bytes} [Mon May 23 12:59:52 2022] POST /identity/v3/users => generated 292 bytes in 101 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) May 23 12:59:52.585447 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76a76e13-8d2a-40ce-92ff-8d7894cc50c9 admin admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:59:52.585718 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 793/1589] 10.222.0.30 () {58 vars in 1121 bytes} [Mon May 23 12:59:52 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) May 23 12:59:52.585718 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cea3344a-96b8-4ce0-bcd3-6de994205a4b admin admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:59:52.585916 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76a76e13-8d2a-40ce-92ff-8d7894cc50c9 admin admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:59:52.586015 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cea3344a-96b8-4ce0-bcd3-6de994205a4b admin admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:59:52.588591 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94b82196-f20a-4f32-a48b-05be0e829b21 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:59:52.589272 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94b82196-f20a-4f32-a48b-05be0e829b21 admin admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:59:52.589557 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94b82196-f20a-4f32-a48b-05be0e829b21 admin admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:59:52.589846 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94b82196-f20a-4f32-a48b-05be0e829b21 admin admin] PATH_INFO: `/v3/roles` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:59:52.591506 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94b82196-f20a-4f32-a48b-05be0e829b21 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:59:52.592551 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13f268fe-667d-4e78-9386-76ac0f1bafc6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:59:52.593126 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94b82196-f20a-4f32-a48b-05be0e829b21 admin admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:59:52.593459 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13f268fe-667d-4e78-9386-76ac0f1bafc6 admin admin] REQUEST_METHOD: `PUT` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:59:52.593739 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13f268fe-667d-4e78-9386-76ac0f1bafc6 admin admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:59:52.594052 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13f268fe-667d-4e78-9386-76ac0f1bafc6 admin admin] PATH_INFO: `/v3/projects/a840824e38be4e3aa8f0f3e09c3fee14/users/8ea87f0f4b434124a60994ad3eaf277c/roles/80fb22507c9a4ff09d44d8f12f6fff1a` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:59:52.604103 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 797/1590] 10.222.0.30 () {58 vars in 1121 bytes} [Mon May 23 12:59:52 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) May 23 12:59:52.604744 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94b82196-f20a-4f32-a48b-05be0e829b21 admin admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:59:52.610981 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13f268fe-667d-4e78-9386-76ac0f1bafc6 admin admin] RBAC: Authorizing `identity:create_grant(project_id=a840824e38be4e3aa8f0f3e09c3fee14, user_id=8ea87f0f4b434124a60994ad3eaf277c, role_id=80fb22507c9a4ff09d44d8f12f6fff1a)` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:59:52.612031 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: WARNING py.warnings [None req-13f268fe-667d-4e78-9386-76ac0f1bafc6 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:create_grant": "(role:admin and system_scope:all) or ((role:admin and domain_id:%(target.user.domain_id)s and domain_id:%(target.project.domain_id)s) or (role:admin and domain_id:%(target.user.domain_id)s and domain_id:%(target.domain.id)s) or (role:admin and domain_id:%(target.group.domain_id)s and domain_id:%(target.project.domain_id)s) or (role:admin and domain_id:%(target.group.domain_id)s and domain_id:%(target.domain.id)s)) and (domain_id:%(target.role.domain_id)s or None:%(target.role.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required May 23 12:59:52.612031 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: warnings.warn(msg) May 23 12:59:52.612031 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: May 23 12:59:52.614004 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13f268fe-667d-4e78-9386-76ac0f1bafc6 admin admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:59:52.614663 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94b82196-f20a-4f32-a48b-05be0e829b21 admin admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:59:52.620358 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2f94cd6-25fa-42e6-97e7-4857cc18a4c7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:59:52.621006 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2f94cd6-25fa-42e6-97e7-4857cc18a4c7 admin admin] REQUEST_METHOD: `POST` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:59:52.621317 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2f94cd6-25fa-42e6-97e7-4857cc18a4c7 admin admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:59:52.621665 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2f94cd6-25fa-42e6-97e7-4857cc18a4c7 admin admin] PATH_INFO: `/v3/users` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:59:52.622366 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f2f94cd6-25fa-42e6-97e7-4857cc18a4c7 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:59:52.624273 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f2f94cd6-25fa-42e6-97e7-4857cc18a4c7 admin admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:59:52.684880 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 794/1591] 10.222.0.30 () {60 vars in 1482 bytes} [Mon May 23 12:59:52 2022] PUT /identity/v3/projects/a840824e38be4e3aa8f0f3e09c3fee14/users/8ea87f0f4b434124a60994ad3eaf277c/roles/80fb22507c9a4ff09d44d8f12f6fff1a => generated 0 bytes in 110 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) May 23 12:59:52.685715 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13f268fe-667d-4e78-9386-76ac0f1bafc6 admin admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:59:52.713648 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 798/1592] 10.222.0.30 () {60 vars in 1143 bytes} [Mon May 23 12:59:52 2022] POST /identity/v3/users => generated 293 bytes in 109 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) May 23 12:59:52.714453 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2f94cd6-25fa-42e6-97e7-4857cc18a4c7 admin admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:59:52.724547 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2f94cd6-25fa-42e6-97e7-4857cc18a4c7 admin admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:59:52.751122 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13f268fe-667d-4e78-9386-76ac0f1bafc6 admin admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:59:52.755792 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-147c9802-58a7-429b-a4c6-1973dbebf3fa admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:59:52.756457 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-147c9802-58a7-429b-a4c6-1973dbebf3fa admin admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:59:52.756750 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-147c9802-58a7-429b-a4c6-1973dbebf3fa admin admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:59:52.757027 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-147c9802-58a7-429b-a4c6-1973dbebf3fa admin admin] PATH_INFO: `/v3/roles` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:59:52.757752 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-147c9802-58a7-429b-a4c6-1973dbebf3fa admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:59:52.759433 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-147c9802-58a7-429b-a4c6-1973dbebf3fa admin admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:59:52.770986 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 795/1593] 10.222.0.30 () {58 vars in 1121 bytes} [Mon May 23 12:59:52 2022] GET /identity/v3/roles => generated 1420 bytes in 86 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) May 23 12:59:52.771819 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-147c9802-58a7-429b-a4c6-1973dbebf3fa admin admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:59:52.782434 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-147c9802-58a7-429b-a4c6-1973dbebf3fa admin admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:59:52.785406 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29233acd-f0dc-479b-b0e8-d594261d44bf admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:59:52.786434 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29233acd-f0dc-479b-b0e8-d594261d44bf admin admin] REQUEST_METHOD: `PUT` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:59:52.786786 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29233acd-f0dc-479b-b0e8-d594261d44bf admin admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:59:52.787066 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29233acd-f0dc-479b-b0e8-d594261d44bf admin admin] PATH_INFO: `/v3/projects/9e3c84deee9a435a9c1e72e8408c18b4/users/a9b80726b3f14828b8815e30fc7e13c5/roles/80fb22507c9a4ff09d44d8f12f6fff1a` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:59:52.787185 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb8b97f0-9682-47e5-a731-3ff88a49bf78 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:59:52.787770 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb8b97f0-9682-47e5-a731-3ff88a49bf78 admin admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:59:52.788056 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb8b97f0-9682-47e5-a731-3ff88a49bf78 admin admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:59:52.788336 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb8b97f0-9682-47e5-a731-3ff88a49bf78 admin admin] PATH_INFO: `/v3/roles` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:59:52.789050 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb8b97f0-9682-47e5-a731-3ff88a49bf78 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:59:52.790684 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb8b97f0-9682-47e5-a731-3ff88a49bf78 admin admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:59:52.801265 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 796/1594] 10.222.0.30 () {58 vars in 1121 bytes} [Mon May 23 12:59:52 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) May 23 12:59:52.804339 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-29233acd-f0dc-479b-b0e8-d594261d44bf admin admin] RBAC: Authorizing `identity:create_grant(project_id=9e3c84deee9a435a9c1e72e8408c18b4, user_id=a9b80726b3f14828b8815e30fc7e13c5, role_id=80fb22507c9a4ff09d44d8f12f6fff1a)` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:59:52.804443 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb8b97f0-9682-47e5-a731-3ff88a49bf78 admin admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:59:52.804553 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-29233acd-f0dc-479b-b0e8-d594261d44bf admin admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:59:52.812760 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb8b97f0-9682-47e5-a731-3ff88a49bf78 admin admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:59:52.817477 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7851d96c-bfe5-4569-9572-0ff6d8f65480 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:59:52.820378 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7851d96c-bfe5-4569-9572-0ff6d8f65480 admin admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:59:52.820378 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7851d96c-bfe5-4569-9572-0ff6d8f65480 admin admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:59:52.820378 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7851d96c-bfe5-4569-9572-0ff6d8f65480 admin admin] PATH_INFO: `/v3/roles` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:59:52.820378 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7851d96c-bfe5-4569-9572-0ff6d8f65480 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:59:52.820984 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7851d96c-bfe5-4569-9572-0ff6d8f65480 admin admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:59:52.830895 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 797/1595] 10.222.0.30 () {58 vars in 1121 bytes} [Mon May 23 12:59:52 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) May 23 12:59:52.831663 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7851d96c-bfe5-4569-9572-0ff6d8f65480 admin admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:59:52.841527 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7851d96c-bfe5-4569-9572-0ff6d8f65480 admin admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:59:52.846021 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-595937bf-1368-4682-a0f0-5776ca097bb1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:59:52.846671 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-595937bf-1368-4682-a0f0-5776ca097bb1 admin admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:59:52.846954 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-595937bf-1368-4682-a0f0-5776ca097bb1 admin admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:59:52.847233 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-595937bf-1368-4682-a0f0-5776ca097bb1 admin admin] PATH_INFO: `/v3/roles` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:59:52.847933 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-595937bf-1368-4682-a0f0-5776ca097bb1 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:59:52.849478 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-595937bf-1368-4682-a0f0-5776ca097bb1 admin admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:59:52.850745 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=90,keystone:INSERT=10 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:59:52.863515 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 798/1596] 10.222.0.30 () {58 vars in 1121 bytes} [Mon May 23 12:59:52 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) May 23 12:59:52.863515 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-595937bf-1368-4682-a0f0-5776ca097bb1 admin admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:59:52.870980 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-595937bf-1368-4682-a0f0-5776ca097bb1 admin admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:59:52.875582 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22415983-63ae-4fc8-bda9-e75c0ffdff41 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:59:52.876198 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22415983-63ae-4fc8-bda9-e75c0ffdff41 admin admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:59:52.876484 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22415983-63ae-4fc8-bda9-e75c0ffdff41 admin admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:59:52.876777 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22415983-63ae-4fc8-bda9-e75c0ffdff41 admin admin] PATH_INFO: `/v3/roles` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:59:52.877476 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-22415983-63ae-4fc8-bda9-e75c0ffdff41 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:59:52.879092 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-22415983-63ae-4fc8-bda9-e75c0ffdff41 admin admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:59:52.884808 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 799/1597] 10.222.0.30 () {60 vars in 1482 bytes} [Mon May 23 12:59:52 2022] PUT /identity/v3/projects/9e3c84deee9a435a9c1e72e8408c18b4/users/a9b80726b3f14828b8815e30fc7e13c5/roles/80fb22507c9a4ff09d44d8f12f6fff1a => generated 0 bytes in 171 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) May 23 12:59:52.886143 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8300562-feab-4587-b964-65ae803bbc1c None None] REQUEST_METHOD: `POST` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:59:52.886435 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8300562-feab-4587-b964-65ae803bbc1c None None] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:59:52.886714 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8300562-feab-4587-b964-65ae803bbc1c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:59:52.889854 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 799/1598] 10.222.0.30 () {58 vars in 1121 bytes} [Mon May 23 12:59:52 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) May 23 12:59:52.891076 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22415983-63ae-4fc8-bda9-e75c0ffdff41 admin admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:59:52.901236 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22415983-63ae-4fc8-bda9-e75c0ffdff41 admin admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:59:52.905703 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.auth.core [None req-a8300562-feab-4587-b964-65ae803bbc1c None None] MFA Rules not processed for user `0f7119f8fa5c4445977671140ded91a4`. Rule list: `[]` (Enabled: `True`). {{(pid=92057) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 12:59:52.906964 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.fernet_utils [None req-a8300562-feab-4587-b964-65ae803bbc1c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92057) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 12:59:52.956959 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5bc014f-41f0-48c0-827c-c685bf581aa4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:59:52.957667 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5bc014f-41f0-48c0-827c-c685bf581aa4 admin admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:59:52.958561 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5bc014f-41f0-48c0-827c-c685bf581aa4 admin admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:59:52.959254 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5bc014f-41f0-48c0-827c-c685bf581aa4 admin admin] PATH_INFO: `/v3/roles` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:59:52.959979 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5bc014f-41f0-48c0-827c-c685bf581aa4 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:59:52.961584 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5bc014f-41f0-48c0-827c-c685bf581aa4 admin admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:59:52.968399 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 800/1599] 10.222.0.30 () {58 vars in 957 bytes} [Mon May 23 12:59:52 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 83 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) May 23 12:59:52.969259 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8300562-feab-4587-b964-65ae803bbc1c None None] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:59:52.975131 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 800/1600] 10.222.0.30 () {58 vars in 1121 bytes} [Mon May 23 12:59:52 2022] GET /identity/v3/roles => generated 1420 bytes in 85 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) May 23 12:59:52.975979 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5bc014f-41f0-48c0-827c-c685bf581aa4 admin admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:59:52.982308 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8300562-feab-4587-b964-65ae803bbc1c None None] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:59:52.986931 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04edbadc-708e-418d-8de2-67abee2671c6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:59:52.987589 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04edbadc-708e-418d-8de2-67abee2671c6 admin admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:59:52.987869 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04edbadc-708e-418d-8de2-67abee2671c6 admin admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:59:52.988144 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04edbadc-708e-418d-8de2-67abee2671c6 admin admin] PATH_INFO: `/v3/roles` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:59:52.988894 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04edbadc-708e-418d-8de2-67abee2671c6 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:59:52.990560 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04edbadc-708e-418d-8de2-67abee2671c6 admin admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:59:52.992823 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5bc014f-41f0-48c0-827c-c685bf581aa4 admin admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:59:52.997623 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbf3ef9d-8e72-47b2-815b-e80baaa15d0b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:59:52.998308 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbf3ef9d-8e72-47b2-815b-e80baaa15d0b admin admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:59:52.998592 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbf3ef9d-8e72-47b2-815b-e80baaa15d0b admin admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:59:52.998876 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbf3ef9d-8e72-47b2-815b-e80baaa15d0b admin admin] PATH_INFO: `/v3/roles` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:59:52.999708 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bbf3ef9d-8e72-47b2-815b-e80baaa15d0b admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:59:53.000937 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 801/1601] 10.222.0.30 () {58 vars in 1121 bytes} [Mon May 23 12:59:52 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) May 23 12:59:53.001299 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bbf3ef9d-8e72-47b2-815b-e80baaa15d0b admin admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:59:53.001713 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04edbadc-708e-418d-8de2-67abee2671c6 admin admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:59:53.014156 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04edbadc-708e-418d-8de2-67abee2671c6 admin admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:59:53.018841 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39e0cc54-71b5-46a1-bc51-a1196958cf62 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:59:53.019717 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39e0cc54-71b5-46a1-bc51-a1196958cf62 admin admin] REQUEST_METHOD: `PUT` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:59:53.020003 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39e0cc54-71b5-46a1-bc51-a1196958cf62 admin admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:59:53.020281 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39e0cc54-71b5-46a1-bc51-a1196958cf62 admin admin] PATH_INFO: `/v3/projects/fb9c780793e7400e9b062200257ab68e/users/19c9bc4d2df346c6a0a53caa18657f48/roles/80fb22507c9a4ff09d44d8f12f6fff1a` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:59:53.022061 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 801/1602] 10.222.0.30 () {58 vars in 1121 bytes} [Mon May 23 12:59:52 2022] GET /identity/v3/roles => generated 1420 bytes in 46 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) May 23 12:59:53.022668 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=93,keystone:INSERT=7 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:59:53.022996 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbf3ef9d-8e72-47b2-815b-e80baaa15d0b admin admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:59:53.039609 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39e0cc54-71b5-46a1-bc51-a1196958cf62 admin admin] RBAC: Authorizing `identity:create_grant(project_id=fb9c780793e7400e9b062200257ab68e, user_id=19c9bc4d2df346c6a0a53caa18657f48, role_id=80fb22507c9a4ff09d44d8f12f6fff1a)` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:59:53.042318 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39e0cc54-71b5-46a1-bc51-a1196958cf62 admin admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:59:53.042438 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbf3ef9d-8e72-47b2-815b-e80baaa15d0b admin admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:59:53.048018 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6ef4113-2eab-42ad-8c93-c68e408a82af admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:59:53.048675 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6ef4113-2eab-42ad-8c93-c68e408a82af admin admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:59:53.048960 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6ef4113-2eab-42ad-8c93-c68e408a82af admin admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:59:53.049239 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6ef4113-2eab-42ad-8c93-c68e408a82af admin admin] PATH_INFO: `/v3/roles` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:59:53.049991 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6ef4113-2eab-42ad-8c93-c68e408a82af admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:59:53.051582 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6ef4113-2eab-42ad-8c93-c68e408a82af admin admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:59:53.062403 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 802/1603] 10.222.0.30 () {58 vars in 1121 bytes} [Mon May 23 12:59:53 2022] GET /identity/v3/roles => generated 1420 bytes in 40 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) May 23 12:59:53.064440 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7cb94553-3ed2-4525-9acd-bf79ff2b2ee6 None None] REQUEST_METHOD: `POST` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:59:53.064440 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7cb94553-3ed2-4525-9acd-bf79ff2b2ee6 None None] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:59:53.064440 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7cb94553-3ed2-4525-9acd-bf79ff2b2ee6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:59:53.080457 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.auth.core [None req-7cb94553-3ed2-4525-9acd-bf79ff2b2ee6 None None] MFA Rules not processed for user `0f7119f8fa5c4445977671140ded91a4`. Rule list: `[]` (Enabled: `True`). {{(pid=92056) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 12:59:53.081683 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.fernet_utils [None req-7cb94553-3ed2-4525-9acd-bf79ff2b2ee6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92056) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 12:59:53.089109 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 803/1604] 10.222.0.30 () {58 vars in 957 bytes} [Mon May 23 12:59:53 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 26 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) May 23 12:59:53.089844 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7cb94553-3ed2-4525-9acd-bf79ff2b2ee6 None None] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:59:53.099228 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7cb94553-3ed2-4525-9acd-bf79ff2b2ee6 None None] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:59:53.103927 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4283250-d457-4668-b0bb-93fee87d9c81 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:59:53.104966 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4283250-d457-4668-b0bb-93fee87d9c81 admin admin] REQUEST_METHOD: `PUT` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:59:53.105274 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4283250-d457-4668-b0bb-93fee87d9c81 admin admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:59:53.105548 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4283250-d457-4668-b0bb-93fee87d9c81 admin admin] PATH_INFO: `/v3/projects/0ae347c79fe74082906f0d8eb9dc6677/users/257251830f5148e5b9e435c2c13828e8/roles/80fb22507c9a4ff09d44d8f12f6fff1a` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:59:53.120360 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4283250-d457-4668-b0bb-93fee87d9c81 admin admin] RBAC: Authorizing `identity:create_grant(project_id=0ae347c79fe74082906f0d8eb9dc6677, user_id=257251830f5148e5b9e435c2c13828e8, role_id=80fb22507c9a4ff09d44d8f12f6fff1a)` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:59:53.123611 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4283250-d457-4668-b0bb-93fee87d9c81 admin admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:59:53.147428 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 802/1605] 10.222.0.30 () {60 vars in 1482 bytes} [Mon May 23 12:59:53 2022] PUT /identity/v3/projects/fb9c780793e7400e9b062200257ab68e/users/19c9bc4d2df346c6a0a53caa18657f48/roles/80fb22507c9a4ff09d44d8f12f6fff1a => generated 0 bytes in 146 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) May 23 12:59:53.150514 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39e0cc54-71b5-46a1-bc51-a1196958cf62 admin admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:59:53.208769 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39e0cc54-71b5-46a1-bc51-a1196958cf62 admin admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:59:53.213587 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e9b8e08-ccef-453a-a4ba-4557d02d91e4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:59:53.214527 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e9b8e08-ccef-453a-a4ba-4557d02d91e4 admin admin] REQUEST_METHOD: `PUT` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:59:53.214808 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e9b8e08-ccef-453a-a4ba-4557d02d91e4 admin admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:59:53.215158 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e9b8e08-ccef-453a-a4ba-4557d02d91e4 admin admin] PATH_INFO: `/v3/projects/b319cf5737a54c86949cf39ba9f98fac/users/3209df37ffc64253a7f86ab40c581bb2/roles/bc3c3c9d654940cbaa5177a6be23d1bc` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:59:53.221958 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 804/1606] 10.222.0.30 () {60 vars in 1482 bytes} [Mon May 23 12:59:53 2022] PUT /identity/v3/projects/0ae347c79fe74082906f0d8eb9dc6677/users/257251830f5148e5b9e435c2c13828e8/roles/80fb22507c9a4ff09d44d8f12f6fff1a => generated 0 bytes in 132 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) May 23 12:59:53.223113 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4283250-d457-4668-b0bb-93fee87d9c81 admin admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:59:53.231634 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e9b8e08-ccef-453a-a4ba-4557d02d91e4 admin admin] RBAC: Authorizing `identity:create_grant(project_id=b319cf5737a54c86949cf39ba9f98fac, user_id=3209df37ffc64253a7f86ab40c581bb2, role_id=bc3c3c9d654940cbaa5177a6be23d1bc)` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:59:53.234360 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e9b8e08-ccef-453a-a4ba-4557d02d91e4 admin admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:59:53.279709 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4283250-d457-4668-b0bb-93fee87d9c81 admin admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:59:53.283278 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d183a6c4-a1b5-45c4-a8b4-3f68b634abee service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2eb83ca38e79449095f3a2f947bbb06a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9c389ae65aaa4542bb5a5c4fd0bacf5b', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:59:53.283910 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d183a6c4-a1b5-45c4-a8b4-3f68b634abee service neutron] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:59:53.284189 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d183a6c4-a1b5-45c4-a8b4-3f68b634abee service neutron] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:59:53.284463 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d183a6c4-a1b5-45c4-a8b4-3f68b634abee service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:59:53.321041 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 803/1607] 10.222.0.30 () {60 vars in 1482 bytes} [Mon May 23 12:59:53 2022] PUT /identity/v3/projects/b319cf5737a54c86949cf39ba9f98fac/users/3209df37ffc64253a7f86ab40c581bb2/roles/bc3c3c9d654940cbaa5177a6be23d1bc => generated 0 bytes in 173 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) May 23 12:59:53.322536 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e9b8e08-ccef-453a-a4ba-4557d02d91e4 admin admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:59:53.330110 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:59:53.359709 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d183a6c4-a1b5-45c4-a8b4-3f68b634abee service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:59:53.359709 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d183a6c4-a1b5-45c4-a8b4-3f68b634abee service neutron] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:59:53.366524 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 805/1608] 10.222.0.30 () {60 vars in 1365 bytes} [Mon May 23 12:59:53 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 144 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) May 23 12:59:53.367764 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d183a6c4-a1b5-45c4-a8b4-3f68b634abee service neutron] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:59:53.380813 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d183a6c4-a1b5-45c4-a8b4-3f68b634abee service neutron] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:59:53.395851 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e9b8e08-ccef-453a-a4ba-4557d02d91e4 admin admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:59:53.400755 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef936ee1-7fab-408e-83f3-ef416ef34cdb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:59:53.401667 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef936ee1-7fab-408e-83f3-ef416ef34cdb admin admin] REQUEST_METHOD: `PUT` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:59:53.401984 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef936ee1-7fab-408e-83f3-ef416ef34cdb admin admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:59:53.402259 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef936ee1-7fab-408e-83f3-ef416ef34cdb admin admin] PATH_INFO: `/v3/projects/5bda5ee012d84d88a3015baa3b80a7d7/users/b18e5f1b24634096995068e41d94cad6/roles/80fb22507c9a4ff09d44d8f12f6fff1a` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:59:53.417423 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef936ee1-7fab-408e-83f3-ef416ef34cdb admin admin] RBAC: Authorizing `identity:create_grant(project_id=5bda5ee012d84d88a3015baa3b80a7d7, user_id=b18e5f1b24634096995068e41d94cad6, role_id=80fb22507c9a4ff09d44d8f12f6fff1a)` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:59:53.420374 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef936ee1-7fab-408e-83f3-ef416ef34cdb admin admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:59:53.439377 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3e493ab-ee55-41a1-a596-25d4cd8b483a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:59:53.440257 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3e493ab-ee55-41a1-a596-25d4cd8b483a admin admin] REQUEST_METHOD: `PUT` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:59:53.440534 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3e493ab-ee55-41a1-a596-25d4cd8b483a admin admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:59:53.440804 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3e493ab-ee55-41a1-a596-25d4cd8b483a admin admin] PATH_INFO: `/v3/projects/6157384de11a49aa99d6e5201e5ea368/users/b75234dc3b6e489b9522c10397b5b69d/roles/80fb22507c9a4ff09d44d8f12f6fff1a` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:59:53.456062 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3e493ab-ee55-41a1-a596-25d4cd8b483a admin admin] RBAC: Authorizing `identity:create_grant(project_id=6157384de11a49aa99d6e5201e5ea368, user_id=b75234dc3b6e489b9522c10397b5b69d, role_id=80fb22507c9a4ff09d44d8f12f6fff1a)` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:59:53.458754 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3e493ab-ee55-41a1-a596-25d4cd8b483a admin admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:59:53.511326 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 804/1609] 10.222.0.30 () {60 vars in 1482 bytes} [Mon May 23 12:59:53 2022] PUT /identity/v3/projects/5bda5ee012d84d88a3015baa3b80a7d7/users/b18e5f1b24634096995068e41d94cad6/roles/80fb22507c9a4ff09d44d8f12f6fff1a => generated 0 bytes in 190 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) May 23 12:59:53.512646 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef936ee1-7fab-408e-83f3-ef416ef34cdb admin admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:59:53.535525 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 806/1610] 10.222.0.30 () {60 vars in 1482 bytes} [Mon May 23 12:59:53 2022] PUT /identity/v3/projects/6157384de11a49aa99d6e5201e5ea368/users/b75234dc3b6e489b9522c10397b5b69d/roles/80fb22507c9a4ff09d44d8f12f6fff1a => generated 0 bytes in 168 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) May 23 12:59:53.537544 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3e493ab-ee55-41a1-a596-25d4cd8b483a admin admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:59:53.577048 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:59:53.591169 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef936ee1-7fab-408e-83f3-ef416ef34cdb admin admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:59:53.604055 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3e493ab-ee55-41a1-a596-25d4cd8b483a admin admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:59:53.607989 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8aea4073-7924-4008-b9e9-9381d3505592 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2eb83ca38e79449095f3a2f947bbb06a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9c389ae65aaa4542bb5a5c4fd0bacf5b', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:59:53.608662 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8aea4073-7924-4008-b9e9-9381d3505592 service neutron] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:59:53.608967 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8aea4073-7924-4008-b9e9-9381d3505592 service neutron] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:59:53.609282 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8aea4073-7924-4008-b9e9-9381d3505592 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:59:53.651173 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71b34981-713c-4146-8323-b3fbde5ec57b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:59:53.652206 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71b34981-713c-4146-8323-b3fbde5ec57b admin admin] REQUEST_METHOD: `PUT` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:59:53.652536 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71b34981-713c-4146-8323-b3fbde5ec57b admin admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:59:53.653855 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71b34981-713c-4146-8323-b3fbde5ec57b admin admin] PATH_INFO: `/v3/projects/e6ff065adc71404184a1fe9ccdf97bee/users/caaf63946cde4042b60389be276025d0/roles/80fb22507c9a4ff09d44d8f12f6fff1a` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:59:53.676850 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8aea4073-7924-4008-b9e9-9381d3505592 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:59:53.677910 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71b34981-713c-4146-8323-b3fbde5ec57b admin admin] RBAC: Authorizing `identity:create_grant(project_id=e6ff065adc71404184a1fe9ccdf97bee, user_id=caaf63946cde4042b60389be276025d0, role_id=80fb22507c9a4ff09d44d8f12f6fff1a)` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:59:53.679395 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8aea4073-7924-4008-b9e9-9381d3505592 service neutron] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:59:53.686611 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71b34981-713c-4146-8323-b3fbde5ec57b admin admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:59:53.692126 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 807/1611] 10.222.0.30 () {60 vars in 1365 bytes} [Mon May 23 12:59:53 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 156 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) May 23 12:59:53.695764 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-466512f8-f7dc-4b96-a2f4-3d5469d06f39 None None] REQUEST_METHOD: `POST` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:59:53.695764 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-466512f8-f7dc-4b96-a2f4-3d5469d06f39 None None] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:59:53.696125 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-466512f8-f7dc-4b96-a2f4-3d5469d06f39 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:59:53.716224 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:59:53.725008 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.auth.core [None req-466512f8-f7dc-4b96-a2f4-3d5469d06f39 None None] MFA Rules not processed for user `19c9bc4d2df346c6a0a53caa18657f48`. Rule list: `[]` (Enabled: `True`). {{(pid=92056) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 12:59:53.726301 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.fernet_utils [None req-466512f8-f7dc-4b96-a2f4-3d5469d06f39 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92056) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 12:59:53.780951 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 805/1612] 10.222.0.30 () {60 vars in 1482 bytes} [Mon May 23 12:59:53 2022] PUT /identity/v3/projects/e6ff065adc71404184a1fe9ccdf97bee/users/caaf63946cde4042b60389be276025d0/roles/80fb22507c9a4ff09d44d8f12f6fff1a => generated 0 bytes in 269 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) May 23 12:59:53.784148 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfc861bd-1ca8-42a6-808e-e970af3b6509 None None] REQUEST_METHOD: `POST` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:59:53.784148 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfc861bd-1ca8-42a6-808e-e970af3b6509 None None] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:59:53.784148 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfc861bd-1ca8-42a6-808e-e970af3b6509 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:59:53.802183 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.auth.core [None req-dfc861bd-1ca8-42a6-808e-e970af3b6509 None None] MFA Rules not processed for user `0f7119f8fa5c4445977671140ded91a4`. Rule list: `[]` (Enabled: `True`). {{(pid=92057) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 12:59:53.803429 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.fernet_utils [None req-dfc861bd-1ca8-42a6-808e-e970af3b6509 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92057) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 12:59:53.810306 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 808/1613] 10.222.0.30 () {58 vars in 957 bytes} [Mon May 23 12:59:53 2022] POST /identity/v3/auth/tokens => generated 3246 bytes in 117 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) May 23 12:59:53.811855 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-466512f8-f7dc-4b96-a2f4-3d5469d06f39 None None] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:59:53.826206 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-466512f8-f7dc-4b96-a2f4-3d5469d06f39 None None] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:59:53.882023 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 806/1614] 10.222.0.30 () {58 vars in 957 bytes} [Mon May 23 12:59:53 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 100 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) May 23 12:59:53.884984 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1f0085e-67bc-4b45-a445-c8031b8e5a85 None None] REQUEST_METHOD: `POST` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:59:53.884984 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1f0085e-67bc-4b45-a445-c8031b8e5a85 None None] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:59:53.885531 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1f0085e-67bc-4b45-a445-c8031b8e5a85 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:59:53.899301 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae08b856-2857-4de6-8095-9716f2d07e6d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:59:53.899835 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae08b856-2857-4de6-8095-9716f2d07e6d admin admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:59:53.900116 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae08b856-2857-4de6-8095-9716f2d07e6d admin admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:59:53.900388 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae08b856-2857-4de6-8095-9716f2d07e6d admin admin] PATH_INFO: `/v3/roles` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:59:53.901112 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae08b856-2857-4de6-8095-9716f2d07e6d admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:59:53.903035 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae08b856-2857-4de6-8095-9716f2d07e6d admin admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:59:53.904540 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.auth.core [None req-b1f0085e-67bc-4b45-a445-c8031b8e5a85 None None] MFA Rules not processed for user `0f7119f8fa5c4445977671140ded91a4`. Rule list: `[]` (Enabled: `True`). {{(pid=92057) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 12:59:53.905748 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.fernet_utils [None req-b1f0085e-67bc-4b45-a445-c8031b8e5a85 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92057) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 12:59:53.914823 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 807/1615] 10.222.0.30 () {58 vars in 957 bytes} [Mon May 23 12:59:53 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) May 23 12:59:53.915860 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 809/1616] 10.222.0.30 () {58 vars in 1121 bytes} [Mon May 23 12:59:53 2022] GET /identity/v3/roles => generated 1420 bytes in 102 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) May 23 12:59:53.915978 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e4fd247-56a5-461a-abdc-fc9146ce7ef2 None None] REQUEST_METHOD: `POST` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:59:53.916063 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93f96643-55ac-4cac-a35b-92241524cac5 None None] REQUEST_METHOD: `POST` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:59:53.916226 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e4fd247-56a5-461a-abdc-fc9146ce7ef2 None None] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:59:53.916315 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93f96643-55ac-4cac-a35b-92241524cac5 None None] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:59:53.916395 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e4fd247-56a5-461a-abdc-fc9146ce7ef2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:59:53.916479 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93f96643-55ac-4cac-a35b-92241524cac5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:59:53.944826 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.auth.core [None req-93f96643-55ac-4cac-a35b-92241524cac5 None None] MFA Rules not processed for user `caaf63946cde4042b60389be276025d0`. Rule list: `[]` (Enabled: `True`). {{(pid=92056) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 12:59:53.951306 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.auth.core [None req-4e4fd247-56a5-461a-abdc-fc9146ce7ef2 None None] MFA Rules not processed for user `b75234dc3b6e489b9522c10397b5b69d`. Rule list: `[]` (Enabled: `True`). {{(pid=92057) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 12:59:53.951561 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.fernet_utils [None req-93f96643-55ac-4cac-a35b-92241524cac5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92056) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 12:59:53.951678 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.fernet_utils [None req-4e4fd247-56a5-461a-abdc-fc9146ce7ef2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92057) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 12:59:54.019065 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 810/1617] 10.222.0.30 () {58 vars in 957 bytes} [Mon May 23 12:59:53 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 105 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) May 23 12:59:54.020111 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93f96643-55ac-4cac-a35b-92241524cac5 None None] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:59:54.035147 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93f96643-55ac-4cac-a35b-92241524cac5 None None] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:59:54.038919 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 808/1618] 10.222.0.30 () {58 vars in 957 bytes} [Mon May 23 12:59:53 2022] POST /identity/v3/auth/tokens => generated 3234 bytes in 125 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) May 23 12:59:54.039931 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e4fd247-56a5-461a-abdc-fc9146ce7ef2 None None] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:59:54.045672 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79aa698a-1a94-4a28-a2bf-06b9b93511a1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:59:54.046483 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79aa698a-1a94-4a28-a2bf-06b9b93511a1 admin admin] REQUEST_METHOD: `POST` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:59:54.046963 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79aa698a-1a94-4a28-a2bf-06b9b93511a1 admin admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:59:54.047292 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79aa698a-1a94-4a28-a2bf-06b9b93511a1 admin admin] PATH_INFO: `/v3/projects` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:59:54.047996 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79aa698a-1a94-4a28-a2bf-06b9b93511a1 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:59:54.050275 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79aa698a-1a94-4a28-a2bf-06b9b93511a1 admin admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:59:54.056246 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:59:54.111510 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e4fd247-56a5-461a-abdc-fc9146ce7ef2 None None] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:59:54.115304 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-313fb04b-efa3-49b1-8bbd-8693b7982c23 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2eb83ca38e79449095f3a2f947bbb06a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9c389ae65aaa4542bb5a5c4fd0bacf5b', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:59:54.116209 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-313fb04b-efa3-49b1-8bbd-8693b7982c23 service neutron] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:59:54.116209 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-313fb04b-efa3-49b1-8bbd-8693b7982c23 service neutron] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:59:54.116500 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-313fb04b-efa3-49b1-8bbd-8693b7982c23 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:59:54.127264 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-313fb04b-efa3-49b1-8bbd-8693b7982c23 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:59:54.128711 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-313fb04b-efa3-49b1-8bbd-8693b7982c23 service neutron] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:59:54.138542 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 809/1619] 10.222.0.30 () {60 vars in 1365 bytes} [Mon May 23 12:59:54 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 99 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) May 23 12:59:54.139641 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-313fb04b-efa3-49b1-8bbd-8693b7982c23 service neutron] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:59:54.154274 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 811/1620] 10.222.0.30 () {60 vars in 1152 bytes} [Mon May 23 12:59:54 2022] POST /identity/v3/projects => generated 381 bytes in 134 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) May 23 12:59:54.155308 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79aa698a-1a94-4a28-a2bf-06b9b93511a1 admin admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:59:54.156601 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-313fb04b-efa3-49b1-8bbd-8693b7982c23 service neutron] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:59:54.162131 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:59:54.228482 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79aa698a-1a94-4a28-a2bf-06b9b93511a1 admin admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:59:54.233349 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0dd3f95-c988-40c9-b8ba-42872e9104d7 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2eb83ca38e79449095f3a2f947bbb06a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9c389ae65aaa4542bb5a5c4fd0bacf5b', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:59:54.234361 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0dd3f95-c988-40c9-b8ba-42872e9104d7 service neutron] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:59:54.234361 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0dd3f95-c988-40c9-b8ba-42872e9104d7 service neutron] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:59:54.250353 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0dd3f95-c988-40c9-b8ba-42872e9104d7 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:59:54.250476 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e47e43f7-0eaa-498d-a72f-db01bb35a28c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:59:54.250476 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e47e43f7-0eaa-498d-a72f-db01bb35a28c admin admin] REQUEST_METHOD: `PUT` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:59:54.250476 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e47e43f7-0eaa-498d-a72f-db01bb35a28c admin admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:59:54.250476 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e47e43f7-0eaa-498d-a72f-db01bb35a28c admin admin] PATH_INFO: `/v3/projects/b319cf5737a54c86949cf39ba9f98fac/users/3209df37ffc64253a7f86ab40c581bb2/roles/bc3c3c9d654940cbaa5177a6be23d1bc` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:59:54.251164 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e47e43f7-0eaa-498d-a72f-db01bb35a28c admin admin] RBAC: Authorizing `identity:create_grant(project_id=b319cf5737a54c86949cf39ba9f98fac, user_id=3209df37ffc64253a7f86ab40c581bb2, role_id=bc3c3c9d654940cbaa5177a6be23d1bc)` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:59:54.254053 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e47e43f7-0eaa-498d-a72f-db01bb35a28c admin admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:59:54.272499 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 810/1621] 10.222.0.30 () {60 vars in 1482 bytes} [Mon May 23 12:59:54 2022] PUT /identity/v3/projects/b319cf5737a54c86949cf39ba9f98fac/users/3209df37ffc64253a7f86ab40c581bb2/roles/bc3c3c9d654940cbaa5177a6be23d1bc => generated 0 bytes in 133 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) May 23 12:59:54.279234 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e47e43f7-0eaa-498d-a72f-db01bb35a28c admin admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:59:54.324475 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0dd3f95-c988-40c9-b8ba-42872e9104d7 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:59:54.326113 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0dd3f95-c988-40c9-b8ba-42872e9104d7 service neutron] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:59:54.336348 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 812/1622] 10.222.0.30 () {60 vars in 1365 bytes} [Mon May 23 12:59:54 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 182 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) May 23 12:59:54.338857 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-030f76a4-5123-41c5-8105-1edc2afdb7c0 None None] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:59:54.339349 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-030f76a4-5123-41c5-8105-1edc2afdb7c0 None None] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:59:54.339623 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-030f76a4-5123-41c5-8105-1edc2afdb7c0 None None] PATH_INFO: `/` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:59:54.343437 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 813/1623] 10.222.0.30 () {54 vars in 961 bytes} [Mon May 23 12:59:54 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) May 23 12:59:54.344348 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-030f76a4-5123-41c5-8105-1edc2afdb7c0 None None] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:59:54.373131 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e47e43f7-0eaa-498d-a72f-db01bb35a28c admin admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:59:54.378412 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e3510b0-1ca8-4117-806b-05e6e2ddafbc service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1666097f8ed545c6b0e4809c93e4775a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9c389ae65aaa4542bb5a5c4fd0bacf5b', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:59:54.378880 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e3510b0-1ca8-4117-806b-05e6e2ddafbc service cinder] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:59:54.379165 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e3510b0-1ca8-4117-806b-05e6e2ddafbc service cinder] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:59:54.379443 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e3510b0-1ca8-4117-806b-05e6e2ddafbc service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:59:54.454635 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-030f76a4-5123-41c5-8105-1edc2afdb7c0 None None] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:59:54.461709 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d6edb5d-b913-4ccf-902c-2e2f6bfe85c9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:59:54.462324 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e3510b0-1ca8-4117-806b-05e6e2ddafbc service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:59:54.463445 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d6edb5d-b913-4ccf-902c-2e2f6bfe85c9 admin admin] REQUEST_METHOD: `POST` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:59:54.464073 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d6edb5d-b913-4ccf-902c-2e2f6bfe85c9 admin admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:59:54.464266 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e3510b0-1ca8-4117-806b-05e6e2ddafbc service cinder] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:59:54.464736 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d6edb5d-b913-4ccf-902c-2e2f6bfe85c9 admin admin] PATH_INFO: `/v3/users` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:59:54.465725 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d6edb5d-b913-4ccf-902c-2e2f6bfe85c9 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:59:54.468245 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d6edb5d-b913-4ccf-902c-2e2f6bfe85c9 admin admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:59:54.482074 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 811/1624] 10.222.0.30 () {60 vars in 1365 bytes} [Mon May 23 12:59:54 2022] GET /identity/v3/auth/tokens => generated 3234 bytes in 209 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) May 23 12:59:54.484016 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ff6d7f9-800c-4f7b-8f1c-55e21f4a8a90 None None] REQUEST_METHOD: `POST` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:59:54.484464 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ff6d7f9-800c-4f7b-8f1c-55e21f4a8a90 None None] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:59:54.484834 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ff6d7f9-800c-4f7b-8f1c-55e21f4a8a90 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:59:54.528218 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.auth.core [None req-6ff6d7f9-800c-4f7b-8f1c-55e21f4a8a90 None None] MFA Rules not processed for user `3209df37ffc64253a7f86ab40c581bb2`. Rule list: `[]` (Enabled: `True`). {{(pid=92057) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 12:59:54.528218 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.fernet_utils [None req-6ff6d7f9-800c-4f7b-8f1c-55e21f4a8a90 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92057) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 12:59:54.587100 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 814/1625] 10.222.0.30 () {60 vars in 1143 bytes} [Mon May 23 12:59:54 2022] POST /identity/v3/users => generated 297 bytes in 243 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) May 23 12:59:54.588511 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6c4d89a-4246-413f-9613-40be35f78186 None None] REQUEST_METHOD: `POST` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:59:54.588805 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6c4d89a-4246-413f-9613-40be35f78186 None None] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:59:54.589083 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6c4d89a-4246-413f-9613-40be35f78186 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:59:54.640168 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.auth.core [None req-d6c4d89a-4246-413f-9613-40be35f78186 None None] MFA Rules not processed for user `cbe9931d94d44cc7ace0e83a7e5015fc`. Rule list: `[]` (Enabled: `True`). {{(pid=92056) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 12:59:54.641584 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.fernet_utils [None req-d6c4d89a-4246-413f-9613-40be35f78186 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92056) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 12:59:54.671396 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 812/1626] 10.222.0.30 () {58 vars in 957 bytes} [Mon May 23 12:59:54 2022] POST /identity/v3/auth/tokens => generated 3307 bytes in 188 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) May 23 12:59:54.672557 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ff6d7f9-800c-4f7b-8f1c-55e21f4a8a90 None None] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:59:54.686486 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ff6d7f9-800c-4f7b-8f1c-55e21f4a8a90 None None] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:59:54.705227 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c60cb6e4-ba2d-457d-a437-cdae90bc4495 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:59:54.706380 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c60cb6e4-ba2d-457d-a437-cdae90bc4495 admin admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:59:54.707617 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c60cb6e4-ba2d-457d-a437-cdae90bc4495 admin admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:59:54.709252 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:59:54.710269 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c60cb6e4-ba2d-457d-a437-cdae90bc4495 admin admin] PATH_INFO: `/v3/roles` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:59:54.711649 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c60cb6e4-ba2d-457d-a437-cdae90bc4495 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:59:54.713480 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c60cb6e4-ba2d-457d-a437-cdae90bc4495 admin admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:59:54.736731 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 813/1627] 10.222.0.30 () {58 vars in 1121 bytes} [Mon May 23 12:59:54 2022] GET /identity/v3/roles => generated 1420 bytes in 65 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) May 23 12:59:54.738959 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c60cb6e4-ba2d-457d-a437-cdae90bc4495 admin admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:59:54.760168 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c60cb6e4-ba2d-457d-a437-cdae90bc4495 admin admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:59:54.770378 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b80fc9cf-a986-4491-bb38-23e7842a2e6e service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1666097f8ed545c6b0e4809c93e4775a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9c389ae65aaa4542bb5a5c4fd0bacf5b', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:59:54.771441 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b80fc9cf-a986-4491-bb38-23e7842a2e6e service cinder] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:59:54.772786 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b80fc9cf-a986-4491-bb38-23e7842a2e6e service cinder] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:59:54.773282 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b80fc9cf-a986-4491-bb38-23e7842a2e6e service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:59:54.776608 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 815/1628] 10.222.0.30 () {58 vars in 1058 bytes} [Mon May 23 12:59:54 2022] POST /identity/v3/auth/tokens => generated 3362 bytes in 189 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) May 23 12:59:54.778236 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6c4d89a-4246-413f-9613-40be35f78186 None None] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:59:54.791143 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b80fc9cf-a986-4491-bb38-23e7842a2e6e service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:59:54.791831 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6c4d89a-4246-413f-9613-40be35f78186 None None] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:59:54.793549 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b80fc9cf-a986-4491-bb38-23e7842a2e6e service cinder] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:59:54.797582 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8172b353-2031-4418-afb6-82e877d91674 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:59:54.798529 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8172b353-2031-4418-afb6-82e877d91674 admin admin] REQUEST_METHOD: `PUT` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:59:54.798831 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8172b353-2031-4418-afb6-82e877d91674 admin admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:59:54.799152 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8172b353-2031-4418-afb6-82e877d91674 admin admin] PATH_INFO: `/v3/projects/52d9418e818d4facaf28bd0ae70276fe/users/89a816fe27194bf38144a959521f3338/roles/bc3c3c9d654940cbaa5177a6be23d1bc` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:59:54.818113 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 814/1629] 10.222.0.30 () {60 vars in 1365 bytes} [Mon May 23 12:59:54 2022] GET /identity/v3/auth/tokens => generated 3307 bytes in 80 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) May 23 12:59:54.819363 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b80fc9cf-a986-4491-bb38-23e7842a2e6e service cinder] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:59:54.821134 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8172b353-2031-4418-afb6-82e877d91674 admin admin] RBAC: Authorizing `identity:create_grant(project_id=52d9418e818d4facaf28bd0ae70276fe, user_id=89a816fe27194bf38144a959521f3338, role_id=bc3c3c9d654940cbaa5177a6be23d1bc)` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:59:54.825580 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8172b353-2031-4418-afb6-82e877d91674 admin admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:59:54.850323 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b80fc9cf-a986-4491-bb38-23e7842a2e6e service cinder] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:59:54.854269 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0534b072-deaf-4462-9ae2-273a4ea9ca63 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cbe9931d94d44cc7ace0e83a7e5015fc', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9c389ae65aaa4542bb5a5c4fd0bacf5b', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'member', 'reader', 'service', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:59:54.855070 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0534b072-deaf-4462-9ae2-273a4ea9ca63 service nova] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:59:54.855388 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0534b072-deaf-4462-9ae2-273a4ea9ca63 service nova] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:59:54.855696 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0534b072-deaf-4462-9ae2-273a4ea9ca63 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:59:54.919164 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0534b072-deaf-4462-9ae2-273a4ea9ca63 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:59:54.921220 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0534b072-deaf-4462-9ae2-273a4ea9ca63 service nova] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:59:54.946012 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 815/1630] 10.222.0.30 () {60 vars in 1365 bytes} [Mon May 23 12:59:54 2022] GET /identity/v3/auth/tokens => generated 3250 bytes in 127 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) May 23 12:59:54.955905 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 816/1631] 10.222.0.30 () {60 vars in 1482 bytes} [Mon May 23 12:59:54 2022] PUT /identity/v3/projects/52d9418e818d4facaf28bd0ae70276fe/users/89a816fe27194bf38144a959521f3338/roles/bc3c3c9d654940cbaa5177a6be23d1bc => generated 0 bytes in 179 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) May 23 12:59:54.959850 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0534b072-deaf-4462-9ae2-273a4ea9ca63 service nova] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:59:54.987128 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0534b072-deaf-4462-9ae2-273a4ea9ca63 service nova] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:59:55.126913 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ddb7031-ecb8-420a-80b9-b932299522a3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:59:55.128404 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ddb7031-ecb8-420a-80b9-b932299522a3 admin admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:59:55.134384 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ddb7031-ecb8-420a-80b9-b932299522a3 admin admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:59:55.135026 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ddb7031-ecb8-420a-80b9-b932299522a3 admin admin] PATH_INFO: `/v3/roles` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:59:55.144656 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ddb7031-ecb8-420a-80b9-b932299522a3 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:59:55.146611 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ddb7031-ecb8-420a-80b9-b932299522a3 admin admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:59:55.180623 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 816/1632] 10.222.0.30 () {58 vars in 1121 bytes} [Mon May 23 12:59:54 2022] GET /identity/v3/roles => generated 1420 bytes in 218 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) May 23 12:59:55.181976 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8172b353-2031-4418-afb6-82e877d91674 admin admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:59:55.203161 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8172b353-2031-4418-afb6-82e877d91674 admin admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:59:55.206159 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:59:55.228137 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c8798e0-2f71-4e2c-b28a-af86079e1fd4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:59:55.230459 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c8798e0-2f71-4e2c-b28a-af86079e1fd4 admin admin] REQUEST_METHOD: `PUT` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:59:55.232366 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c8798e0-2f71-4e2c-b28a-af86079e1fd4 admin admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:59:55.232490 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ddb7031-ecb8-420a-80b9-b932299522a3 admin admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:59:55.232590 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c8798e0-2f71-4e2c-b28a-af86079e1fd4 admin admin] PATH_INFO: `/v3/projects/52d9418e818d4facaf28bd0ae70276fe/users/89a816fe27194bf38144a959521f3338/roles/bc3c3c9d654940cbaa5177a6be23d1bc` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:59:55.236508 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c8798e0-2f71-4e2c-b28a-af86079e1fd4 admin admin] RBAC: Authorizing `identity:create_grant(project_id=52d9418e818d4facaf28bd0ae70276fe, user_id=89a816fe27194bf38144a959521f3338, role_id=bc3c3c9d654940cbaa5177a6be23d1bc)` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:59:55.240356 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c8798e0-2f71-4e2c-b28a-af86079e1fd4 admin admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:59:55.257332 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 817/1633] 10.222.0.30 () {60 vars in 1482 bytes} [Mon May 23 12:59:55 2022] PUT /identity/v3/projects/52d9418e818d4facaf28bd0ae70276fe/users/89a816fe27194bf38144a959521f3338/roles/bc3c3c9d654940cbaa5177a6be23d1bc => generated 0 bytes in 76 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) May 23 12:59:55.270802 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd9c8e8e-f5f8-4ca9-8fa1-ba64c93de87b None None] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:59:55.271215 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd9c8e8e-f5f8-4ca9-8fa1-ba64c93de87b None None] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:59:55.272031 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd9c8e8e-f5f8-4ca9-8fa1-ba64c93de87b None None] PATH_INFO: `/` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:59:55.273568 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 818/1634] 10.222.0.30 () {54 vars in 912 bytes} [Mon May 23 12:59:55 2022] GET /identity => generated 270 bytes in 12 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) May 23 12:59:55.274926 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6be0214-03f4-431c-a4b2-41037fe09a4e None None] REQUEST_METHOD: `POST` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:59:55.275309 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6be0214-03f4-431c-a4b2-41037fe09a4e None None] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:59:55.275711 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6be0214-03f4-431c-a4b2-41037fe09a4e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:59:55.310415 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.auth.core [None req-f6be0214-03f4-431c-a4b2-41037fe09a4e None None] MFA Rules not processed for user `89a816fe27194bf38144a959521f3338`. Rule list: `[]` (Enabled: `True`). {{(pid=92056) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 12:59:55.311715 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.fernet_utils [None req-f6be0214-03f4-431c-a4b2-41037fe09a4e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92056) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 12:59:55.371972 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ddb7031-ecb8-420a-80b9-b932299522a3 admin admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:59:55.429213 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 819/1635] 10.222.0.30 () {58 vars in 957 bytes} [Mon May 23 12:59:55 2022] POST /identity/v3/auth/tokens => generated 3315 bytes in 155 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) May 23 12:59:55.431271 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f1bd186-842f-486c-b644-2d81dfc6b71e None None] REQUEST_METHOD: `POST` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:59:55.431738 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f1bd186-842f-486c-b644-2d81dfc6b71e None None] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:59:55.432373 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f1bd186-842f-486c-b644-2d81dfc6b71e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:59:55.462711 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.auth.core [None req-1f1bd186-842f-486c-b644-2d81dfc6b71e None None] MFA Rules not processed for user `cbe9931d94d44cc7ace0e83a7e5015fc`. Rule list: `[]` (Enabled: `True`). {{(pid=92056) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 12:59:55.464300 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.fernet_utils [None req-1f1bd186-842f-486c-b644-2d81dfc6b71e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92056) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 12:59:55.490871 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cfcfb469-e74d-4bb4-9c06-1986819635ce service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1666097f8ed545c6b0e4809c93e4775a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9c389ae65aaa4542bb5a5c4fd0bacf5b', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:59:55.491532 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfcfb469-e74d-4bb4-9c06-1986819635ce service cinder] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:59:55.491815 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfcfb469-e74d-4bb4-9c06-1986819635ce service cinder] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:59:55.492091 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfcfb469-e74d-4bb4-9c06-1986819635ce service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:59:55.503029 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=100 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:59:55.581389 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 820/1636] 10.222.0.30 () {58 vars in 1009 bytes} [Mon May 23 12:59:55 2022] POST /identity/v3/auth/tokens => generated 3362 bytes in 151 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) May 23 12:59:55.583824 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3db7549a-9727-4cd6-8ea3-77cbf5b6286b None None] REQUEST_METHOD: `POST` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:59:55.584192 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3db7549a-9727-4cd6-8ea3-77cbf5b6286b None None] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:59:55.584700 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3db7549a-9727-4cd6-8ea3-77cbf5b6286b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:59:55.613356 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cfcfb469-e74d-4bb4-9c06-1986819635ce service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:59:55.615123 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cfcfb469-e74d-4bb4-9c06-1986819635ce service cinder] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:59:55.620817 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.auth.core [None req-3db7549a-9727-4cd6-8ea3-77cbf5b6286b None None] MFA Rules not processed for user `8ea87f0f4b434124a60994ad3eaf277c`. Rule list: `[]` (Enabled: `True`). {{(pid=92056) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 12:59:55.622249 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.fernet_utils [None req-3db7549a-9727-4cd6-8ea3-77cbf5b6286b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92056) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 12:59:55.627383 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 817/1637] 10.222.0.30 () {60 vars in 1365 bytes} [Mon May 23 12:59:55 2022] GET /identity/v3/auth/tokens => generated 3238 bytes in 397 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) May 23 12:59:55.714578 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 821/1638] 10.222.0.30 () {58 vars in 957 bytes} [Mon May 23 12:59:55 2022] POST /identity/v3/auth/tokens => generated 3270 bytes in 132 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) May 23 12:59:55.719638 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cfcfb469-e74d-4bb4-9c06-1986819635ce service cinder] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:59:55.806668 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cfcfb469-e74d-4bb4-9c06-1986819635ce service cinder] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:59:55.813505 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d723272-32f0-4d8d-83c6-3b4363974db1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:59:55.814853 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d723272-32f0-4d8d-83c6-3b4363974db1 admin admin] REQUEST_METHOD: `POST` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:59:55.815147 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d723272-32f0-4d8d-83c6-3b4363974db1 admin admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:59:55.815437 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d723272-32f0-4d8d-83c6-3b4363974db1 admin admin] PATH_INFO: `/v3/projects` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:59:55.816188 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d723272-32f0-4d8d-83c6-3b4363974db1 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:59:55.818849 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d723272-32f0-4d8d-83c6-3b4363974db1 admin admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:59:55.830610 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3db7549a-9727-4cd6-8ea3-77cbf5b6286b None None] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:59:55.841183 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3db7549a-9727-4cd6-8ea3-77cbf5b6286b None None] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:59:55.846093 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0cd17e2-fdcd-4280-a670-3b6c2efcf758 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1666097f8ed545c6b0e4809c93e4775a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9c389ae65aaa4542bb5a5c4fd0bacf5b', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:59:55.850113 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0cd17e2-fdcd-4280-a670-3b6c2efcf758 service cinder] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:59:55.850113 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0cd17e2-fdcd-4280-a670-3b6c2efcf758 service cinder] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:59:55.850113 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0cd17e2-fdcd-4280-a670-3b6c2efcf758 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:59:55.870763 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0cd17e2-fdcd-4280-a670-3b6c2efcf758 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:59:55.870978 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=100 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:59:55.873626 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0cd17e2-fdcd-4280-a670-3b6c2efcf758 service cinder] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:59:55.894923 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 822/1639] 10.222.0.30 () {60 vars in 1365 bytes} [Mon May 23 12:59:55 2022] GET /identity/v3/auth/tokens => generated 3315 bytes in 65 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) May 23 12:59:55.979207 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f485301a-4a74-4391-8223-54380e229666 None None] REQUEST_METHOD: `POST` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:59:55.979646 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f485301a-4a74-4391-8223-54380e229666 None None] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:59:55.979965 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f485301a-4a74-4391-8223-54380e229666 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:59:55.983584 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 818/1640] 10.222.0.30 () {60 vars in 1152 bytes} [Mon May 23 12:59:55 2022] POST /identity/v3/projects => generated 403 bytes in 264 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) May 23 12:59:55.988052 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d723272-32f0-4d8d-83c6-3b4363974db1 admin admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:59:56.025278 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.auth.core [None req-f485301a-4a74-4391-8223-54380e229666 None None] MFA Rules not processed for user `a9b80726b3f14828b8815e30fc7e13c5`. Rule list: `[]` (Enabled: `True`). {{(pid=92056) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 12:59:56.029319 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.fernet_utils [None req-f485301a-4a74-4391-8223-54380e229666 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92056) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 12:59:56.056405 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:59:56.146512 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d723272-32f0-4d8d-83c6-3b4363974db1 admin admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:59:56.157802 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24da2e37-da81-4ae3-a4a7-fefad154307c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:59:56.158971 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24da2e37-da81-4ae3-a4a7-fefad154307c admin admin] REQUEST_METHOD: `POST` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:59:56.161327 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24da2e37-da81-4ae3-a4a7-fefad154307c admin admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:59:56.161524 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24da2e37-da81-4ae3-a4a7-fefad154307c admin admin] PATH_INFO: `/v3/users` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:59:56.163017 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24da2e37-da81-4ae3-a4a7-fefad154307c admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:59:56.166147 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24da2e37-da81-4ae3-a4a7-fefad154307c admin admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:59:56.187209 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 823/1641] 10.222.0.30 () {58 vars in 957 bytes} [Mon May 23 12:59:55 2022] POST /identity/v3/auth/tokens => generated 3246 bytes in 209 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) May 23 12:59:56.188327 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f485301a-4a74-4391-8223-54380e229666 None None] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:59:56.286032 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f485301a-4a74-4391-8223-54380e229666 None None] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:59:56.291159 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87f813b0-1455-4d4c-823a-01662b009699 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1666097f8ed545c6b0e4809c93e4775a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9c389ae65aaa4542bb5a5c4fd0bacf5b', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:59:56.292117 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87f813b0-1455-4d4c-823a-01662b009699 service cinder] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:59:56.292399 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87f813b0-1455-4d4c-823a-01662b009699 service cinder] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:59:56.292673 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87f813b0-1455-4d4c-823a-01662b009699 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:59:56.310400 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 819/1642] 10.222.0.30 () {60 vars in 1143 bytes} [Mon May 23 12:59:55 2022] POST /identity/v3/users => generated 308 bytes in 323 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) May 23 12:59:56.312005 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29710b2a-4936-4f53-ad85-568baa8c4434 None None] REQUEST_METHOD: `POST` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:59:56.315989 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29710b2a-4936-4f53-ad85-568baa8c4434 None None] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:59:56.315989 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29710b2a-4936-4f53-ad85-568baa8c4434 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:59:56.352259 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.auth.core [None req-29710b2a-4936-4f53-ad85-568baa8c4434 None None] MFA Rules not processed for user `a9b80726b3f14828b8815e30fc7e13c5`. Rule list: `[]` (Enabled: `True`). {{(pid=92057) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 12:59:56.354627 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.fernet_utils [None req-29710b2a-4936-4f53-ad85-568baa8c4434 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92057) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 12:59:56.373174 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 820/1643] 10.222.0.30 () {58 vars in 957 bytes} [Mon May 23 12:59:56 2022] POST /identity/v3/auth/tokens => generated 3246 bytes in 62 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) May 23 12:59:56.376882 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29710b2a-4936-4f53-ad85-568baa8c4434 None None] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:59:56.405240 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29710b2a-4936-4f53-ad85-568baa8c4434 None None] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:59:56.420047 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-200043f0-829e-42ec-824a-0bda798a6f39 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:59:56.420881 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-200043f0-829e-42ec-824a-0bda798a6f39 admin admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:59:56.421453 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-200043f0-829e-42ec-824a-0bda798a6f39 admin admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:59:56.421453 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-200043f0-829e-42ec-824a-0bda798a6f39 admin admin] PATH_INFO: `/v3/roles` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:59:56.422176 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-200043f0-829e-42ec-824a-0bda798a6f39 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:59:56.423781 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-200043f0-829e-42ec-824a-0bda798a6f39 admin admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:59:56.453350 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87f813b0-1455-4d4c-823a-01662b009699 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:59:56.456201 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87f813b0-1455-4d4c-823a-01662b009699 service cinder] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:59:56.468854 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 821/1644] 10.222.0.30 () {58 vars in 1121 bytes} [Mon May 23 12:59:56 2022] GET /identity/v3/roles => generated 1420 bytes in 91 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) May 23 12:59:56.470908 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0de5fa47-99e9-4928-aafd-f18ba57cf065 None None] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:59:56.470908 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0de5fa47-99e9-4928-aafd-f18ba57cf065 None None] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:59:56.470908 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0de5fa47-99e9-4928-aafd-f18ba57cf065 None None] PATH_INFO: `/` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:59:56.472137 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 824/1645] 10.222.0.30 () {60 vars in 1365 bytes} [Mon May 23 12:59:56 2022] GET /identity/v3/auth/tokens => generated 3362 bytes in 284 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) May 23 12:59:56.472571 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 822/1646] 10.222.0.30 () {54 vars in 961 bytes} [Mon May 23 12:59:56 2022] GET /identity => generated 270 bytes in 7 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) May 23 12:59:56.473139 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87f813b0-1455-4d4c-823a-01662b009699 service cinder] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:59:56.478250 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-451cd4ca-9e35-4e5e-b23d-28e535f0efd5 None None] REQUEST_METHOD: `POST` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:59:56.478871 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-451cd4ca-9e35-4e5e-b23d-28e535f0efd5 None None] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:59:56.479389 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-451cd4ca-9e35-4e5e-b23d-28e535f0efd5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:59:56.485582 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87f813b0-1455-4d4c-823a-01662b009699 service cinder] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:59:56.492914 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aaa950fd-9667-4d43-bc58-944cd99dbc3d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:59:56.494512 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aaa950fd-9667-4d43-bc58-944cd99dbc3d admin admin] REQUEST_METHOD: `PUT` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:59:56.494997 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aaa950fd-9667-4d43-bc58-944cd99dbc3d admin admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:59:56.510578 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aaa950fd-9667-4d43-bc58-944cd99dbc3d admin admin] PATH_INFO: `/v3/projects/993a79a363064d3fab6dfc5e93442b47/users/7d89f63934f048d999a46a322628dfa4/roles/bc3c3c9d654940cbaa5177a6be23d1bc` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:59:56.510864 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.auth.core [None req-451cd4ca-9e35-4e5e-b23d-28e535f0efd5 None None] MFA Rules not processed for user `cbe9931d94d44cc7ace0e83a7e5015fc`. Rule list: `[]` (Enabled: `True`). {{(pid=92057) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 12:59:56.510864 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.fernet_utils [None req-451cd4ca-9e35-4e5e-b23d-28e535f0efd5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92057) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 12:59:56.520647 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aaa950fd-9667-4d43-bc58-944cd99dbc3d admin admin] RBAC: Authorizing `identity:create_grant(project_id=993a79a363064d3fab6dfc5e93442b47, user_id=7d89f63934f048d999a46a322628dfa4, role_id=bc3c3c9d654940cbaa5177a6be23d1bc)` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:59:56.521056 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 823/1647] 10.222.0.30 () {58 vars in 1058 bytes} [Mon May 23 12:59:56 2022] POST /identity/v3/auth/tokens => generated 3362 bytes in 44 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) May 23 12:59:56.528623 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aaa950fd-9667-4d43-bc58-944cd99dbc3d admin admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:59:56.530373 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-451cd4ca-9e35-4e5e-b23d-28e535f0efd5 None None] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:59:56.543611 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-451cd4ca-9e35-4e5e-b23d-28e535f0efd5 None None] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:59:56.550091 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05034062-823b-4e0b-9366-776f3446beeb service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cbe9931d94d44cc7ace0e83a7e5015fc', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9c389ae65aaa4542bb5a5c4fd0bacf5b', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'member', 'reader', 'service', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:59:56.550879 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05034062-823b-4e0b-9366-776f3446beeb service nova] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:59:56.551270 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05034062-823b-4e0b-9366-776f3446beeb service nova] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:59:56.551646 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05034062-823b-4e0b-9366-776f3446beeb service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:59:56.561465 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05034062-823b-4e0b-9366-776f3446beeb service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:59:56.563457 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05034062-823b-4e0b-9366-776f3446beeb service nova] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:59:56.595335 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 824/1648] 10.222.0.30 () {60 vars in 1365 bytes} [Mon May 23 12:59:56 2022] GET /identity/v3/auth/tokens => generated 3246 bytes in 66 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) May 23 12:59:56.629457 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 825/1649] 10.222.0.30 () {60 vars in 1482 bytes} [Mon May 23 12:59:56 2022] PUT /identity/v3/projects/993a79a363064d3fab6dfc5e93442b47/users/7d89f63934f048d999a46a322628dfa4/roles/bc3c3c9d654940cbaa5177a6be23d1bc => generated 0 bytes in 157 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) May 23 12:59:56.634376 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05034062-823b-4e0b-9366-776f3446beeb service nova] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:59:56.651030 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05034062-823b-4e0b-9366-776f3446beeb service nova] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:59:56.732015 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aaa950fd-9667-4d43-bc58-944cd99dbc3d admin admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:59:56.796146 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dcb0c817-6692-49a0-9a09-6d6bb8464f38 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:59:56.797310 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dcb0c817-6692-49a0-9a09-6d6bb8464f38 admin admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:59:56.797310 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dcb0c817-6692-49a0-9a09-6d6bb8464f38 admin admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:59:56.797519 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dcb0c817-6692-49a0-9a09-6d6bb8464f38 admin admin] PATH_INFO: `/v3/roles` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:59:56.798128 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dcb0c817-6692-49a0-9a09-6d6bb8464f38 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:59:56.799756 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dcb0c817-6692-49a0-9a09-6d6bb8464f38 admin admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:59:56.830999 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 825/1650] 10.222.0.30 () {58 vars in 1121 bytes} [Mon May 23 12:59:56 2022] GET /identity/v3/roles => generated 1420 bytes in 198 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) May 23 12:59:56.832582 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aaa950fd-9667-4d43-bc58-944cd99dbc3d admin admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:59:56.835059 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dcb0c817-6692-49a0-9a09-6d6bb8464f38 admin admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:59:56.848692 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dcb0c817-6692-49a0-9a09-6d6bb8464f38 admin admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:59:56.849017 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb95add2-137c-4c54-9cb7-06b4b75d6e36 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1666097f8ed545c6b0e4809c93e4775a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9c389ae65aaa4542bb5a5c4fd0bacf5b', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:59:56.849998 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb95add2-137c-4c54-9cb7-06b4b75d6e36 service cinder] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:59:56.850487 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb95add2-137c-4c54-9cb7-06b4b75d6e36 service cinder] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:59:56.850911 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb95add2-137c-4c54-9cb7-06b4b75d6e36 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:59:56.851881 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc86d0da-aa1f-4b32-8335-e9120ffb2e2a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:59:56.852817 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc86d0da-aa1f-4b32-8335-e9120ffb2e2a admin admin] REQUEST_METHOD: `PUT` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:59:56.853098 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc86d0da-aa1f-4b32-8335-e9120ffb2e2a admin admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:59:56.853380 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc86d0da-aa1f-4b32-8335-e9120ffb2e2a admin admin] PATH_INFO: `/v3/projects/993a79a363064d3fab6dfc5e93442b47/users/7d89f63934f048d999a46a322628dfa4/roles/bc3c3c9d654940cbaa5177a6be23d1bc` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:59:56.855774 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc86d0da-aa1f-4b32-8335-e9120ffb2e2a admin admin] RBAC: Authorizing `identity:create_grant(project_id=993a79a363064d3fab6dfc5e93442b47, user_id=7d89f63934f048d999a46a322628dfa4, role_id=bc3c3c9d654940cbaa5177a6be23d1bc)` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:59:56.859497 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc86d0da-aa1f-4b32-8335-e9120ffb2e2a admin admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:59:56.889708 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 826/1651] 10.222.0.30 () {60 vars in 1482 bytes} [Mon May 23 12:59:56 2022] PUT /identity/v3/projects/993a79a363064d3fab6dfc5e93442b47/users/7d89f63934f048d999a46a322628dfa4/roles/bc3c3c9d654940cbaa5177a6be23d1bc => generated 0 bytes in 55 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) May 23 12:59:56.999874 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb95add2-137c-4c54-9cb7-06b4b75d6e36 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:59:57.002627 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb95add2-137c-4c54-9cb7-06b4b75d6e36 service cinder] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:59:57.006361 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81b0560e-e9cc-4c57-9ada-50a859ae9352 None None] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:59:57.009498 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81b0560e-e9cc-4c57-9ada-50a859ae9352 None None] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:59:57.010488 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81b0560e-e9cc-4c57-9ada-50a859ae9352 None None] PATH_INFO: `/` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:59:57.015416 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 827/1652] 10.222.0.30 () {54 vars in 912 bytes} [Mon May 23 12:59:57 2022] GET /identity => generated 270 bytes in 10 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) May 23 12:59:57.025184 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 826/1653] 10.222.0.30 () {60 vars in 1365 bytes} [Mon May 23 12:59:56 2022] GET /identity/v3/auth/tokens => generated 3246 bytes in 291 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) May 23 12:59:57.237740 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23e0ebfb-f617-4009-b979-e56262f627fe None None] REQUEST_METHOD: `POST` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:59:57.237740 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23e0ebfb-f617-4009-b979-e56262f627fe None None] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:59:57.237740 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23e0ebfb-f617-4009-b979-e56262f627fe None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:59:57.237740 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.auth.core [None req-23e0ebfb-f617-4009-b979-e56262f627fe None None] MFA Rules not processed for user `cbe9931d94d44cc7ace0e83a7e5015fc`. Rule list: `[]` (Enabled: `True`). {{(pid=92057) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 12:59:57.237740 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.fernet_utils [None req-23e0ebfb-f617-4009-b979-e56262f627fe None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92057) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 12:59:57.237740 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:59:57.240746 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 828/1654] 10.222.0.30 () {58 vars in 1009 bytes} [Mon May 23 12:59:57 2022] POST /identity/v3/auth/tokens => generated 3362 bytes in 144 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) May 23 12:59:57.241375 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb95add2-137c-4c54-9cb7-06b4b75d6e36 service cinder] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:59:57.241375 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.fernet_utils [None req-fb95add2-137c-4c54-9cb7-06b4b75d6e36 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92056) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 12:59:57.320715 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb95add2-137c-4c54-9cb7-06b4b75d6e36 service cinder] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:59:57.378241 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e08aacd-c122-4175-b953-b3fa310ed3fe service glance] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '5cea0feed4254ec085854986f99acf30', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9c389ae65aaa4542bb5a5c4fd0bacf5b', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:59:57.378897 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e08aacd-c122-4175-b953-b3fa310ed3fe service glance] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:59:57.379187 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e08aacd-c122-4175-b953-b3fa310ed3fe service glance] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:59:57.379489 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e08aacd-c122-4175-b953-b3fa310ed3fe service glance] PATH_INFO: `/v3/auth/tokens` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:59:57.390624 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e08aacd-c122-4175-b953-b3fa310ed3fe service glance] RBAC: Authorizing `identity:validate_token()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:59:57.392330 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e08aacd-c122-4175-b953-b3fa310ed3fe service glance] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:59:57.417027 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 827/1655] 10.222.0.30 () {60 vars in 1365 bytes} [Mon May 23 12:59:57 2022] GET /identity/v3/auth/tokens => generated 3362 bytes in 240 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) May 23 12:59:57.608802 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=100 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:59:58.385775 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62273e2c-c28c-4369-88cd-90e20a3ed88e None None] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:59:58.386188 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62273e2c-c28c-4369-88cd-90e20a3ed88e None None] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:59:58.386446 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62273e2c-c28c-4369-88cd-90e20a3ed88e None None] PATH_INFO: `/` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:59:58.388416 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 829/1656] 10.222.0.30 () {54 vars in 912 bytes} [Mon May 23 12:59:58 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) May 23 12:59:58.394340 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48de69a1-e335-4a79-bca2-cb13684b5b93 None None] REQUEST_METHOD: `POST` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:59:58.394816 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48de69a1-e335-4a79-bca2-cb13684b5b93 None None] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:59:58.395526 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48de69a1-e335-4a79-bca2-cb13684b5b93 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:59:58.477291 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.auth.core [None req-48de69a1-e335-4a79-bca2-cb13684b5b93 None None] MFA Rules not processed for user `2eb83ca38e79449095f3a2f947bbb06a`. Rule list: `[]` (Enabled: `True`). {{(pid=92056) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 12:59:58.479846 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 12:59:58.481018 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.fernet_utils [None req-48de69a1-e335-4a79-bca2-cb13684b5b93 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92056) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 12:59:58.598023 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 828/1657] 10.222.0.30 () {58 vars in 1009 bytes} [Mon May 23 12:59:58 2022] POST /identity/v3/auth/tokens => generated 3111 bytes in 205 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) May 23 12:59:58.635061 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62273e2c-c28c-4369-88cd-90e20a3ed88e None None] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 12:59:58.658680 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62273e2c-c28c-4369-88cd-90e20a3ed88e None None] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 12:59:58.665319 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7cdb3db6-d611-456f-94f5-3e8604139f72 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2eb83ca38e79449095f3a2f947bbb06a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9c389ae65aaa4542bb5a5c4fd0bacf5b', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 12:59:58.665965 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7cdb3db6-d611-456f-94f5-3e8604139f72 service neutron] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:59:58.666258 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7cdb3db6-d611-456f-94f5-3e8604139f72 service neutron] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:59:58.666528 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7cdb3db6-d611-456f-94f5-3e8604139f72 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:59:58.679002 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7cdb3db6-d611-456f-94f5-3e8604139f72 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 12:59:58.680464 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7cdb3db6-d611-456f-94f5-3e8604139f72 service neutron] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 12:59:58.702757 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 830/1658] 10.222.0.30 () {60 vars in 1365 bytes} [Mon May 23 12:59:58 2022] GET /identity/v3/auth/tokens => generated 3111 bytes in 69 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) May 23 12:59:59.877267 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a4a8b4b-a218-41df-90fc-6b1c0b8ef05d None None] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:59:59.877791 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a4a8b4b-a218-41df-90fc-6b1c0b8ef05d None None] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:59:59.882896 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a4a8b4b-a218-41df-90fc-6b1c0b8ef05d None None] PATH_INFO: `/` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:59:59.882896 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 829/1659] 10.222.0.30 () {54 vars in 960 bytes} [Mon May 23 12:59:59 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) May 23 12:59:59.886477 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7dac062-3b9d-49df-9dea-99b0630a991d None None] REQUEST_METHOD: `POST` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:59:59.887682 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7dac062-3b9d-49df-9dea-99b0630a991d None None] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:59:59.887682 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7dac062-3b9d-49df-9dea-99b0630a991d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 12:59:59.920363 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.auth.core [None req-e7dac062-3b9d-49df-9dea-99b0630a991d None None] MFA Rules not processed for user `d5fb51f8d0fa46439cb29666970b885b`. Rule list: `[]` (Enabled: `True`). {{(pid=92057) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 12:59:59.925377 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.fernet_utils [None req-e7dac062-3b9d-49df-9dea-99b0630a991d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92057) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 12:59:59.953238 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9665a48-c366-4e63-8171-a164ef538804 None None] REQUEST_METHOD: `POST` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 12:59:59.953814 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9665a48-c366-4e63-8171-a164ef538804 None None] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 12:59:59.954241 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9665a48-c366-4e63-8171-a164ef538804 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:00:00.043359 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.auth.core [None req-a9665a48-c366-4e63-8171-a164ef538804 None None] MFA Rules not processed for user `7d89f63934f048d999a46a322628dfa4`. Rule list: `[]` (Enabled: `True`). {{(pid=92056) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 13:00:00.046396 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.fernet_utils [None req-a9665a48-c366-4e63-8171-a164ef538804 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92056) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:00:00.159139 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 831/1660] 10.222.0.30 () {58 vars in 1057 bytes} [Mon May 23 12:59:59 2022] POST /identity/v3/auth/tokens => generated 3298 bytes in 274 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) May 23 13:00:00.167599 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7dac062-3b9d-49df-9dea-99b0630a991d None None] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:00:00.181598 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7dac062-3b9d-49df-9dea-99b0630a991d None None] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:00:00.189847 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cdc8089c-c3aa-4d13-94e8-0bfc9f92f3c6 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd5fb51f8d0fa46439cb29666970b885b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9c389ae65aaa4542bb5a5c4fd0bacf5b', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'service', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:00:00.191063 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdc8089c-c3aa-4d13-94e8-0bfc9f92f3c6 service placement] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:00:00.191782 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdc8089c-c3aa-4d13-94e8-0bfc9f92f3c6 service placement] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:00:00.192118 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdc8089c-c3aa-4d13-94e8-0bfc9f92f3c6 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:00:00.195864 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.fernet_utils [None req-cdc8089c-c3aa-4d13-94e8-0bfc9f92f3c6 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92057) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:00:00.214443 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cdc8089c-c3aa-4d13-94e8-0bfc9f92f3c6 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:00:00.217522 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cdc8089c-c3aa-4d13-94e8-0bfc9f92f3c6 service placement] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:00:00.256980 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 830/1661] 10.222.0.30 () {58 vars in 957 bytes} [Mon May 23 12:59:59 2022] POST /identity/v3/auth/tokens => generated 3337 bytes in 305 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) May 23 13:00:00.272496 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 832/1662] 10.222.0.30 () {60 vars in 1365 bytes} [Mon May 23 13:00:00 2022] GET /identity/v3/auth/tokens => generated 3298 bytes in 105 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) May 23 13:00:02.384661 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-577a3f52-0014-4ecb-9313-6e538ae37471 None None] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:00:02.385204 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-577a3f52-0014-4ecb-9313-6e538ae37471 None None] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:00:02.385204 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-577a3f52-0014-4ecb-9313-6e538ae37471 None None] PATH_INFO: `/` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:00:02.387519 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 831/1663] 10.222.0.30 () {54 vars in 921 bytes} [Mon May 23 13:00:02 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) May 23 13:00:02.409669 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b698dd4-1b2a-44e2-950b-792036d61606 None None] REQUEST_METHOD: `POST` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:00:02.410819 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b698dd4-1b2a-44e2-950b-792036d61606 None None] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:00:02.411382 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b698dd4-1b2a-44e2-950b-792036d61606 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:00:02.453560 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.auth.core [None req-3b698dd4-1b2a-44e2-950b-792036d61606 None None] MFA Rules not processed for user `d5fb51f8d0fa46439cb29666970b885b`. Rule list: `[]` (Enabled: `True`). {{(pid=92057) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 13:00:02.455457 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.fernet_utils [None req-3b698dd4-1b2a-44e2-950b-792036d61606 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92057) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:00:02.486871 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 833/1664] 10.222.0.30 () {58 vars in 1018 bytes} [Mon May 23 13:00:02 2022] POST /identity/v3/auth/tokens => generated 3298 bytes in 80 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) May 23 13:00:02.540541 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-577a3f52-0014-4ecb-9313-6e538ae37471 None None] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:00:02.566988 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-577a3f52-0014-4ecb-9313-6e538ae37471 None None] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:00:02.579967 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef08dcc9-df49-44cf-8b1b-d4c03d68ee66 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd5fb51f8d0fa46439cb29666970b885b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9c389ae65aaa4542bb5a5c4fd0bacf5b', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'service', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:00:02.580812 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef08dcc9-df49-44cf-8b1b-d4c03d68ee66 service placement] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:00:02.581159 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef08dcc9-df49-44cf-8b1b-d4c03d68ee66 service placement] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:00:02.581500 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef08dcc9-df49-44cf-8b1b-d4c03d68ee66 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:00:02.596639 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef08dcc9-df49-44cf-8b1b-d4c03d68ee66 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:00:02.598691 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef08dcc9-df49-44cf-8b1b-d4c03d68ee66 service placement] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:00:02.616037 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 832/1665] 10.222.0.30 () {60 vars in 1365 bytes} [Mon May 23 13:00:02 2022] GET /identity/v3/auth/tokens => generated 3298 bytes in 76 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) May 23 13:00:04.037576 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48684b4d-9d96-4d1d-93bc-4c97ac6bfbc0 None None] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:00:04.038246 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48684b4d-9d96-4d1d-93bc-4c97ac6bfbc0 None None] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:00:04.038246 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48684b4d-9d96-4d1d-93bc-4c97ac6bfbc0 None None] PATH_INFO: `/` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:00:04.039697 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 834/1666] 10.222.0.30 () {54 vars in 921 bytes} [Mon May 23 13:00:04 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) May 23 13:00:04.060729 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfa48ebb-6687-4d54-9a2e-9cd5ca19a2c1 None None] REQUEST_METHOD: `POST` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:00:04.061207 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfa48ebb-6687-4d54-9a2e-9cd5ca19a2c1 None None] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:00:04.061302 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfa48ebb-6687-4d54-9a2e-9cd5ca19a2c1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:00:04.082329 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.auth.core [None req-dfa48ebb-6687-4d54-9a2e-9cd5ca19a2c1 None None] MFA Rules not processed for user `d5fb51f8d0fa46439cb29666970b885b`. Rule list: `[]` (Enabled: `True`). {{(pid=92056) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 13:00:04.083892 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.fernet_utils [None req-dfa48ebb-6687-4d54-9a2e-9cd5ca19a2c1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92056) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:00:04.094651 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 833/1667] 10.222.0.30 () {58 vars in 1018 bytes} [Mon May 23 13:00:04 2022] POST /identity/v3/auth/tokens => generated 3298 bytes in 35 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) May 23 13:00:04.141263 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48684b4d-9d96-4d1d-93bc-4c97ac6bfbc0 None None] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:00:04.164948 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48684b4d-9d96-4d1d-93bc-4c97ac6bfbc0 None None] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:00:04.172105 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9bdb769c-c296-495e-a158-ef0012732498 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd5fb51f8d0fa46439cb29666970b885b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9c389ae65aaa4542bb5a5c4fd0bacf5b', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'service', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:00:04.173154 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bdb769c-c296-495e-a158-ef0012732498 service placement] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:00:04.173578 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bdb769c-c296-495e-a158-ef0012732498 service placement] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:00:04.174087 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bdb769c-c296-495e-a158-ef0012732498 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:00:04.190130 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9bdb769c-c296-495e-a158-ef0012732498 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:00:04.192538 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9bdb769c-c296-495e-a158-ef0012732498 service placement] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:00:04.226900 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 835/1668] 10.222.0.30 () {60 vars in 1365 bytes} [Mon May 23 13:00:04 2022] GET /identity/v3/auth/tokens => generated 3298 bytes in 83 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) May 23 13:00:04.324929 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dfa48ebb-6687-4d54-9a2e-9cd5ca19a2c1 None None] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:00:04.389263 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=100 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 13:00:04.405848 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbc16380-296f-419e-a59d-36e40d528a92 None None] REQUEST_METHOD: `POST` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:00:04.406300 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbc16380-296f-419e-a59d-36e40d528a92 None None] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:00:04.406689 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbc16380-296f-419e-a59d-36e40d528a92 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:00:04.469520 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.auth.core [None req-cbc16380-296f-419e-a59d-36e40d528a92 None None] MFA Rules not processed for user `257251830f5148e5b9e435c2c13828e8`. Rule list: `[]` (Enabled: `True`). {{(pid=92057) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 13:00:04.470961 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.fernet_utils [None req-cbc16380-296f-419e-a59d-36e40d528a92 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92057) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:00:04.476930 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dfa48ebb-6687-4d54-9a2e-9cd5ca19a2c1 None None] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:00:04.482950 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3463dba-a4f5-4995-b815-1fe88855282c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:00:04.484080 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3463dba-a4f5-4995-b815-1fe88855282c admin admin] REQUEST_METHOD: `DELETE` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:00:04.484499 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3463dba-a4f5-4995-b815-1fe88855282c admin admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:00:04.484906 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3463dba-a4f5-4995-b815-1fe88855282c admin admin] PATH_INFO: `/v3/users/8ea87f0f4b434124a60994ad3eaf277c` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:00:04.486457 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3463dba-a4f5-4995-b815-1fe88855282c admin admin] RBAC: Authorizing `identity:delete_user(user_id=8ea87f0f4b434124a60994ad3eaf277c)` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:00:04.487499 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: WARNING py.warnings [None req-b3463dba-a4f5-4995-b815-1fe88855282c admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:delete_user": "(role:admin and system_scope:all) or (role:admin and token.domain.id:%(target.user.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required May 23 13:00:04.487499 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: warnings.warn(msg) May 23 13:00:04.487499 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: May 23 13:00:04.493020 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3463dba-a4f5-4995-b815-1fe88855282c admin admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:00:04.554389 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=100 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 13:00:04.577512 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 836/1669] 10.222.0.30 () {58 vars in 957 bytes} [Mon May 23 13:00:04 2022] POST /identity/v3/auth/tokens => generated 3252 bytes in 173 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) May 23 13:00:04.594219 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cbc16380-296f-419e-a59d-36e40d528a92 None None] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:00:04.614626 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cbc16380-296f-419e-a59d-36e40d528a92 None None] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:00:04.622034 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44c1ac1a-4e01-42b2-b573-178f82074bc1 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cbe9931d94d44cc7ace0e83a7e5015fc', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9c389ae65aaa4542bb5a5c4fd0bacf5b', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'member', 'reader', 'service', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:00:04.622918 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44c1ac1a-4e01-42b2-b573-178f82074bc1 service nova] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:00:04.623342 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44c1ac1a-4e01-42b2-b573-178f82074bc1 service nova] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:00:04.623754 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44c1ac1a-4e01-42b2-b573-178f82074bc1 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:00:04.634707 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44c1ac1a-4e01-42b2-b573-178f82074bc1 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:00:04.637323 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44c1ac1a-4e01-42b2-b573-178f82074bc1 service nova] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:00:04.649198 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 837/1670] 10.222.0.30 () {60 vars in 1365 bytes} [Mon May 23 13:00:04 2022] GET /identity/v3/auth/tokens => generated 3252 bytes in 56 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) May 23 13:00:04.803002 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-b3463dba-a4f5-4995-b815-1fe88855282c admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '8ea87f0f4b434124a60994ad3eaf277c'} {{(pid=92056) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:00:04.805866 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-b3463dba-a4f5-4995-b815-1fe88855282c admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '8ea87f0f4b434124a60994ad3eaf277c'} {{(pid=92056) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:00:04.908228 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-b3463dba-a4f5-4995-b815-1fe88855282c admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '8ea87f0f4b434124a60994ad3eaf277c'} {{(pid=92056) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:00:04.950261 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-b3463dba-a4f5-4995-b815-1fe88855282c admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '8ea87f0f4b434124a60994ad3eaf277c'} {{(pid=92056) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:00:04.967491 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-b3463dba-a4f5-4995-b815-1fe88855282c admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '8ea87f0f4b434124a60994ad3eaf277c'} {{(pid=92056) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:00:04.975060 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 834/1671] 10.222.0.30 () {58 vars in 1223 bytes} [Mon May 23 13:00:04 2022] DELETE /identity/v3/users/8ea87f0f4b434124a60994ad3eaf277c => generated 0 bytes in 651 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) May 23 13:00:05.324342 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e746cf55-ea96-4f95-a418-41edb5136e28 None None] REQUEST_METHOD: `POST` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:00:05.325347 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e746cf55-ea96-4f95-a418-41edb5136e28 None None] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:00:05.325644 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e746cf55-ea96-4f95-a418-41edb5136e28 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:00:05.395061 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.auth.core [None req-e746cf55-ea96-4f95-a418-41edb5136e28 None None] MFA Rules not processed for user `b18e5f1b24634096995068e41d94cad6`. Rule list: `[]` (Enabled: `True`). {{(pid=92057) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 13:00:05.396272 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.fernet_utils [None req-e746cf55-ea96-4f95-a418-41edb5136e28 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92057) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:00:05.499353 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3463dba-a4f5-4995-b815-1fe88855282c admin admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:00:05.501328 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.fernet_utils [None req-b3463dba-a4f5-4995-b815-1fe88855282c admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92056) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:00:05.546531 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 838/1672] 10.222.0.30 () {58 vars in 957 bytes} [Mon May 23 13:00:05 2022] POST /identity/v3/auth/tokens => generated 3258 bytes in 224 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) May 23 13:00:05.558752 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e475081-4a02-44f4-9121-9c7d4f111b3e None None] REQUEST_METHOD: `POST` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:00:05.559038 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e475081-4a02-44f4-9121-9c7d4f111b3e None None] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:00:05.559323 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e475081-4a02-44f4-9121-9c7d4f111b3e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:00:05.591335 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.auth.core [None req-6e475081-4a02-44f4-9121-9c7d4f111b3e None None] MFA Rules not processed for user `b18e5f1b24634096995068e41d94cad6`. Rule list: `[]` (Enabled: `True`). {{(pid=92057) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 13:00:05.594879 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.fernet_utils [None req-6e475081-4a02-44f4-9121-9c7d4f111b3e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92057) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:00:05.614577 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 839/1673] 10.222.0.30 () {58 vars in 957 bytes} [Mon May 23 13:00:05 2022] POST /identity/v3/auth/tokens => generated 3258 bytes in 57 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) May 23 13:00:05.628398 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e475081-4a02-44f4-9121-9c7d4f111b3e None None] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:00:05.629821 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.fernet_utils [None req-6e475081-4a02-44f4-9121-9c7d4f111b3e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92057) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:00:05.786484 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e475081-4a02-44f4-9121-9c7d4f111b3e None None] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:00:05.795483 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e02cbc96-81b2-45cf-a9da-b868990da1d8 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cbe9931d94d44cc7ace0e83a7e5015fc', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9c389ae65aaa4542bb5a5c4fd0bacf5b', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'member', 'reader', 'service', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:00:05.796214 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e02cbc96-81b2-45cf-a9da-b868990da1d8 service nova] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:00:05.796699 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e02cbc96-81b2-45cf-a9da-b868990da1d8 service nova] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:00:05.801495 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e02cbc96-81b2-45cf-a9da-b868990da1d8 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:00:05.813367 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e02cbc96-81b2-45cf-a9da-b868990da1d8 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:00:05.815149 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e02cbc96-81b2-45cf-a9da-b868990da1d8 service nova] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:00:05.835397 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3463dba-a4f5-4995-b815-1fe88855282c admin admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:00:05.840883 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30ebe85f-fec1-43bc-987e-9797a0d58104 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:00:05.842282 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30ebe85f-fec1-43bc-987e-9797a0d58104 admin admin] REQUEST_METHOD: `DELETE` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:00:05.842695 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30ebe85f-fec1-43bc-987e-9797a0d58104 admin admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:00:05.843044 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30ebe85f-fec1-43bc-987e-9797a0d58104 admin admin] PATH_INFO: `/v3/projects/a840824e38be4e3aa8f0f3e09c3fee14` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:00:05.844832 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-30ebe85f-fec1-43bc-987e-9797a0d58104 admin admin] RBAC: Authorizing `identity:delete_project(project_id=a840824e38be4e3aa8f0f3e09c3fee14)` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:00:05.847403 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-30ebe85f-fec1-43bc-987e-9797a0d58104 admin admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:00:05.857558 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 840/1674] 10.222.0.30 () {60 vars in 1365 bytes} [Mon May 23 13:00:05 2022] GET /identity/v3/auth/tokens => generated 3258 bytes in 229 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) May 23 13:00:05.999091 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=87,keystone:DELETE=12,keystone:INSERT=1 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 13:00:06.009394 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-30ebe85f-fec1-43bc-987e-9797a0d58104 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'a840824e38be4e3aa8f0f3e09c3fee14', 'request_id': 'req-30ebe85f-fec1-43bc-987e-9797a0d58104'} {{(pid=92056) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:00:06.087793 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-30ebe85f-fec1-43bc-987e-9797a0d58104 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'a840824e38be4e3aa8f0f3e09c3fee14', 'request_id': 'req-30ebe85f-fec1-43bc-987e-9797a0d58104'} {{(pid=92056) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:00:06.095721 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-30ebe85f-fec1-43bc-987e-9797a0d58104 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'a840824e38be4e3aa8f0f3e09c3fee14', 'request_id': 'req-30ebe85f-fec1-43bc-987e-9797a0d58104'} {{(pid=92056) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:00:06.108148 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-30ebe85f-fec1-43bc-987e-9797a0d58104 admin admin] The token cache is being invalidate because project a840824e38be4e3aa8f0f3e09c3fee14 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=92056) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} May 23 13:00:06.108588 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-30ebe85f-fec1-43bc-987e-9797a0d58104 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=92056) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:00:06.109307 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-30ebe85f-fec1-43bc-987e-9797a0d58104 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=92056) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:00:06.111275 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 835/1675] 10.222.0.30 () {58 vars in 1232 bytes} [Mon May 23 13:00:05 2022] DELETE /identity/v3/projects/a840824e38be4e3aa8f0f3e09c3fee14 => generated 0 bytes in 612 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) May 23 13:00:06.114999 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e02cbc96-81b2-45cf-a9da-b868990da1d8 service nova] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:00:06.126036 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e02cbc96-81b2-45cf-a9da-b868990da1d8 service nova] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:00:06.182466 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=100 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 13:00:06.201624 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a707416-214b-46bf-99a3-6448179a4597 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:00:06.202533 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a707416-214b-46bf-99a3-6448179a4597 admin admin] REQUEST_METHOD: `DELETE` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:00:06.203358 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a707416-214b-46bf-99a3-6448179a4597 admin admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:00:06.203725 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a707416-214b-46bf-99a3-6448179a4597 admin admin] PATH_INFO: `/v3/users/7d89f63934f048d999a46a322628dfa4` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:00:06.204944 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a707416-214b-46bf-99a3-6448179a4597 admin admin] RBAC: Authorizing `identity:delete_user(user_id=7d89f63934f048d999a46a322628dfa4)` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:00:06.206911 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a707416-214b-46bf-99a3-6448179a4597 admin admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:00:06.441123 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [None req-4a707416-214b-46bf-99a3-6448179a4597 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '7d89f63934f048d999a46a322628dfa4'} {{(pid=92057) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:00:06.547486 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [None req-4a707416-214b-46bf-99a3-6448179a4597 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '7d89f63934f048d999a46a322628dfa4'} {{(pid=92057) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:00:06.562464 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [None req-4a707416-214b-46bf-99a3-6448179a4597 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '7d89f63934f048d999a46a322628dfa4'} {{(pid=92057) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:00:06.565224 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [None req-4a707416-214b-46bf-99a3-6448179a4597 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '7d89f63934f048d999a46a322628dfa4'} {{(pid=92057) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:00:06.603142 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [None req-4a707416-214b-46bf-99a3-6448179a4597 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '7d89f63934f048d999a46a322628dfa4'} {{(pid=92057) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:00:06.611262 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 841/1676] 10.222.0.30 () {58 vars in 1223 bytes} [Mon May 23 13:00:06 2022] DELETE /identity/v3/users/7d89f63934f048d999a46a322628dfa4 => generated 0 bytes in 497 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) May 23 13:00:07.051841 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fbed113-b51e-4702-9fad-6bf3f191996c None None] REQUEST_METHOD: `POST` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:00:07.057189 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fbed113-b51e-4702-9fad-6bf3f191996c None None] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:00:07.057189 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fbed113-b51e-4702-9fad-6bf3f191996c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:00:07.080334 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.auth.core [None req-3fbed113-b51e-4702-9fad-6bf3f191996c None None] MFA Rules not processed for user `b18e5f1b24634096995068e41d94cad6`. Rule list: `[]` (Enabled: `True`). {{(pid=92056) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 13:00:07.086038 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.fernet_utils [None req-3fbed113-b51e-4702-9fad-6bf3f191996c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92056) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:00:07.207355 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a707416-214b-46bf-99a3-6448179a4597 admin admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:00:07.208768 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.fernet_utils [None req-4a707416-214b-46bf-99a3-6448179a4597 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92057) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:00:07.258726 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 836/1677] 10.222.0.30 () {58 vars in 957 bytes} [Mon May 23 13:00:07 2022] POST /identity/v3/auth/tokens => generated 3258 bytes in 208 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) May 23 13:00:07.349649 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a707416-214b-46bf-99a3-6448179a4597 admin admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:00:07.373828 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8e38d34-ac13-4599-9fb3-c37c0c2dbf02 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:00:07.373828 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8e38d34-ac13-4599-9fb3-c37c0c2dbf02 admin admin] REQUEST_METHOD: `DELETE` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:00:07.373828 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8e38d34-ac13-4599-9fb3-c37c0c2dbf02 admin admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:00:07.373828 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8e38d34-ac13-4599-9fb3-c37c0c2dbf02 admin admin] PATH_INFO: `/v3/projects/993a79a363064d3fab6dfc5e93442b47` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:00:07.374541 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8e38d34-ac13-4599-9fb3-c37c0c2dbf02 admin admin] RBAC: Authorizing `identity:delete_project(project_id=993a79a363064d3fab6dfc5e93442b47)` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:00:07.374541 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: WARNING py.warnings [None req-a8e38d34-ac13-4599-9fb3-c37c0c2dbf02 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:delete_project": "(role:admin and system_scope:all) or (role:admin and domain_id:%(target.project.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required May 23 13:00:07.374541 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: warnings.warn(msg) May 23 13:00:07.374541 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: May 23 13:00:07.374541 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8e38d34-ac13-4599-9fb3-c37c0c2dbf02 admin admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:00:07.646652 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [None req-a8e38d34-ac13-4599-9fb3-c37c0c2dbf02 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '993a79a363064d3fab6dfc5e93442b47', 'request_id': 'req-a8e38d34-ac13-4599-9fb3-c37c0c2dbf02'} {{(pid=92057) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:00:07.656978 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [None req-a8e38d34-ac13-4599-9fb3-c37c0c2dbf02 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '993a79a363064d3fab6dfc5e93442b47', 'request_id': 'req-a8e38d34-ac13-4599-9fb3-c37c0c2dbf02'} {{(pid=92057) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:00:07.746712 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [None req-a8e38d34-ac13-4599-9fb3-c37c0c2dbf02 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '993a79a363064d3fab6dfc5e93442b47', 'request_id': 'req-a8e38d34-ac13-4599-9fb3-c37c0c2dbf02'} {{(pid=92057) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:00:07.771056 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=81,keystone:DELETE=17,keystone:INSERT=2 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 13:00:07.786481 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [None req-a8e38d34-ac13-4599-9fb3-c37c0c2dbf02 admin admin] The token cache is being invalidate because project 993a79a363064d3fab6dfc5e93442b47 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=92057) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} May 23 13:00:07.786481 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [None req-a8e38d34-ac13-4599-9fb3-c37c0c2dbf02 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=92057) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:00:07.787044 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [None req-a8e38d34-ac13-4599-9fb3-c37c0c2dbf02 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=92057) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:00:07.788847 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 842/1678] 10.222.0.30 () {58 vars in 1232 bytes} [Mon May 23 13:00:07 2022] DELETE /identity/v3/projects/993a79a363064d3fab6dfc5e93442b47 => generated 0 bytes in 582 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) May 23 13:00:07.795376 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a46626f-e128-4706-a674-6cdd14e96774 None None] REQUEST_METHOD: `POST` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:00:07.795855 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a46626f-e128-4706-a674-6cdd14e96774 None None] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:00:07.796712 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a46626f-e128-4706-a674-6cdd14e96774 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:00:07.836535 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.auth.core [None req-4a46626f-e128-4706-a674-6cdd14e96774 None None] MFA Rules not processed for user `0f7119f8fa5c4445977671140ded91a4`. Rule list: `[]` (Enabled: `True`). {{(pid=92056) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 13:00:07.837792 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.fernet_utils [None req-4a46626f-e128-4706-a674-6cdd14e96774 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92056) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:00:07.974040 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 837/1679] 10.222.0.30 () {58 vars in 957 bytes} [Mon May 23 13:00:07 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 180 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) May 23 13:00:07.982568 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f41f4f6b-23a2-4efb-926a-ae60706bc512 None None] REQUEST_METHOD: `POST` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:00:07.982743 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f41f4f6b-23a2-4efb-926a-ae60706bc512 None None] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:00:07.982987 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f41f4f6b-23a2-4efb-926a-ae60706bc512 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:00:08.031427 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.auth.core [None req-f41f4f6b-23a2-4efb-926a-ae60706bc512 None None] MFA Rules not processed for user `0f7119f8fa5c4445977671140ded91a4`. Rule list: `[]` (Enabled: `True`). {{(pid=92057) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 13:00:08.034391 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.fernet_utils [None req-f41f4f6b-23a2-4efb-926a-ae60706bc512 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92057) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:00:08.066645 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 843/1680] 10.222.0.30 () {58 vars in 957 bytes} [Mon May 23 13:00:07 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) May 23 13:00:08.067694 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a46626f-e128-4706-a674-6cdd14e96774 None None] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:00:08.081513 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a46626f-e128-4706-a674-6cdd14e96774 None None] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:00:08.094951 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72636f7d-149e-4565-a0a4-6dc3cca918e8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:00:08.096302 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72636f7d-149e-4565-a0a4-6dc3cca918e8 admin admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:00:08.096586 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72636f7d-149e-4565-a0a4-6dc3cca918e8 admin admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:00:08.096867 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72636f7d-149e-4565-a0a4-6dc3cca918e8 admin admin] PATH_INFO: `/v3/domains` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:00:08.097611 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72636f7d-149e-4565-a0a4-6dc3cca918e8 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=92056) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} May 23 13:00:08.097951 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72636f7d-149e-4565-a0a4-6dc3cca918e8 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:00:08.110078 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72636f7d-149e-4565-a0a4-6dc3cca918e8 admin admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:00:08.122937 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 838/1681] 10.222.0.30 () {58 vars in 1152 bytes} [Mon May 23 13:00:08 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 56 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) May 23 13:00:08.127278 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f41f4f6b-23a2-4efb-926a-ae60706bc512 None None] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:00:08.139632 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f41f4f6b-23a2-4efb-926a-ae60706bc512 None None] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:00:08.153649 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b851f84-438f-4100-95a4-78c5d935dc4d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:00:08.154770 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b851f84-438f-4100-95a4-78c5d935dc4d admin admin] REQUEST_METHOD: `POST` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:00:08.154770 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b851f84-438f-4100-95a4-78c5d935dc4d admin admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:00:08.154979 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b851f84-438f-4100-95a4-78c5d935dc4d admin admin] PATH_INFO: `/v3/projects` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:00:08.155635 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b851f84-438f-4100-95a4-78c5d935dc4d admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:00:08.158574 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b851f84-438f-4100-95a4-78c5d935dc4d admin admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:00:08.293061 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 844/1682] 10.222.0.30 () {60 vars in 1152 bytes} [Mon May 23 13:00:08 2022] POST /identity/v3/projects => generated 361 bytes in 166 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) May 23 13:00:08.294242 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72636f7d-149e-4565-a0a4-6dc3cca918e8 admin admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:00:08.307554 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72636f7d-149e-4565-a0a4-6dc3cca918e8 admin admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:00:08.477445 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:DELETE=5,keystone:SELECT=94,keystone:INSERT=1 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 13:00:08.494326 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75db6efd-522e-4446-8e37-07504b0d3c50 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:00:08.496085 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75db6efd-522e-4446-8e37-07504b0d3c50 admin admin] REQUEST_METHOD: `POST` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:00:08.497176 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75db6efd-522e-4446-8e37-07504b0d3c50 admin admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:00:08.497928 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75db6efd-522e-4446-8e37-07504b0d3c50 admin admin] PATH_INFO: `/v3/users` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:00:08.499480 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75db6efd-522e-4446-8e37-07504b0d3c50 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:00:08.504219 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75db6efd-522e-4446-8e37-07504b0d3c50 admin admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:00:08.625970 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 839/1683] 10.222.0.30 () {60 vars in 1143 bytes} [Mon May 23 13:00:08 2022] POST /identity/v3/users => generated 281 bytes in 332 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) May 23 13:00:08.635573 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b851f84-438f-4100-95a4-78c5d935dc4d admin admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:00:08.648734 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b851f84-438f-4100-95a4-78c5d935dc4d admin admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:00:08.663248 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37f4b262-ace9-47bd-94ad-0ff5f0705b34 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:00:08.664974 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37f4b262-ace9-47bd-94ad-0ff5f0705b34 admin admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:00:08.665272 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37f4b262-ace9-47bd-94ad-0ff5f0705b34 admin admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:00:08.666009 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37f4b262-ace9-47bd-94ad-0ff5f0705b34 admin admin] PATH_INFO: `/v3/roles` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:00:08.666352 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37f4b262-ace9-47bd-94ad-0ff5f0705b34 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:00:08.675449 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37f4b262-ace9-47bd-94ad-0ff5f0705b34 admin admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:00:08.701569 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75db6efd-522e-4446-8e37-07504b0d3c50 admin admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:00:08.708909 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 845/1684] 10.222.0.30 () {58 vars in 1121 bytes} [Mon May 23 13:00:08 2022] GET /identity/v3/roles => generated 1420 bytes in 68 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) May 23 13:00:08.713234 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75db6efd-522e-4446-8e37-07504b0d3c50 admin admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:00:08.719475 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9919bb3b-e2cc-434a-9eee-fb689911ac63 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:00:08.720350 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9919bb3b-e2cc-434a-9eee-fb689911ac63 admin admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:00:08.720814 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9919bb3b-e2cc-434a-9eee-fb689911ac63 admin admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:00:08.721234 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9919bb3b-e2cc-434a-9eee-fb689911ac63 admin admin] PATH_INFO: `/v3/roles` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:00:08.722167 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9919bb3b-e2cc-434a-9eee-fb689911ac63 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:00:08.724246 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9919bb3b-e2cc-434a-9eee-fb689911ac63 admin admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:00:08.774525 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37f4b262-ace9-47bd-94ad-0ff5f0705b34 admin admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:00:08.783192 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 840/1685] 10.222.0.30 () {58 vars in 1121 bytes} [Mon May 23 13:00:08 2022] GET /identity/v3/roles => generated 1420 bytes in 81 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) May 23 13:00:08.785678 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37f4b262-ace9-47bd-94ad-0ff5f0705b34 admin admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:00:08.798707 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-456ab333-5ca6-4e14-a500-4adc8585b534 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:00:08.799899 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-456ab333-5ca6-4e14-a500-4adc8585b534 admin admin] REQUEST_METHOD: `PUT` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:00:08.800329 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-456ab333-5ca6-4e14-a500-4adc8585b534 admin admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:00:08.800768 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-456ab333-5ca6-4e14-a500-4adc8585b534 admin admin] PATH_INFO: `/v3/projects/6f29f2c0c95249789535d751bc85691a/users/43864f9047cb45779972178ca5088e4b/roles/80fb22507c9a4ff09d44d8f12f6fff1a` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:00:08.820932 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-456ab333-5ca6-4e14-a500-4adc8585b534 admin admin] RBAC: Authorizing `identity:create_grant(project_id=6f29f2c0c95249789535d751bc85691a, user_id=43864f9047cb45779972178ca5088e4b, role_id=80fb22507c9a4ff09d44d8f12f6fff1a)` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:00:08.824890 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-456ab333-5ca6-4e14-a500-4adc8585b534 admin admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:00:08.960232 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 846/1686] 10.222.0.30 () {60 vars in 1482 bytes} [Mon May 23 13:00:08 2022] PUT /identity/v3/projects/6f29f2c0c95249789535d751bc85691a/users/43864f9047cb45779972178ca5088e4b/roles/80fb22507c9a4ff09d44d8f12f6fff1a => generated 0 bytes in 184 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) May 23 13:00:08.976425 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87856b59-d076-458c-9f6c-67978ec6f451 None None] REQUEST_METHOD: `POST` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:00:08.976713 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87856b59-d076-458c-9f6c-67978ec6f451 None None] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:00:08.977365 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87856b59-d076-458c-9f6c-67978ec6f451 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:00:08.982684 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c47ce35-2bf4-4d74-b85f-d4d81824ce1a None None] REQUEST_METHOD: `POST` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:00:08.982978 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c47ce35-2bf4-4d74-b85f-d4d81824ce1a None None] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:00:08.983257 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c47ce35-2bf4-4d74-b85f-d4d81824ce1a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:00:09.002622 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.auth.core [None req-87856b59-d076-458c-9f6c-67978ec6f451 None None] MFA Rules not processed for user `257251830f5148e5b9e435c2c13828e8`. Rule list: `[]` (Enabled: `True`). {{(pid=92056) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 13:00:09.004054 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.fernet_utils [None req-87856b59-d076-458c-9f6c-67978ec6f451 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92056) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:00:09.023470 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.auth.core [None req-4c47ce35-2bf4-4d74-b85f-d4d81824ce1a None None] MFA Rules not processed for user `43864f9047cb45779972178ca5088e4b`. Rule list: `[]` (Enabled: `True`). {{(pid=92057) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 13:00:09.025078 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.fernet_utils [None req-4c47ce35-2bf4-4d74-b85f-d4d81824ce1a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92057) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:00:09.153277 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 847/1687] 10.222.0.30 () {58 vars in 957 bytes} [Mon May 23 13:00:08 2022] POST /identity/v3/auth/tokens => generated 3228 bytes in 174 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) May 23 13:00:09.158230 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c47ce35-2bf4-4d74-b85f-d4d81824ce1a None None] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:00:09.166707 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 841/1688] 10.222.0.30 () {58 vars in 957 bytes} [Mon May 23 13:00:08 2022] POST /identity/v3/auth/tokens => generated 3252 bytes in 193 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) May 23 13:00:09.252256 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c47ce35-2bf4-4d74-b85f-d4d81824ce1a None None] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:00:09.258217 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22283f6f-ce8c-4af7-9eec-ac3b670289fb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:00:09.258966 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22283f6f-ce8c-4af7-9eec-ac3b670289fb admin admin] REQUEST_METHOD: `POST` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:00:09.259307 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22283f6f-ce8c-4af7-9eec-ac3b670289fb admin admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:00:09.259644 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22283f6f-ce8c-4af7-9eec-ac3b670289fb admin admin] PATH_INFO: `/v3/projects` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:00:09.261450 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-22283f6f-ce8c-4af7-9eec-ac3b670289fb admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:00:09.264147 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-22283f6f-ce8c-4af7-9eec-ac3b670289fb admin admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:00:09.430959 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 848/1689] 10.222.0.30 () {60 vars in 1152 bytes} [Mon May 23 13:00:09 2022] POST /identity/v3/projects => generated 365 bytes in 271 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) May 23 13:00:09.432357 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87856b59-d076-458c-9f6c-67978ec6f451 None None] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:00:09.447985 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87856b59-d076-458c-9f6c-67978ec6f451 None None] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:00:09.559826 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd056251-b9e2-45ad-9db4-98b846da4541 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:00:09.559826 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd056251-b9e2-45ad-9db4-98b846da4541 admin admin] REQUEST_METHOD: `POST` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:00:09.559826 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd056251-b9e2-45ad-9db4-98b846da4541 admin admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:00:09.559826 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd056251-b9e2-45ad-9db4-98b846da4541 admin admin] PATH_INFO: `/v3/users` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:00:09.561364 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd056251-b9e2-45ad-9db4-98b846da4541 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:00:09.561364 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd056251-b9e2-45ad-9db4-98b846da4541 admin admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:00:09.678492 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 842/1690] 10.222.0.30 () {60 vars in 1143 bytes} [Mon May 23 13:00:09 2022] POST /identity/v3/users => generated 289 bytes in 246 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) May 23 13:00:09.678649 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22283f6f-ce8c-4af7-9eec-ac3b670289fb admin admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:00:09.700089 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22283f6f-ce8c-4af7-9eec-ac3b670289fb admin admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:00:09.705914 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d96c223-09cc-4039-9c92-720a6bf4b35b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:00:09.706609 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d96c223-09cc-4039-9c92-720a6bf4b35b admin admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:00:09.713792 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d96c223-09cc-4039-9c92-720a6bf4b35b admin admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:00:09.714504 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d96c223-09cc-4039-9c92-720a6bf4b35b admin admin] PATH_INFO: `/v3/roles` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:00:09.715472 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d96c223-09cc-4039-9c92-720a6bf4b35b admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:00:09.717292 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d96c223-09cc-4039-9c92-720a6bf4b35b admin admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:00:09.719242 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 13:00:09.731671 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 849/1691] 10.222.0.30 () {58 vars in 1121 bytes} [Mon May 23 13:00:09 2022] GET /identity/v3/roles => generated 1420 bytes in 54 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) May 23 13:00:09.736190 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd056251-b9e2-45ad-9db4-98b846da4541 admin admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:00:09.746889 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd056251-b9e2-45ad-9db4-98b846da4541 admin admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:00:09.751868 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9a10fee-d7c9-4385-a22b-8d0664fcdafb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:00:09.752776 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9a10fee-d7c9-4385-a22b-8d0664fcdafb admin admin] REQUEST_METHOD: `PUT` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:00:09.753068 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9a10fee-d7c9-4385-a22b-8d0664fcdafb admin admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:00:09.753346 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9a10fee-d7c9-4385-a22b-8d0664fcdafb admin admin] PATH_INFO: `/v3/projects/9a92e19888894a078cef943afa1841dc/users/2c4fb8716d944288ab4b19170a8d83f6/roles/bc3c3c9d654940cbaa5177a6be23d1bc` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:00:09.772620 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9a10fee-d7c9-4385-a22b-8d0664fcdafb admin admin] RBAC: Authorizing `identity:create_grant(project_id=9a92e19888894a078cef943afa1841dc, user_id=2c4fb8716d944288ab4b19170a8d83f6, role_id=bc3c3c9d654940cbaa5177a6be23d1bc)` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:00:09.775413 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9a10fee-d7c9-4385-a22b-8d0664fcdafb admin admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:00:09.899370 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 843/1692] 10.222.0.30 () {60 vars in 1482 bytes} [Mon May 23 13:00:09 2022] PUT /identity/v3/projects/9a92e19888894a078cef943afa1841dc/users/2c4fb8716d944288ab4b19170a8d83f6/roles/bc3c3c9d654940cbaa5177a6be23d1bc => generated 0 bytes in 163 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) May 23 13:00:09.903488 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d96c223-09cc-4039-9c92-720a6bf4b35b admin admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:00:09.921967 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d96c223-09cc-4039-9c92-720a6bf4b35b admin admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:00:10.023846 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-abecdcfd-572f-4fcd-82b0-009f3f271ebe admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:00:10.024541 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abecdcfd-572f-4fcd-82b0-009f3f271ebe admin admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:00:10.024817 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abecdcfd-572f-4fcd-82b0-009f3f271ebe admin admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:00:10.025179 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abecdcfd-572f-4fcd-82b0-009f3f271ebe admin admin] PATH_INFO: `/v3/roles` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:00:10.026006 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-abecdcfd-572f-4fcd-82b0-009f3f271ebe admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:00:10.030238 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-abecdcfd-572f-4fcd-82b0-009f3f271ebe admin admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:00:10.047943 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 850/1693] 10.222.0.30 () {58 vars in 1121 bytes} [Mon May 23 13:00:09 2022] GET /identity/v3/roles => generated 1420 bytes in 142 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) May 23 13:00:10.048641 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9a10fee-d7c9-4385-a22b-8d0664fcdafb admin admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:00:10.067261 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9a10fee-d7c9-4385-a22b-8d0664fcdafb admin admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:00:10.081756 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47ef47ce-a18f-4222-a6b7-9c62eaa16b71 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:00:10.082699 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47ef47ce-a18f-4222-a6b7-9c62eaa16b71 admin admin] REQUEST_METHOD: `PUT` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:00:10.082989 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47ef47ce-a18f-4222-a6b7-9c62eaa16b71 admin admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:00:10.083269 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47ef47ce-a18f-4222-a6b7-9c62eaa16b71 admin admin] PATH_INFO: `/v3/projects/9a92e19888894a078cef943afa1841dc/users/2c4fb8716d944288ab4b19170a8d83f6/roles/bc3c3c9d654940cbaa5177a6be23d1bc` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:00:10.098808 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47ef47ce-a18f-4222-a6b7-9c62eaa16b71 admin admin] RBAC: Authorizing `identity:create_grant(project_id=9a92e19888894a078cef943afa1841dc, user_id=2c4fb8716d944288ab4b19170a8d83f6, role_id=bc3c3c9d654940cbaa5177a6be23d1bc)` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:00:10.101559 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47ef47ce-a18f-4222-a6b7-9c62eaa16b71 admin admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:00:10.142153 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fe625db-0357-494d-a788-cf0720b14566 None None] REQUEST_METHOD: `POST` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:00:10.146306 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fe625db-0357-494d-a788-cf0720b14566 None None] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:00:10.149852 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fe625db-0357-494d-a788-cf0720b14566 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:00:10.151946 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 844/1694] 10.222.0.30 () {60 vars in 1482 bytes} [Mon May 23 13:00:10 2022] PUT /identity/v3/projects/9a92e19888894a078cef943afa1841dc/users/2c4fb8716d944288ab4b19170a8d83f6/roles/bc3c3c9d654940cbaa5177a6be23d1bc => generated 0 bytes in 79 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) May 23 13:00:10.188363 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.auth.core [None req-9fe625db-0357-494d-a788-cf0720b14566 None None] MFA Rules not processed for user `2c4fb8716d944288ab4b19170a8d83f6`. Rule list: `[]` (Enabled: `True`). {{(pid=92057) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 13:00:10.189753 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.fernet_utils [None req-9fe625db-0357-494d-a788-cf0720b14566 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92057) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:00:10.305003 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 851/1695] 10.222.0.30 () {58 vars in 957 bytes} [Mon May 23 13:00:10 2022] POST /identity/v3/auth/tokens => generated 3299 bytes in 164 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) May 23 13:00:10.318764 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47ef47ce-a18f-4222-a6b7-9c62eaa16b71 admin admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:00:10.326095 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.fernet_utils [None req-47ef47ce-a18f-4222-a6b7-9c62eaa16b71 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92056) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:00:10.393505 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=92,keystone:INSERT=8 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 13:00:10.429078 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47ef47ce-a18f-4222-a6b7-9c62eaa16b71 admin admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:00:10.436472 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac360b61-30fd-4754-8f1f-7c4047d0086c service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1666097f8ed545c6b0e4809c93e4775a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9c389ae65aaa4542bb5a5c4fd0bacf5b', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:00:10.437136 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac360b61-30fd-4754-8f1f-7c4047d0086c service cinder] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:00:10.437419 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac360b61-30fd-4754-8f1f-7c4047d0086c service cinder] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:00:10.437693 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac360b61-30fd-4754-8f1f-7c4047d0086c service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:00:10.448993 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac360b61-30fd-4754-8f1f-7c4047d0086c service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:00:10.450804 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac360b61-30fd-4754-8f1f-7c4047d0086c service cinder] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:00:10.464612 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 845/1696] 10.222.0.30 () {60 vars in 1365 bytes} [Mon May 23 13:00:10 2022] GET /identity/v3/auth/tokens => generated 3299 bytes in 146 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) May 23 13:00:11.550559 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9fe625db-0357-494d-a788-cf0720b14566 None None] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:00:11.638982 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9fe625db-0357-494d-a788-cf0720b14566 None None] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:00:11.643607 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e269fac4-f0fe-430c-8209-cf5f706277dd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:00:11.644474 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e269fac4-f0fe-430c-8209-cf5f706277dd admin admin] REQUEST_METHOD: `DELETE` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:00:11.644753 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e269fac4-f0fe-430c-8209-cf5f706277dd admin admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:00:11.645031 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e269fac4-f0fe-430c-8209-cf5f706277dd admin admin] PATH_INFO: `/v3/users/43864f9047cb45779972178ca5088e4b` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:00:11.646235 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e269fac4-f0fe-430c-8209-cf5f706277dd admin admin] RBAC: Authorizing `identity:delete_user(user_id=43864f9047cb45779972178ca5088e4b)` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:00:11.648381 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e269fac4-f0fe-430c-8209-cf5f706277dd admin admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:00:11.676179 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=98,keystone:DELETE=2 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 13:00:11.908934 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [None req-e269fac4-f0fe-430c-8209-cf5f706277dd admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '43864f9047cb45779972178ca5088e4b'} {{(pid=92057) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:00:12.020882 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [None req-e269fac4-f0fe-430c-8209-cf5f706277dd admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '43864f9047cb45779972178ca5088e4b'} {{(pid=92057) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:00:12.030892 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [None req-e269fac4-f0fe-430c-8209-cf5f706277dd admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '43864f9047cb45779972178ca5088e4b'} {{(pid=92057) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:00:12.031741 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [None req-e269fac4-f0fe-430c-8209-cf5f706277dd admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '43864f9047cb45779972178ca5088e4b'} {{(pid=92057) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:00:12.047832 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [None req-e269fac4-f0fe-430c-8209-cf5f706277dd admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '43864f9047cb45779972178ca5088e4b'} {{(pid=92057) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:00:12.054882 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 852/1697] 10.222.0.30 () {58 vars in 1223 bytes} [Mon May 23 13:00:11 2022] DELETE /identity/v3/users/43864f9047cb45779972178ca5088e4b => generated 0 bytes in 505 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) May 23 13:00:12.059491 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23535281-43b9-4398-aaef-1ab6c820eef8 None None] REQUEST_METHOD: `POST` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:00:12.059995 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23535281-43b9-4398-aaef-1ab6c820eef8 None None] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:00:12.060084 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23535281-43b9-4398-aaef-1ab6c820eef8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:00:12.077489 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.auth.core [None req-23535281-43b9-4398-aaef-1ab6c820eef8 None None] MFA Rules not processed for user `0f7119f8fa5c4445977671140ded91a4`. Rule list: `[]` (Enabled: `True`). {{(pid=92056) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 13:00:12.078794 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.fernet_utils [None req-23535281-43b9-4398-aaef-1ab6c820eef8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92056) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:00:12.169269 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 846/1698] 10.222.0.30 () {58 vars in 957 bytes} [Mon May 23 13:00:12 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 111 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) May 23 13:00:12.179340 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e269fac4-f0fe-430c-8209-cf5f706277dd admin admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:00:12.181270 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.fernet_utils [None req-e269fac4-f0fe-430c-8209-cf5f706277dd admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92057) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:00:12.259511 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e269fac4-f0fe-430c-8209-cf5f706277dd admin admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:00:12.265480 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64250f7d-dcb1-4347-8afa-b57c11d2da57 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2eb83ca38e79449095f3a2f947bbb06a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9c389ae65aaa4542bb5a5c4fd0bacf5b', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:00:12.266254 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64250f7d-dcb1-4347-8afa-b57c11d2da57 service neutron] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:00:12.266549 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64250f7d-dcb1-4347-8afa-b57c11d2da57 service neutron] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:00:12.266821 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64250f7d-dcb1-4347-8afa-b57c11d2da57 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:00:12.287327 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-64250f7d-dcb1-4347-8afa-b57c11d2da57 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:00:12.289733 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-64250f7d-dcb1-4347-8afa-b57c11d2da57 service neutron] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:00:12.310221 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 853/1699] 10.222.0.30 () {60 vars in 1365 bytes} [Mon May 23 13:00:12 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 131 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) May 23 13:00:12.881139 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23535281-43b9-4398-aaef-1ab6c820eef8 None None] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:00:12.882641 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.fernet_utils [None req-23535281-43b9-4398-aaef-1ab6c820eef8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92056) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:00:12.897787 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23535281-43b9-4398-aaef-1ab6c820eef8 None None] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:00:12.904220 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e195eff-1641-481e-a7c0-c40dbd2edfc6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:00:12.905422 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e195eff-1641-481e-a7c0-c40dbd2edfc6 admin admin] REQUEST_METHOD: `DELETE` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:00:12.905803 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e195eff-1641-481e-a7c0-c40dbd2edfc6 admin admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:00:12.906191 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e195eff-1641-481e-a7c0-c40dbd2edfc6 admin admin] PATH_INFO: `/v3/projects/6f29f2c0c95249789535d751bc85691a` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:00:12.914618 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e195eff-1641-481e-a7c0-c40dbd2edfc6 admin admin] RBAC: Authorizing `identity:delete_project(project_id=6f29f2c0c95249789535d751bc85691a)` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:00:12.917831 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e195eff-1641-481e-a7c0-c40dbd2edfc6 admin admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:00:13.071277 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-5e195eff-1641-481e-a7c0-c40dbd2edfc6 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '6f29f2c0c95249789535d751bc85691a', 'request_id': 'req-5e195eff-1641-481e-a7c0-c40dbd2edfc6'} {{(pid=92056) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:00:13.190549 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-5e195eff-1641-481e-a7c0-c40dbd2edfc6 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '6f29f2c0c95249789535d751bc85691a', 'request_id': 'req-5e195eff-1641-481e-a7c0-c40dbd2edfc6'} {{(pid=92056) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:00:13.199929 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-5e195eff-1641-481e-a7c0-c40dbd2edfc6 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '6f29f2c0c95249789535d751bc85691a', 'request_id': 'req-5e195eff-1641-481e-a7c0-c40dbd2edfc6'} {{(pid=92056) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:00:13.235282 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-5e195eff-1641-481e-a7c0-c40dbd2edfc6 admin admin] The token cache is being invalidate because project 6f29f2c0c95249789535d751bc85691a was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=92056) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} May 23 13:00:13.235748 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-5e195eff-1641-481e-a7c0-c40dbd2edfc6 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=92056) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:00:13.236688 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-5e195eff-1641-481e-a7c0-c40dbd2edfc6 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=92056) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:00:13.241802 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 847/1700] 10.222.0.30 () {58 vars in 1232 bytes} [Mon May 23 13:00:12 2022] DELETE /identity/v3/projects/6f29f2c0c95249789535d751bc85691a => generated 0 bytes in 361 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) May 23 13:00:13.246383 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64250f7d-dcb1-4347-8afa-b57c11d2da57 service neutron] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:00:13.257505 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64250f7d-dcb1-4347-8afa-b57c11d2da57 service neutron] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:00:13.262981 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fe81664-424e-4a57-a313-ca052c669ac6 None None] REQUEST_METHOD: `POST` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:00:13.263959 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fe81664-424e-4a57-a313-ca052c669ac6 None None] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:00:13.264540 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fe81664-424e-4a57-a313-ca052c669ac6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:00:13.287445 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.auth.core [None req-2fe81664-424e-4a57-a313-ca052c669ac6 None None] MFA Rules not processed for user `6d21a56126fd4ccd837c33ccc08ab609`. Rule list: `[]` (Enabled: `True`). {{(pid=92056) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 13:00:13.289039 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.fernet_utils [None req-2fe81664-424e-4a57-a313-ca052c669ac6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92056) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:00:13.364942 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb2f6b39-56a5-4e88-bd15-01da0fc760b1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:00:13.366503 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb2f6b39-56a5-4e88-bd15-01da0fc760b1 admin admin] REQUEST_METHOD: `DELETE` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:00:13.366503 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb2f6b39-56a5-4e88-bd15-01da0fc760b1 admin admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:00:13.366763 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb2f6b39-56a5-4e88-bd15-01da0fc760b1 admin admin] PATH_INFO: `/v3/users/2c4fb8716d944288ab4b19170a8d83f6` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:00:13.368070 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb2f6b39-56a5-4e88-bd15-01da0fc760b1 admin admin] RBAC: Authorizing `identity:delete_user(user_id=2c4fb8716d944288ab4b19170a8d83f6)` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:00:13.370402 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb2f6b39-56a5-4e88-bd15-01da0fc760b1 admin admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:00:13.483867 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 848/1701] 10.222.0.30 () {58 vars in 1009 bytes} [Mon May 23 13:00:13 2022] POST /identity/v3/auth/tokens => generated 3185 bytes in 222 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) May 23 13:00:13.701020 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:DELETE=14,keystone:SELECT=85,keystone:INSERT=1 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 13:00:13.717853 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [None req-eb2f6b39-56a5-4e88-bd15-01da0fc760b1 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '2c4fb8716d944288ab4b19170a8d83f6'} {{(pid=92057) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:00:13.811013 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [None req-eb2f6b39-56a5-4e88-bd15-01da0fc760b1 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '2c4fb8716d944288ab4b19170a8d83f6'} {{(pid=92057) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:00:13.818513 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [None req-eb2f6b39-56a5-4e88-bd15-01da0fc760b1 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '2c4fb8716d944288ab4b19170a8d83f6'} {{(pid=92057) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:00:13.819198 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [None req-eb2f6b39-56a5-4e88-bd15-01da0fc760b1 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '2c4fb8716d944288ab4b19170a8d83f6'} {{(pid=92057) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:00:13.832134 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [None req-eb2f6b39-56a5-4e88-bd15-01da0fc760b1 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '2c4fb8716d944288ab4b19170a8d83f6'} {{(pid=92057) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:00:13.838687 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 854/1702] 10.222.0.30 () {58 vars in 1223 bytes} [Mon May 23 13:00:13 2022] DELETE /identity/v3/users/2c4fb8716d944288ab4b19170a8d83f6 => generated 0 bytes in 593 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) May 23 13:00:13.943277 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2fe81664-424e-4a57-a313-ca052c669ac6 None None] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:00:13.945323 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.fernet_utils [None req-2fe81664-424e-4a57-a313-ca052c669ac6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92056) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:00:13.952795 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 13:00:14.062262 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2fe81664-424e-4a57-a313-ca052c669ac6 None None] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:00:14.156840 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0cf0c01e-f53b-4475-a7cb-21b871ac016d service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '138d91edb99f450e846765f9eef00d62', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9c389ae65aaa4542bb5a5c4fd0bacf5b', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'service', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:00:14.157630 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cf0c01e-f53b-4475-a7cb-21b871ac016d service swift] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:00:14.158011 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cf0c01e-f53b-4475-a7cb-21b871ac016d service swift] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:00:14.158441 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cf0c01e-f53b-4475-a7cb-21b871ac016d service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:00:14.163274 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.fernet_utils [None req-0cf0c01e-f53b-4475-a7cb-21b871ac016d service swift] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92056) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:00:14.269645 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0cf0c01e-f53b-4475-a7cb-21b871ac016d service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:00:14.271940 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0cf0c01e-f53b-4475-a7cb-21b871ac016d service swift] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:00:14.279020 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-658c3a21-26ab-4ba9-acc6-4bc0eeb612c7 None None] REQUEST_METHOD: `POST` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:00:14.279550 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-658c3a21-26ab-4ba9-acc6-4bc0eeb612c7 None None] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:00:14.279625 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-658c3a21-26ab-4ba9-acc6-4bc0eeb612c7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:00:14.285521 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 849/1703] 10.222.0.30 () {60 vars in 1384 bytes} [Mon May 23 13:00:13 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 343 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) May 23 13:00:14.320097 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.auth.core [None req-658c3a21-26ab-4ba9-acc6-4bc0eeb612c7 None None] MFA Rules not processed for user `2eb83ca38e79449095f3a2f947bbb06a`. Rule list: `[]` (Enabled: `True`). {{(pid=92057) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 13:00:14.321406 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.fernet_utils [None req-658c3a21-26ab-4ba9-acc6-4bc0eeb612c7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92057) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:00:14.491837 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 855/1704] 10.222.0.49 () {58 vars in 1015 bytes} [Mon May 23 13:00:14 2022] POST /identity/v3/auth/tokens => generated 3111 bytes in 215 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) May 23 13:00:14.968077 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0cf0c01e-f53b-4475-a7cb-21b871ac016d service swift] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:00:14.969598 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.fernet_utils [None req-0cf0c01e-f53b-4475-a7cb-21b871ac016d service swift] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92056) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:00:15.070205 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0cf0c01e-f53b-4475-a7cb-21b871ac016d service swift] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:00:15.081743 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-630b71d1-d9e6-40f2-b9e4-548e35a3c8a4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:00:15.083062 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-630b71d1-d9e6-40f2-b9e4-548e35a3c8a4 admin admin] REQUEST_METHOD: `DELETE` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:00:15.083552 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-630b71d1-d9e6-40f2-b9e4-548e35a3c8a4 admin admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:00:15.084060 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-630b71d1-d9e6-40f2-b9e4-548e35a3c8a4 admin admin] PATH_INFO: `/v3/projects/9a92e19888894a078cef943afa1841dc` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:00:15.086833 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-630b71d1-d9e6-40f2-b9e4-548e35a3c8a4 admin admin] RBAC: Authorizing `identity:delete_project(project_id=9a92e19888894a078cef943afa1841dc)` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:00:15.096579 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-630b71d1-d9e6-40f2-b9e4-548e35a3c8a4 admin admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:00:15.158211 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=100 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 13:00:15.319174 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-630b71d1-d9e6-40f2-b9e4-548e35a3c8a4 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '9a92e19888894a078cef943afa1841dc', 'request_id': 'req-630b71d1-d9e6-40f2-b9e4-548e35a3c8a4'} {{(pid=92056) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:00:15.440377 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-630b71d1-d9e6-40f2-b9e4-548e35a3c8a4 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '9a92e19888894a078cef943afa1841dc', 'request_id': 'req-630b71d1-d9e6-40f2-b9e4-548e35a3c8a4'} {{(pid=92056) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:00:15.449145 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-630b71d1-d9e6-40f2-b9e4-548e35a3c8a4 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '9a92e19888894a078cef943afa1841dc', 'request_id': 'req-630b71d1-d9e6-40f2-b9e4-548e35a3c8a4'} {{(pid=92056) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:00:15.538044 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-630b71d1-d9e6-40f2-b9e4-548e35a3c8a4 admin admin] The token cache is being invalidate because project 9a92e19888894a078cef943afa1841dc was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=92056) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} May 23 13:00:15.538044 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-630b71d1-d9e6-40f2-b9e4-548e35a3c8a4 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=92056) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:00:15.538044 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-630b71d1-d9e6-40f2-b9e4-548e35a3c8a4 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=92056) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:00:15.538044 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 850/1705] 10.222.0.30 () {58 vars in 1232 bytes} [Mon May 23 13:00:14 2022] DELETE /identity/v3/projects/9a92e19888894a078cef943afa1841dc => generated 0 bytes in 555 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) May 23 13:00:15.544355 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e35ac40-a3d7-4221-aa80-2ea34bd41350 None None] REQUEST_METHOD: `POST` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:00:15.545112 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e35ac40-a3d7-4221-aa80-2ea34bd41350 None None] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:00:15.545112 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e35ac40-a3d7-4221-aa80-2ea34bd41350 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:00:15.580367 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.auth.core [None req-0e35ac40-a3d7-4221-aa80-2ea34bd41350 None None] MFA Rules not processed for user `0f7119f8fa5c4445977671140ded91a4`. Rule list: `[]` (Enabled: `True`). {{(pid=92057) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 13:00:15.581709 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.fernet_utils [None req-0e35ac40-a3d7-4221-aa80-2ea34bd41350 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92057) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:00:15.785361 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 856/1706] 10.222.0.30 () {58 vars in 957 bytes} [Mon May 23 13:00:15 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 136 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) May 23 13:00:15.804199 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d54a07c7-9bd8-4de9-887c-02d822324186 None None] REQUEST_METHOD: `POST` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:00:15.804485 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d54a07c7-9bd8-4de9-887c-02d822324186 None None] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:00:15.805149 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d54a07c7-9bd8-4de9-887c-02d822324186 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:00:15.831087 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.auth.core [None req-d54a07c7-9bd8-4de9-887c-02d822324186 None None] MFA Rules not processed for user `0f7119f8fa5c4445977671140ded91a4`. Rule list: `[]` (Enabled: `True`). {{(pid=92056) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 13:00:15.831087 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.fernet_utils [None req-d54a07c7-9bd8-4de9-887c-02d822324186 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92056) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:00:15.846725 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e35ac40-a3d7-4221-aa80-2ea34bd41350 None None] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:00:15.847549 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 851/1707] 10.222.0.30 () {58 vars in 957 bytes} [Mon May 23 13:00:15 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 44 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) May 23 13:00:15.866426 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e35ac40-a3d7-4221-aa80-2ea34bd41350 None None] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:00:15.874361 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a04f0aa-30df-41cf-8783-f0472a87ae60 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:00:15.875334 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a04f0aa-30df-41cf-8783-f0472a87ae60 admin admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:00:15.875623 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a04f0aa-30df-41cf-8783-f0472a87ae60 admin admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:00:15.875904 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a04f0aa-30df-41cf-8783-f0472a87ae60 admin admin] PATH_INFO: `/v3/domains` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:00:15.877129 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a04f0aa-30df-41cf-8783-f0472a87ae60 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=92057) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} May 23 13:00:15.877506 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a04f0aa-30df-41cf-8783-f0472a87ae60 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:00:15.879216 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a04f0aa-30df-41cf-8783-f0472a87ae60 admin admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:00:15.917904 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 857/1708] 10.222.0.30 () {58 vars in 1152 bytes} [Mon May 23 13:00:15 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 72 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) May 23 13:00:15.919000 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d54a07c7-9bd8-4de9-887c-02d822324186 None None] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:00:15.930707 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d54a07c7-9bd8-4de9-887c-02d822324186 None None] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:00:15.936879 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd84d65f-219c-4f21-b1ef-1bfd4b606cd7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:00:15.937585 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd84d65f-219c-4f21-b1ef-1bfd4b606cd7 admin admin] REQUEST_METHOD: `POST` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:00:15.937907 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd84d65f-219c-4f21-b1ef-1bfd4b606cd7 admin admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:00:15.938185 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd84d65f-219c-4f21-b1ef-1bfd4b606cd7 admin admin] PATH_INFO: `/v3/projects` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:00:15.938949 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd84d65f-219c-4f21-b1ef-1bfd4b606cd7 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:00:15.940868 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd84d65f-219c-4f21-b1ef-1bfd4b606cd7 admin admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:00:16.148588 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 852/1709] 10.222.0.30 () {60 vars in 1152 bytes} [Mon May 23 13:00:15 2022] POST /identity/v3/projects => generated 401 bytes in 230 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) May 23 13:00:16.155804 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a04f0aa-30df-41cf-8783-f0472a87ae60 admin admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:00:16.188461 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a04f0aa-30df-41cf-8783-f0472a87ae60 admin admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:00:16.264030 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=95,keystone:DELETE=4,keystone:INSERT=1 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 13:00:16.273870 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a62b0a6-3954-4d28-b9c8-e185796f3cc2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:00:16.274610 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a62b0a6-3954-4d28-b9c8-e185796f3cc2 admin admin] REQUEST_METHOD: `POST` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:00:16.274901 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a62b0a6-3954-4d28-b9c8-e185796f3cc2 admin admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:00:16.284081 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a62b0a6-3954-4d28-b9c8-e185796f3cc2 admin admin] PATH_INFO: `/v3/users` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:00:16.284947 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a62b0a6-3954-4d28-b9c8-e185796f3cc2 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:00:16.287137 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a62b0a6-3954-4d28-b9c8-e185796f3cc2 admin admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:00:16.397793 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 858/1710] 10.222.0.30 () {60 vars in 1143 bytes} [Mon May 23 13:00:16 2022] POST /identity/v3/users => generated 301 bytes in 239 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) May 23 13:00:16.402252 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd84d65f-219c-4f21-b1ef-1bfd4b606cd7 admin admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:00:16.445423 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd84d65f-219c-4f21-b1ef-1bfd4b606cd7 admin admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:00:16.454096 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81d6c412-eb6a-465e-8632-9876aebfc574 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:00:16.455179 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81d6c412-eb6a-465e-8632-9876aebfc574 admin admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:00:16.455647 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81d6c412-eb6a-465e-8632-9876aebfc574 admin admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:00:16.456107 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81d6c412-eb6a-465e-8632-9876aebfc574 admin admin] PATH_INFO: `/v3/roles` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:00:16.457555 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81d6c412-eb6a-465e-8632-9876aebfc574 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:00:16.460478 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81d6c412-eb6a-465e-8632-9876aebfc574 admin admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:00:16.482244 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 853/1711] 10.222.0.30 () {58 vars in 1121 bytes} [Mon May 23 13:00:16 2022] GET /identity/v3/roles => generated 1420 bytes in 82 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) May 23 13:00:16.486485 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a62b0a6-3954-4d28-b9c8-e185796f3cc2 admin admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:00:16.502277 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a62b0a6-3954-4d28-b9c8-e185796f3cc2 admin admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:00:16.511118 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df214e38-90ed-48c9-ac23-c98279cfc30f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:00:16.511809 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df214e38-90ed-48c9-ac23-c98279cfc30f admin admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:00:16.512098 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df214e38-90ed-48c9-ac23-c98279cfc30f admin admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:00:16.512374 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df214e38-90ed-48c9-ac23-c98279cfc30f admin admin] PATH_INFO: `/v3/roles` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:00:16.513361 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df214e38-90ed-48c9-ac23-c98279cfc30f admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:00:16.515022 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df214e38-90ed-48c9-ac23-c98279cfc30f admin admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:00:16.526697 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 859/1712] 10.222.0.30 () {58 vars in 1121 bytes} [Mon May 23 13:00:16 2022] GET /identity/v3/roles => generated 1420 bytes in 41 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) May 23 13:00:16.530801 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81d6c412-eb6a-465e-8632-9876aebfc574 admin admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:00:16.540610 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81d6c412-eb6a-465e-8632-9876aebfc574 admin admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:00:16.545331 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-065c3f94-99af-49ed-8924-d3b2771a7fd1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:00:16.546243 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-065c3f94-99af-49ed-8924-d3b2771a7fd1 admin admin] REQUEST_METHOD: `PUT` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:00:16.546704 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-065c3f94-99af-49ed-8924-d3b2771a7fd1 admin admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:00:16.546985 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-065c3f94-99af-49ed-8924-d3b2771a7fd1 admin admin] PATH_INFO: `/v3/projects/ac05364258c444ae9ce20b6d509f7ed0/users/55d0be2e77e240a6b6195ffcb9f6d42b/roles/80fb22507c9a4ff09d44d8f12f6fff1a` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:00:16.562233 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-065c3f94-99af-49ed-8924-d3b2771a7fd1 admin admin] RBAC: Authorizing `identity:create_grant(project_id=ac05364258c444ae9ce20b6d509f7ed0, user_id=55d0be2e77e240a6b6195ffcb9f6d42b, role_id=80fb22507c9a4ff09d44d8f12f6fff1a)` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:00:16.567778 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-065c3f94-99af-49ed-8924-d3b2771a7fd1 admin admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:00:16.636909 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 854/1713] 10.222.0.30 () {60 vars in 1482 bytes} [Mon May 23 13:00:16 2022] PUT /identity/v3/projects/ac05364258c444ae9ce20b6d509f7ed0/users/55d0be2e77e240a6b6195ffcb9f6d42b/roles/80fb22507c9a4ff09d44d8f12f6fff1a => generated 0 bytes in 107 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) May 23 13:00:16.647636 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25d3559b-79c7-43c2-964d-2d3fd6d13a03 None None] REQUEST_METHOD: `POST` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:00:16.648188 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25d3559b-79c7-43c2-964d-2d3fd6d13a03 None None] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:00:16.648188 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25d3559b-79c7-43c2-964d-2d3fd6d13a03 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:00:16.675247 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.auth.core [None req-25d3559b-79c7-43c2-964d-2d3fd6d13a03 None None] MFA Rules not processed for user `55d0be2e77e240a6b6195ffcb9f6d42b`. Rule list: `[]` (Enabled: `True`). {{(pid=92057) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 13:00:16.676442 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.fernet_utils [None req-25d3559b-79c7-43c2-964d-2d3fd6d13a03 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92057) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:00:16.742461 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 860/1714] 10.222.0.30 () {58 vars in 957 bytes} [Mon May 23 13:00:16 2022] POST /identity/v3/auth/tokens => generated 3268 bytes in 96 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) May 23 13:00:16.747148 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-065c3f94-99af-49ed-8924-d3b2771a7fd1 admin admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:00:16.812086 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-065c3f94-99af-49ed-8924-d3b2771a7fd1 admin admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:00:16.816739 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e58109cd-a1cc-41cb-b355-0366703e7b33 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:00:16.817410 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e58109cd-a1cc-41cb-b355-0366703e7b33 admin admin] REQUEST_METHOD: `POST` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:00:16.817696 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e58109cd-a1cc-41cb-b355-0366703e7b33 admin admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:00:16.818075 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e58109cd-a1cc-41cb-b355-0366703e7b33 admin admin] PATH_INFO: `/v3/projects` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:00:16.818773 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e58109cd-a1cc-41cb-b355-0366703e7b33 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:00:16.820609 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e58109cd-a1cc-41cb-b355-0366703e7b33 admin admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:00:16.939051 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=89,keystone:DELETE=7,keystone:INSERT=4 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 13:00:16.947251 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 855/1715] 10.222.0.30 () {60 vars in 1152 bytes} [Mon May 23 13:00:16 2022] POST /identity/v3/projects => generated 399 bytes in 201 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) May 23 13:00:16.952040 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25d3559b-79c7-43c2-964d-2d3fd6d13a03 None None] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:00:16.963893 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25d3559b-79c7-43c2-964d-2d3fd6d13a03 None None] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:00:17.046596 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9837507-9fa9-4bac-a487-350b36b4a1f1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:00:17.048491 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9837507-9fa9-4bac-a487-350b36b4a1f1 admin admin] REQUEST_METHOD: `POST` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:00:17.048491 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9837507-9fa9-4bac-a487-350b36b4a1f1 admin admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:00:17.048491 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9837507-9fa9-4bac-a487-350b36b4a1f1 admin admin] PATH_INFO: `/v3/users` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:00:17.048491 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9837507-9fa9-4bac-a487-350b36b4a1f1 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:00:17.050462 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9837507-9fa9-4bac-a487-350b36b4a1f1 admin admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:00:17.123314 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 861/1716] 10.222.0.30 () {60 vars in 1143 bytes} [Mon May 23 13:00:16 2022] POST /identity/v3/users => generated 306 bytes in 172 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) May 23 13:00:17.127411 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e58109cd-a1cc-41cb-b355-0366703e7b33 admin admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:00:17.141564 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e58109cd-a1cc-41cb-b355-0366703e7b33 admin admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:00:17.149393 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5eba14de-7745-4b80-ad3a-43d7afd68bec admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:00:17.150129 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5eba14de-7745-4b80-ad3a-43d7afd68bec admin admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:00:17.150800 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5eba14de-7745-4b80-ad3a-43d7afd68bec admin admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:00:17.151700 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5eba14de-7745-4b80-ad3a-43d7afd68bec admin admin] PATH_INFO: `/v3/roles` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:00:17.152443 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5eba14de-7745-4b80-ad3a-43d7afd68bec admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:00:17.155677 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5eba14de-7745-4b80-ad3a-43d7afd68bec admin admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:00:17.171060 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 856/1717] 10.222.0.30 () {58 vars in 1121 bytes} [Mon May 23 13:00:17 2022] GET /identity/v3/roles => generated 1420 bytes in 41 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) May 23 13:00:17.172256 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9837507-9fa9-4bac-a487-350b36b4a1f1 admin admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:00:17.183578 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9837507-9fa9-4bac-a487-350b36b4a1f1 admin admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:00:17.188585 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-512a6a75-68a1-4359-b9f2-b141f3a32019 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:00:17.189209 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-512a6a75-68a1-4359-b9f2-b141f3a32019 admin admin] REQUEST_METHOD: `PUT` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:00:17.189607 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-512a6a75-68a1-4359-b9f2-b141f3a32019 admin admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:00:17.189994 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-512a6a75-68a1-4359-b9f2-b141f3a32019 admin admin] PATH_INFO: `/v3/projects/7c8fe87d0aec4f5cacce879cc336d002/users/184466528299445ea47589748666f040/roles/bc3c3c9d654940cbaa5177a6be23d1bc` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:00:17.213082 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-512a6a75-68a1-4359-b9f2-b141f3a32019 admin admin] RBAC: Authorizing `identity:create_grant(project_id=7c8fe87d0aec4f5cacce879cc336d002, user_id=184466528299445ea47589748666f040, role_id=bc3c3c9d654940cbaa5177a6be23d1bc)` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:00:17.216645 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-512a6a75-68a1-4359-b9f2-b141f3a32019 admin admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:00:17.275583 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 862/1718] 10.222.0.30 () {60 vars in 1482 bytes} [Mon May 23 13:00:17 2022] PUT /identity/v3/projects/7c8fe87d0aec4f5cacce879cc336d002/users/184466528299445ea47589748666f040/roles/bc3c3c9d654940cbaa5177a6be23d1bc => generated 0 bytes in 104 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) May 23 13:00:17.280637 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5eba14de-7745-4b80-ad3a-43d7afd68bec admin admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:00:17.291122 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5eba14de-7745-4b80-ad3a-43d7afd68bec admin admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:00:17.347671 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2303d75b-7599-4dec-a8d7-c1366ed9a6d0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:00:17.348269 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2303d75b-7599-4dec-a8d7-c1366ed9a6d0 admin admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:00:17.348553 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2303d75b-7599-4dec-a8d7-c1366ed9a6d0 admin admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:00:17.348829 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2303d75b-7599-4dec-a8d7-c1366ed9a6d0 admin admin] PATH_INFO: `/v3/roles` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:00:17.349559 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2303d75b-7599-4dec-a8d7-c1366ed9a6d0 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:00:17.351261 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2303d75b-7599-4dec-a8d7-c1366ed9a6d0 admin admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:00:17.361346 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 857/1719] 10.222.0.30 () {58 vars in 1121 bytes} [Mon May 23 13:00:17 2022] GET /identity/v3/roles => generated 1420 bytes in 82 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) May 23 13:00:17.365419 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-512a6a75-68a1-4359-b9f2-b141f3a32019 admin admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:00:17.375661 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-512a6a75-68a1-4359-b9f2-b141f3a32019 admin admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:00:17.380237 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4bd69e3c-4587-4d54-879b-653cb83949a8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:00:17.381106 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bd69e3c-4587-4d54-879b-653cb83949a8 admin admin] REQUEST_METHOD: `PUT` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:00:17.381389 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bd69e3c-4587-4d54-879b-653cb83949a8 admin admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:00:17.381661 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bd69e3c-4587-4d54-879b-653cb83949a8 admin admin] PATH_INFO: `/v3/projects/7c8fe87d0aec4f5cacce879cc336d002/users/184466528299445ea47589748666f040/roles/bc3c3c9d654940cbaa5177a6be23d1bc` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:00:17.383640 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4bd69e3c-4587-4d54-879b-653cb83949a8 admin admin] RBAC: Authorizing `identity:create_grant(project_id=7c8fe87d0aec4f5cacce879cc336d002, user_id=184466528299445ea47589748666f040, role_id=bc3c3c9d654940cbaa5177a6be23d1bc)` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:00:17.386832 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4bd69e3c-4587-4d54-879b-653cb83949a8 admin admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:00:17.397221 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 863/1720] 10.222.0.30 () {60 vars in 1482 bytes} [Mon May 23 13:00:17 2022] PUT /identity/v3/projects/7c8fe87d0aec4f5cacce879cc336d002/users/184466528299445ea47589748666f040/roles/bc3c3c9d654940cbaa5177a6be23d1bc => generated 0 bytes in 32 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) May 23 13:00:17.408605 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76ffe68d-fa3c-435d-97cf-160906845ce7 None None] REQUEST_METHOD: `POST` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:00:17.409229 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76ffe68d-fa3c-435d-97cf-160906845ce7 None None] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:00:17.409229 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76ffe68d-fa3c-435d-97cf-160906845ce7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:00:17.436671 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.auth.core [None req-76ffe68d-fa3c-435d-97cf-160906845ce7 None None] MFA Rules not processed for user `184466528299445ea47589748666f040`. Rule list: `[]` (Enabled: `True`). {{(pid=92056) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 13:00:17.437922 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.fernet_utils [None req-76ffe68d-fa3c-435d-97cf-160906845ce7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92056) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:00:17.508775 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 858/1721] 10.222.0.30 () {58 vars in 957 bytes} [Mon May 23 13:00:17 2022] POST /identity/v3/auth/tokens => generated 3333 bytes in 101 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) May 23 13:00:17.519436 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4bd69e3c-4587-4d54-879b-653cb83949a8 admin admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:00:17.521166 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.fernet_utils [None req-4bd69e3c-4587-4d54-879b-653cb83949a8 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92057) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:00:17.540674 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=92,keystone:INSERT=8 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 13:00:17.580109 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4bd69e3c-4587-4d54-879b-653cb83949a8 admin admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:00:17.583945 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29f679b7-ad15-4859-8434-e0f173198c06 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1666097f8ed545c6b0e4809c93e4775a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9c389ae65aaa4542bb5a5c4fd0bacf5b', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:00:17.584731 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29f679b7-ad15-4859-8434-e0f173198c06 service cinder] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:00:17.585028 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29f679b7-ad15-4859-8434-e0f173198c06 service cinder] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:00:17.585731 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29f679b7-ad15-4859-8434-e0f173198c06 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:00:17.596957 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-29f679b7-ad15-4859-8434-e0f173198c06 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:00:17.598555 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-29f679b7-ad15-4859-8434-e0f173198c06 service cinder] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:00:17.608680 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 864/1722] 10.222.0.30 () {60 vars in 1365 bytes} [Mon May 23 13:00:17 2022] GET /identity/v3/auth/tokens => generated 3333 bytes in 90 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) May 23 13:00:17.863013 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76ffe68d-fa3c-435d-97cf-160906845ce7 None None] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:00:17.873115 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76ffe68d-fa3c-435d-97cf-160906845ce7 None None] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:00:17.876871 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7731dc0-fba8-4a1e-ac7d-f691f87c6934 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1666097f8ed545c6b0e4809c93e4775a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9c389ae65aaa4542bb5a5c4fd0bacf5b', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:00:17.877509 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7731dc0-fba8-4a1e-ac7d-f691f87c6934 service cinder] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:00:17.877792 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7731dc0-fba8-4a1e-ac7d-f691f87c6934 service cinder] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:00:17.878096 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7731dc0-fba8-4a1e-ac7d-f691f87c6934 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:00:17.934781 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d7731dc0-fba8-4a1e-ac7d-f691f87c6934 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:00:17.936232 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d7731dc0-fba8-4a1e-ac7d-f691f87c6934 service cinder] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:00:17.944965 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 859/1723] 10.222.0.30 () {60 vars in 1365 bytes} [Mon May 23 13:00:17 2022] GET /identity/v3/auth/tokens => generated 3268 bytes in 83 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) May 23 13:00:22.361908 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29f679b7-ad15-4859-8434-e0f173198c06 service cinder] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:00:22.363747 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.fernet_utils [None req-29f679b7-ad15-4859-8434-e0f173198c06 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92057) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:00:22.484818 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29f679b7-ad15-4859-8434-e0f173198c06 service cinder] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:00:22.493827 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7bfb8d2-890c-4ac4-bfbb-319d9e861022 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:00:22.495059 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7bfb8d2-890c-4ac4-bfbb-319d9e861022 admin admin] REQUEST_METHOD: `DELETE` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:00:22.495350 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7bfb8d2-890c-4ac4-bfbb-319d9e861022 admin admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:00:22.495652 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7bfb8d2-890c-4ac4-bfbb-319d9e861022 admin admin] PATH_INFO: `/v3/users/caaf63946cde4042b60389be276025d0` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:00:22.496895 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7bfb8d2-890c-4ac4-bfbb-319d9e861022 admin admin] RBAC: Authorizing `identity:delete_user(user_id=caaf63946cde4042b60389be276025d0)` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:00:22.499361 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7bfb8d2-890c-4ac4-bfbb-319d9e861022 admin admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:00:22.756223 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [None req-e7bfb8d2-890c-4ac4-bfbb-319d9e861022 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'caaf63946cde4042b60389be276025d0'} {{(pid=92057) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:00:22.853188 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [None req-e7bfb8d2-890c-4ac4-bfbb-319d9e861022 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'caaf63946cde4042b60389be276025d0'} {{(pid=92057) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:00:22.865479 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [None req-e7bfb8d2-890c-4ac4-bfbb-319d9e861022 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'caaf63946cde4042b60389be276025d0'} {{(pid=92057) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:00:22.866529 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [None req-e7bfb8d2-890c-4ac4-bfbb-319d9e861022 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'caaf63946cde4042b60389be276025d0'} {{(pid=92057) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:00:22.899744 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [None req-e7bfb8d2-890c-4ac4-bfbb-319d9e861022 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'caaf63946cde4042b60389be276025d0'} {{(pid=92057) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:00:22.920929 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 865/1724] 10.222.0.30 () {58 vars in 1223 bytes} [Mon May 23 13:00:22 2022] DELETE /identity/v3/users/caaf63946cde4042b60389be276025d0 => generated 0 bytes in 558 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) May 23 13:00:22.925022 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fae4e8b1-5961-4bc5-9e3b-588c259bdfe0 None None] REQUEST_METHOD: `POST` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:00:22.925508 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fae4e8b1-5961-4bc5-9e3b-588c259bdfe0 None None] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:00:22.925593 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fae4e8b1-5961-4bc5-9e3b-588c259bdfe0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:00:22.958161 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=100 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 13:00:22.976500 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.auth.core [None req-fae4e8b1-5961-4bc5-9e3b-588c259bdfe0 None None] MFA Rules not processed for user `0f7119f8fa5c4445977671140ded91a4`. Rule list: `[]` (Enabled: `True`). {{(pid=92056) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 13:00:22.977759 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.fernet_utils [None req-fae4e8b1-5961-4bc5-9e3b-588c259bdfe0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92056) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:00:23.092115 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 860/1725] 10.222.0.30 () {58 vars in 957 bytes} [Mon May 23 13:00:22 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 169 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) May 23 13:00:23.103590 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7bfb8d2-890c-4ac4-bfbb-319d9e861022 admin admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:00:23.107070 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.fernet_utils [None req-e7bfb8d2-890c-4ac4-bfbb-319d9e861022 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92057) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:00:23.176494 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 13:00:23.217435 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7bfb8d2-890c-4ac4-bfbb-319d9e861022 admin admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:00:23.225948 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e86668b-628c-4fc6-8160-227f35149c66 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2eb83ca38e79449095f3a2f947bbb06a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9c389ae65aaa4542bb5a5c4fd0bacf5b', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:00:23.226842 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e86668b-628c-4fc6-8160-227f35149c66 service neutron] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:00:23.226957 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e86668b-628c-4fc6-8160-227f35149c66 service neutron] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:00:23.227176 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e86668b-628c-4fc6-8160-227f35149c66 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:00:23.251921 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e86668b-628c-4fc6-8160-227f35149c66 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:00:23.253383 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e86668b-628c-4fc6-8160-227f35149c66 service neutron] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:00:23.276181 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 866/1726] 10.222.0.30 () {60 vars in 1365 bytes} [Mon May 23 13:00:23 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 172 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) May 23 13:00:23.332843 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fae4e8b1-5961-4bc5-9e3b-588c259bdfe0 None None] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:00:23.339146 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e86668b-628c-4fc6-8160-227f35149c66 service neutron] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:00:23.340551 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.fernet_utils [None req-5e86668b-628c-4fc6-8160-227f35149c66 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92057) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:00:23.344210 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fae4e8b1-5961-4bc5-9e3b-588c259bdfe0 None None] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:00:23.348080 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5afd753-6da9-4e57-a762-180c4a74824a service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2eb83ca38e79449095f3a2f947bbb06a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9c389ae65aaa4542bb5a5c4fd0bacf5b', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:00:23.348807 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5afd753-6da9-4e57-a762-180c4a74824a service neutron] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:00:23.349166 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5afd753-6da9-4e57-a762-180c4a74824a service neutron] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:00:23.349513 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5afd753-6da9-4e57-a762-180c4a74824a service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:00:23.352091 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.fernet_utils [None req-a5afd753-6da9-4e57-a762-180c4a74824a service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92056) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:00:23.359638 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e86668b-628c-4fc6-8160-227f35149c66 service neutron] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:00:23.364731 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e56e79f-7bf1-44e7-9412-9409452f95be service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2eb83ca38e79449095f3a2f947bbb06a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9c389ae65aaa4542bb5a5c4fd0bacf5b', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:00:23.367838 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e56e79f-7bf1-44e7-9412-9409452f95be service neutron] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:00:23.368464 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e56e79f-7bf1-44e7-9412-9409452f95be service neutron] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:00:23.369732 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e56e79f-7bf1-44e7-9412-9409452f95be service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:00:23.370911 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a5afd753-6da9-4e57-a762-180c4a74824a service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:00:23.373907 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a5afd753-6da9-4e57-a762-180c4a74824a service neutron] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:00:23.384272 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e56e79f-7bf1-44e7-9412-9409452f95be service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:00:23.387398 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 861/1727] 10.222.0.30 () {60 vars in 1365 bytes} [Mon May 23 13:00:23 2022] GET /identity/v3/auth/tokens => generated 3111 bytes in 55 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) May 23 13:00:23.389027 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e56e79f-7bf1-44e7-9412-9409452f95be service neutron] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:00:23.390393 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5afd753-6da9-4e57-a762-180c4a74824a service neutron] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:00:23.410271 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 867/1728] 10.222.0.30 () {60 vars in 1365 bytes} [Mon May 23 13:00:23 2022] GET /identity/v3/auth/tokens => generated 3111 bytes in 72 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) May 23 13:00:23.419499 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5afd753-6da9-4e57-a762-180c4a74824a service neutron] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:00:23.434378 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-817ec46f-7b58-4cad-9c3a-267519fcf3c1 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2eb83ca38e79449095f3a2f947bbb06a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9c389ae65aaa4542bb5a5c4fd0bacf5b', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:00:23.435092 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-817ec46f-7b58-4cad-9c3a-267519fcf3c1 service neutron] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:00:23.439767 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e56e79f-7bf1-44e7-9412-9409452f95be service neutron] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:00:23.440800 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-817ec46f-7b58-4cad-9c3a-267519fcf3c1 service neutron] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:00:23.441215 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-817ec46f-7b58-4cad-9c3a-267519fcf3c1 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:00:23.453282 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-817ec46f-7b58-4cad-9c3a-267519fcf3c1 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:00:23.455063 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-817ec46f-7b58-4cad-9c3a-267519fcf3c1 service neutron] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:00:23.465817 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 862/1729] 10.222.0.30 () {60 vars in 1365 bytes} [Mon May 23 13:00:23 2022] GET /identity/v3/auth/tokens => generated 3111 bytes in 76 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) May 23 13:00:23.466665 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e56e79f-7bf1-44e7-9412-9409452f95be service neutron] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:00:23.486662 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96ec449a-a344-4e54-b157-8e04266a67b8 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2eb83ca38e79449095f3a2f947bbb06a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9c389ae65aaa4542bb5a5c4fd0bacf5b', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:00:23.490557 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96ec449a-a344-4e54-b157-8e04266a67b8 service neutron] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:00:23.490909 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96ec449a-a344-4e54-b157-8e04266a67b8 service neutron] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:00:23.491195 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96ec449a-a344-4e54-b157-8e04266a67b8 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:00:23.514160 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96ec449a-a344-4e54-b157-8e04266a67b8 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:00:23.516838 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96ec449a-a344-4e54-b157-8e04266a67b8 service neutron] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:00:23.540865 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 868/1730] 10.222.0.30 () {60 vars in 1365 bytes} [Mon May 23 13:00:23 2022] GET /identity/v3/auth/tokens => generated 3111 bytes in 102 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) May 23 13:00:24.732424 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-817ec46f-7b58-4cad-9c3a-267519fcf3c1 service neutron] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:00:24.734141 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.fernet_utils [None req-817ec46f-7b58-4cad-9c3a-267519fcf3c1 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92056) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:00:24.756692 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-817ec46f-7b58-4cad-9c3a-267519fcf3c1 service neutron] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:00:24.767935 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a4befc6-d766-4742-a56c-396c43d1a939 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:00:24.770560 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a4befc6-d766-4742-a56c-396c43d1a939 admin admin] REQUEST_METHOD: `DELETE` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:00:24.770560 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a4befc6-d766-4742-a56c-396c43d1a939 admin admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:00:24.770560 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a4befc6-d766-4742-a56c-396c43d1a939 admin admin] PATH_INFO: `/v3/users/b75234dc3b6e489b9522c10397b5b69d` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:00:24.771543 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a4befc6-d766-4742-a56c-396c43d1a939 admin admin] RBAC: Authorizing `identity:delete_user(user_id=b75234dc3b6e489b9522c10397b5b69d)` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:00:24.774073 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a4befc6-d766-4742-a56c-396c43d1a939 admin admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:00:24.999115 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-889b6edc-9455-4248-9692-99ece19d8bbf None None] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:00:24.999612 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-889b6edc-9455-4248-9692-99ece19d8bbf None None] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:00:24.999714 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-889b6edc-9455-4248-9692-99ece19d8bbf None None] PATH_INFO: `/` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:00:25.001548 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 869/1731] 10.222.0.30 () {54 vars in 921 bytes} [Mon May 23 13:00:24 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) May 23 13:00:25.019707 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f5d8e49-a581-4b68-ad9b-d64c0dc97617 None None] REQUEST_METHOD: `POST` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:00:25.020936 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f5d8e49-a581-4b68-ad9b-d64c0dc97617 None None] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:00:25.021752 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f5d8e49-a581-4b68-ad9b-d64c0dc97617 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:00:25.062857 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.auth.core [None req-2f5d8e49-a581-4b68-ad9b-d64c0dc97617 None None] MFA Rules not processed for user `cbe9931d94d44cc7ace0e83a7e5015fc`. Rule list: `[]` (Enabled: `True`). {{(pid=92057) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 13:00:25.065379 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.fernet_utils [None req-2f5d8e49-a581-4b68-ad9b-d64c0dc97617 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92057) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:00:25.072105 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-8a4befc6-d766-4742-a56c-396c43d1a939 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'b75234dc3b6e489b9522c10397b5b69d'} {{(pid=92056) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:00:25.074469 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-8a4befc6-d766-4742-a56c-396c43d1a939 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'b75234dc3b6e489b9522c10397b5b69d'} {{(pid=92056) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:00:25.148089 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-8a4befc6-d766-4742-a56c-396c43d1a939 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'b75234dc3b6e489b9522c10397b5b69d'} {{(pid=92056) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:00:25.173368 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-8a4befc6-d766-4742-a56c-396c43d1a939 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'b75234dc3b6e489b9522c10397b5b69d'} {{(pid=92056) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:00:25.187113 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-8a4befc6-d766-4742-a56c-396c43d1a939 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'b75234dc3b6e489b9522c10397b5b69d'} {{(pid=92056) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:00:25.192568 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 870/1732] 10.222.0.30 () {58 vars in 1018 bytes} [Mon May 23 13:00:25 2022] POST /identity/v3/auth/tokens => generated 3362 bytes in 176 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) May 23 13:00:25.193968 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f5d8e49-a581-4b68-ad9b-d64c0dc97617 None None] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:00:25.195963 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.fernet_utils [None req-2f5d8e49-a581-4b68-ad9b-d64c0dc97617 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92057) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:00:25.199245 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 863/1733] 10.222.0.30 () {58 vars in 1223 bytes} [Mon May 23 13:00:24 2022] DELETE /identity/v3/users/b75234dc3b6e489b9522c10397b5b69d => generated 0 bytes in 464 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) May 23 13:00:25.200922 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cd583de-7ef4-4852-bc81-f637762460eb None None] REQUEST_METHOD: `POST` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:00:25.201407 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cd583de-7ef4-4852-bc81-f637762460eb None None] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:00:25.201697 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cd583de-7ef4-4852-bc81-f637762460eb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:00:25.225073 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f5d8e49-a581-4b68-ad9b-d64c0dc97617 None None] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:00:25.228232 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.auth.core [None req-0cd583de-7ef4-4852-bc81-f637762460eb None None] MFA Rules not processed for user `0f7119f8fa5c4445977671140ded91a4`. Rule list: `[]` (Enabled: `True`). {{(pid=92056) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 13:00:25.229942 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.fernet_utils [None req-0cd583de-7ef4-4852-bc81-f637762460eb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92056) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:00:25.321729 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 13:00:25.344961 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 864/1734] 10.222.0.30 () {58 vars in 957 bytes} [Mon May 23 13:00:25 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 145 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) May 23 13:00:25.346177 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0cd583de-7ef4-4852-bc81-f637762460eb None None] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:00:25.347755 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.fernet_utils [None req-0cd583de-7ef4-4852-bc81-f637762460eb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92056) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:00:25.354543 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6dcf0eaa-7c46-42fb-b93d-e2cb1ef8d3b0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:00:25.355548 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6dcf0eaa-7c46-42fb-b93d-e2cb1ef8d3b0 admin admin] REQUEST_METHOD: `DELETE` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:00:25.355842 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6dcf0eaa-7c46-42fb-b93d-e2cb1ef8d3b0 admin admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:00:25.358599 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6dcf0eaa-7c46-42fb-b93d-e2cb1ef8d3b0 admin admin] PATH_INFO: `/v3/users/19c9bc4d2df346c6a0a53caa18657f48` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:00:25.366731 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6dcf0eaa-7c46-42fb-b93d-e2cb1ef8d3b0 admin admin] RBAC: Authorizing `identity:delete_user(user_id=19c9bc4d2df346c6a0a53caa18657f48)` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:00:25.368751 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6dcf0eaa-7c46-42fb-b93d-e2cb1ef8d3b0 admin admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:00:25.506024 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0cd583de-7ef4-4852-bc81-f637762460eb None None] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:00:25.517391 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18d93001-7da0-4d6e-9d94-c2ecf1e19068 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cbe9931d94d44cc7ace0e83a7e5015fc', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9c389ae65aaa4542bb5a5c4fd0bacf5b', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'member', 'reader', 'service', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:00:25.518607 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18d93001-7da0-4d6e-9d94-c2ecf1e19068 service nova] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:00:25.518607 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18d93001-7da0-4d6e-9d94-c2ecf1e19068 service nova] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:00:25.518815 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18d93001-7da0-4d6e-9d94-c2ecf1e19068 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:00:25.551442 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18d93001-7da0-4d6e-9d94-c2ecf1e19068 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:00:25.553636 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18d93001-7da0-4d6e-9d94-c2ecf1e19068 service nova] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:00:25.574544 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 865/1735] 10.222.0.30 () {60 vars in 1365 bytes} [Mon May 23 13:00:25 2022] GET /identity/v3/auth/tokens => generated 3362 bytes in 228 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) May 23 13:00:25.575500 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18d93001-7da0-4d6e-9d94-c2ecf1e19068 service nova] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:00:25.577451 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.fernet_utils [None req-18d93001-7da0-4d6e-9d94-c2ecf1e19068 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92056) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:00:25.666811 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [None req-6dcf0eaa-7c46-42fb-b93d-e2cb1ef8d3b0 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '19c9bc4d2df346c6a0a53caa18657f48'} {{(pid=92057) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:00:25.668808 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=93,keystone:DELETE=7 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 13:00:25.757073 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [None req-6dcf0eaa-7c46-42fb-b93d-e2cb1ef8d3b0 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '19c9bc4d2df346c6a0a53caa18657f48'} {{(pid=92057) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:00:25.772129 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [None req-6dcf0eaa-7c46-42fb-b93d-e2cb1ef8d3b0 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '19c9bc4d2df346c6a0a53caa18657f48'} {{(pid=92057) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:00:25.773222 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [None req-6dcf0eaa-7c46-42fb-b93d-e2cb1ef8d3b0 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '19c9bc4d2df346c6a0a53caa18657f48'} {{(pid=92057) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:00:25.795154 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [None req-6dcf0eaa-7c46-42fb-b93d-e2cb1ef8d3b0 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '19c9bc4d2df346c6a0a53caa18657f48'} {{(pid=92057) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:00:25.810605 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 871/1736] 10.222.0.30 () {58 vars in 1223 bytes} [Mon May 23 13:00:25 2022] DELETE /identity/v3/users/19c9bc4d2df346c6a0a53caa18657f48 => generated 0 bytes in 617 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) May 23 13:00:25.828415 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18d93001-7da0-4d6e-9d94-c2ecf1e19068 service nova] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:00:25.834243 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f84780ee-cfb4-45ae-9244-7b4d11c857a4 None None] REQUEST_METHOD: `POST` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:00:25.834674 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f84780ee-cfb4-45ae-9244-7b4d11c857a4 None None] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:00:25.835156 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f84780ee-cfb4-45ae-9244-7b4d11c857a4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:00:25.896526 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.auth.core [None req-f84780ee-cfb4-45ae-9244-7b4d11c857a4 None None] MFA Rules not processed for user `0f7119f8fa5c4445977671140ded91a4`. Rule list: `[]` (Enabled: `True`). {{(pid=92057) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 13:00:25.903776 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.fernet_utils [None req-f84780ee-cfb4-45ae-9244-7b4d11c857a4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92057) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:00:25.933870 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7fea53f-f221-409f-b6d5-c24ac59fbc1f service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2eb83ca38e79449095f3a2f947bbb06a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9c389ae65aaa4542bb5a5c4fd0bacf5b', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:00:25.935519 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7fea53f-f221-409f-b6d5-c24ac59fbc1f service neutron] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:00:25.936211 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7fea53f-f221-409f-b6d5-c24ac59fbc1f service neutron] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:00:25.936832 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7fea53f-f221-409f-b6d5-c24ac59fbc1f service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:00:25.939682 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.fernet_utils [None req-a7fea53f-f221-409f-b6d5-c24ac59fbc1f service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92056) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:00:26.037043 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 872/1737] 10.222.0.30 () {58 vars in 957 bytes} [Mon May 23 13:00:25 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 220 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) May 23 13:00:26.059530 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=100 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 13:00:26.096504 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7fea53f-f221-409f-b6d5-c24ac59fbc1f service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:00:26.098138 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7fea53f-f221-409f-b6d5-c24ac59fbc1f service neutron] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:00:26.122580 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 866/1738] 10.222.0.30 () {60 vars in 1365 bytes} [Mon May 23 13:00:25 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 547 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) May 23 13:00:26.200577 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f84780ee-cfb4-45ae-9244-7b4d11c857a4 None None] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:00:26.203010 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.fernet_utils [None req-f84780ee-cfb4-45ae-9244-7b4d11c857a4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92057) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:00:26.237363 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f84780ee-cfb4-45ae-9244-7b4d11c857a4 None None] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:00:26.244557 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7fea53f-f221-409f-b6d5-c24ac59fbc1f service neutron] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:00:26.247198 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.fernet_utils [None req-a7fea53f-f221-409f-b6d5-c24ac59fbc1f service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92056) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:00:26.252196 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb3cbadc-d01a-4284-926e-d7212c39ab3b service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2eb83ca38e79449095f3a2f947bbb06a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9c389ae65aaa4542bb5a5c4fd0bacf5b', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:00:26.256854 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb3cbadc-d01a-4284-926e-d7212c39ab3b service neutron] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:00:26.256854 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb3cbadc-d01a-4284-926e-d7212c39ab3b service neutron] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:00:26.256854 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb3cbadc-d01a-4284-926e-d7212c39ab3b service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:00:26.279045 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7fea53f-f221-409f-b6d5-c24ac59fbc1f service neutron] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:00:26.286096 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7849cb51-1329-4eec-8ba4-836bd4d56eec admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:00:26.287255 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7849cb51-1329-4eec-8ba4-836bd4d56eec admin admin] REQUEST_METHOD: `DELETE` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:00:26.287596 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7849cb51-1329-4eec-8ba4-836bd4d56eec admin admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:00:26.287872 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7849cb51-1329-4eec-8ba4-836bd4d56eec admin admin] PATH_INFO: `/v3/users/55d0be2e77e240a6b6195ffcb9f6d42b` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:00:26.289547 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7849cb51-1329-4eec-8ba4-836bd4d56eec admin admin] RBAC: Authorizing `identity:delete_user(user_id=55d0be2e77e240a6b6195ffcb9f6d42b)` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:00:26.292372 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7849cb51-1329-4eec-8ba4-836bd4d56eec admin admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:00:26.293898 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb3cbadc-d01a-4284-926e-d7212c39ab3b service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:00:26.304723 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb3cbadc-d01a-4284-926e-d7212c39ab3b service neutron] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:00:26.318973 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 873/1739] 10.222.0.30 () {60 vars in 1365 bytes} [Mon May 23 13:00:26 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 120 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) May 23 13:00:26.571516 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-7849cb51-1329-4eec-8ba4-836bd4d56eec admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '55d0be2e77e240a6b6195ffcb9f6d42b'} {{(pid=92056) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:00:26.572447 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-7849cb51-1329-4eec-8ba4-836bd4d56eec admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '55d0be2e77e240a6b6195ffcb9f6d42b'} {{(pid=92056) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:00:26.678344 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-7849cb51-1329-4eec-8ba4-836bd4d56eec admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '55d0be2e77e240a6b6195ffcb9f6d42b'} {{(pid=92056) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:00:26.721132 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-7849cb51-1329-4eec-8ba4-836bd4d56eec admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '55d0be2e77e240a6b6195ffcb9f6d42b'} {{(pid=92056) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:00:26.737292 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb3cbadc-d01a-4284-926e-d7212c39ab3b service neutron] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:00:26.741927 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.fernet_utils [None req-eb3cbadc-d01a-4284-926e-d7212c39ab3b service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92057) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:00:26.747240 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-7849cb51-1329-4eec-8ba4-836bd4d56eec admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '55d0be2e77e240a6b6195ffcb9f6d42b'} {{(pid=92056) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:00:26.762731 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 867/1740] 10.222.0.30 () {58 vars in 1223 bytes} [Mon May 23 13:00:26 2022] DELETE /identity/v3/users/55d0be2e77e240a6b6195ffcb9f6d42b => generated 0 bytes in 516 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) May 23 13:00:26.764780 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a4385fc-41fe-47e0-ade4-d8891bc7eadf None None] REQUEST_METHOD: `POST` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:00:26.765110 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a4385fc-41fe-47e0-ade4-d8891bc7eadf None None] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:00:26.765438 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a4385fc-41fe-47e0-ade4-d8891bc7eadf None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:00:26.769624 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb3cbadc-d01a-4284-926e-d7212c39ab3b service neutron] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:00:26.808318 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.auth.core [None req-1a4385fc-41fe-47e0-ade4-d8891bc7eadf None None] MFA Rules not processed for user `0f7119f8fa5c4445977671140ded91a4`. Rule list: `[]` (Enabled: `True`). {{(pid=92056) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 13:00:26.809483 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.fernet_utils [None req-1a4385fc-41fe-47e0-ade4-d8891bc7eadf None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92056) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:00:26.992847 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab29cfea-5040-4126-b903-deff694e0cc9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:00:26.994945 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab29cfea-5040-4126-b903-deff694e0cc9 admin admin] REQUEST_METHOD: `DELETE` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:00:26.994945 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab29cfea-5040-4126-b903-deff694e0cc9 admin admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:00:27.002099 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 868/1741] 10.222.0.30 () {58 vars in 957 bytes} [Mon May 23 13:00:26 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 238 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) May 23 13:00:27.007007 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab29cfea-5040-4126-b903-deff694e0cc9 admin admin] PATH_INFO: `/v3/projects/e6ff065adc71404184a1fe9ccdf97bee` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:00:27.008730 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab29cfea-5040-4126-b903-deff694e0cc9 admin admin] RBAC: Authorizing `identity:delete_project(project_id=e6ff065adc71404184a1fe9ccdf97bee)` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:00:27.011190 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab29cfea-5040-4126-b903-deff694e0cc9 admin admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:00:27.064084 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a4385fc-41fe-47e0-ade4-d8891bc7eadf None None] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:00:27.065848 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.fernet_utils [None req-1a4385fc-41fe-47e0-ade4-d8891bc7eadf None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92056) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:00:27.175799 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a4385fc-41fe-47e0-ade4-d8891bc7eadf None None] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:00:27.183796 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f3dbf45-555c-4608-8f55-c1fe5fccd5d0 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2eb83ca38e79449095f3a2f947bbb06a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9c389ae65aaa4542bb5a5c4fd0bacf5b', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:00:27.194216 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f3dbf45-555c-4608-8f55-c1fe5fccd5d0 service neutron] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:00:27.214042 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f3dbf45-555c-4608-8f55-c1fe5fccd5d0 service neutron] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:00:27.231379 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f3dbf45-555c-4608-8f55-c1fe5fccd5d0 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:00:27.261666 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [None req-ab29cfea-5040-4126-b903-deff694e0cc9 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'e6ff065adc71404184a1fe9ccdf97bee', 'request_id': 'req-ab29cfea-5040-4126-b903-deff694e0cc9'} {{(pid=92057) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:00:27.271780 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:INSERT=1,keystone:DELETE=8,keystone:SELECT=91 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 13:00:27.279046 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f3dbf45-555c-4608-8f55-c1fe5fccd5d0 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:00:27.280167 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [None req-ab29cfea-5040-4126-b903-deff694e0cc9 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'e6ff065adc71404184a1fe9ccdf97bee', 'request_id': 'req-ab29cfea-5040-4126-b903-deff694e0cc9'} {{(pid=92057) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:00:27.285547 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f3dbf45-555c-4608-8f55-c1fe5fccd5d0 service neutron] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:00:27.299006 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 13:00:27.336530 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 869/1742] 10.222.0.30 () {60 vars in 1365 bytes} [Mon May 23 13:00:27 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 268 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) May 23 13:00:27.391574 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [None req-ab29cfea-5040-4126-b903-deff694e0cc9 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'e6ff065adc71404184a1fe9ccdf97bee', 'request_id': 'req-ab29cfea-5040-4126-b903-deff694e0cc9'} {{(pid=92057) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:00:27.406474 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [None req-ab29cfea-5040-4126-b903-deff694e0cc9 admin admin] The token cache is being invalidate because project e6ff065adc71404184a1fe9ccdf97bee was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=92057) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} May 23 13:00:27.406688 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [None req-ab29cfea-5040-4126-b903-deff694e0cc9 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=92057) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:00:27.407433 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [None req-ab29cfea-5040-4126-b903-deff694e0cc9 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=92057) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:00:27.409201 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 874/1743] 10.222.0.30 () {58 vars in 1232 bytes} [Mon May 23 13:00:26 2022] DELETE /identity/v3/projects/e6ff065adc71404184a1fe9ccdf97bee => generated 0 bytes in 673 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) May 23 13:00:27.415323 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-162d6c7e-58f9-478e-b609-4e65de2bc21a None None] REQUEST_METHOD: `POST` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:00:27.418397 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-162d6c7e-58f9-478e-b609-4e65de2bc21a None None] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:00:27.418948 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-162d6c7e-58f9-478e-b609-4e65de2bc21a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:00:27.480035 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.auth.core [None req-162d6c7e-58f9-478e-b609-4e65de2bc21a None None] MFA Rules not processed for user `0f7119f8fa5c4445977671140ded91a4`. Rule list: `[]` (Enabled: `True`). {{(pid=92056) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 13:00:27.483597 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.fernet_utils [None req-162d6c7e-58f9-478e-b609-4e65de2bc21a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92056) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:00:27.590449 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab29cfea-5040-4126-b903-deff694e0cc9 admin admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:00:27.591948 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.fernet_utils [None req-ab29cfea-5040-4126-b903-deff694e0cc9 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92057) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:00:27.733841 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 870/1744] 10.222.0.30 () {58 vars in 957 bytes} [Mon May 23 13:00:27 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 260 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) May 23 13:00:27.733841 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-162d6c7e-58f9-478e-b609-4e65de2bc21a None None] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:00:27.733841 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.fernet_utils [None req-162d6c7e-58f9-478e-b609-4e65de2bc21a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92056) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:00:27.733841 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-162d6c7e-58f9-478e-b609-4e65de2bc21a None None] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:00:27.734503 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2cc154f4-d2f3-477c-b207-fbdbb1321a87 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:00:27.734503 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cc154f4-d2f3-477c-b207-fbdbb1321a87 admin admin] REQUEST_METHOD: `DELETE` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:00:27.734503 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cc154f4-d2f3-477c-b207-fbdbb1321a87 admin admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:00:27.734503 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cc154f4-d2f3-477c-b207-fbdbb1321a87 admin admin] PATH_INFO: `/v3/projects/fb9c780793e7400e9b062200257ab68e` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:00:27.734820 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2cc154f4-d2f3-477c-b207-fbdbb1321a87 admin admin] RBAC: Authorizing `identity:delete_project(project_id=fb9c780793e7400e9b062200257ab68e)` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:00:27.734820 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2cc154f4-d2f3-477c-b207-fbdbb1321a87 admin admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:00:27.737330 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab29cfea-5040-4126-b903-deff694e0cc9 admin admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:00:27.743689 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-861f9857-5b29-43d3-b44e-bf2227c975c4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:00:27.746301 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-861f9857-5b29-43d3-b44e-bf2227c975c4 admin admin] REQUEST_METHOD: `DELETE` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:00:27.746577 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-861f9857-5b29-43d3-b44e-bf2227c975c4 admin admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:00:27.747089 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-861f9857-5b29-43d3-b44e-bf2227c975c4 admin admin] PATH_INFO: `/v3/projects/6157384de11a49aa99d6e5201e5ea368` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:00:27.748955 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-861f9857-5b29-43d3-b44e-bf2227c975c4 admin admin] RBAC: Authorizing `identity:delete_project(project_id=6157384de11a49aa99d6e5201e5ea368)` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:00:27.751505 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-861f9857-5b29-43d3-b44e-bf2227c975c4 admin admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:00:28.101866 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-2cc154f4-d2f3-477c-b207-fbdbb1321a87 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'fb9c780793e7400e9b062200257ab68e', 'request_id': 'req-2cc154f4-d2f3-477c-b207-fbdbb1321a87'} {{(pid=92056) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:00:28.192271 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-2cc154f4-d2f3-477c-b207-fbdbb1321a87 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'fb9c780793e7400e9b062200257ab68e', 'request_id': 'req-2cc154f4-d2f3-477c-b207-fbdbb1321a87'} {{(pid=92056) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:00:28.214190 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-2cc154f4-d2f3-477c-b207-fbdbb1321a87 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'fb9c780793e7400e9b062200257ab68e', 'request_id': 'req-2cc154f4-d2f3-477c-b207-fbdbb1321a87'} {{(pid=92056) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:00:28.233596 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [None req-861f9857-5b29-43d3-b44e-bf2227c975c4 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '6157384de11a49aa99d6e5201e5ea368', 'request_id': 'req-861f9857-5b29-43d3-b44e-bf2227c975c4'} {{(pid=92057) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:00:28.264786 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-2cc154f4-d2f3-477c-b207-fbdbb1321a87 admin admin] The token cache is being invalidate because project fb9c780793e7400e9b062200257ab68e was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=92056) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} May 23 13:00:28.265085 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-2cc154f4-d2f3-477c-b207-fbdbb1321a87 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=92056) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:00:28.266152 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [None req-861f9857-5b29-43d3-b44e-bf2227c975c4 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '6157384de11a49aa99d6e5201e5ea368', 'request_id': 'req-861f9857-5b29-43d3-b44e-bf2227c975c4'} {{(pid=92057) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:00:28.276270 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-2cc154f4-d2f3-477c-b207-fbdbb1321a87 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=92056) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:00:28.278564 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 871/1745] 10.222.0.30 () {58 vars in 1232 bytes} [Mon May 23 13:00:27 2022] DELETE /identity/v3/projects/fb9c780793e7400e9b062200257ab68e => generated 0 bytes in 604 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) May 23 13:00:28.291694 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b6f7a1c-7413-4a9e-843c-de3198be7c34 None None] REQUEST_METHOD: `POST` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:00:28.292026 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b6f7a1c-7413-4a9e-843c-de3198be7c34 None None] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:00:28.292316 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b6f7a1c-7413-4a9e-843c-de3198be7c34 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:00:28.377591 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [None req-861f9857-5b29-43d3-b44e-bf2227c975c4 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '6157384de11a49aa99d6e5201e5ea368', 'request_id': 'req-861f9857-5b29-43d3-b44e-bf2227c975c4'} {{(pid=92057) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:00:28.393113 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.auth.core [None req-7b6f7a1c-7413-4a9e-843c-de3198be7c34 None None] MFA Rules not processed for user `0f7119f8fa5c4445977671140ded91a4`. Rule list: `[]` (Enabled: `True`). {{(pid=92056) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 13:00:28.404549 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.fernet_utils [None req-7b6f7a1c-7413-4a9e-843c-de3198be7c34 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92056) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:00:28.407421 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [None req-861f9857-5b29-43d3-b44e-bf2227c975c4 admin admin] The token cache is being invalidate because project 6157384de11a49aa99d6e5201e5ea368 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=92057) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} May 23 13:00:28.407879 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [None req-861f9857-5b29-43d3-b44e-bf2227c975c4 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=92057) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:00:28.408863 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [None req-861f9857-5b29-43d3-b44e-bf2227c975c4 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=92057) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:00:28.411080 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 875/1746] 10.222.0.30 () {58 vars in 1232 bytes} [Mon May 23 13:00:27 2022] DELETE /identity/v3/projects/6157384de11a49aa99d6e5201e5ea368 => generated 0 bytes in 821 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) May 23 13:00:28.413052 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c9e6036-3874-4967-a0bc-c32eb0a0484f None None] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:00:28.413392 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c9e6036-3874-4967-a0bc-c32eb0a0484f None None] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:00:28.413791 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c9e6036-3874-4967-a0bc-c32eb0a0484f None None] PATH_INFO: `/` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:00:28.416499 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 876/1747] 10.222.0.30 () {54 vars in 921 bytes} [Mon May 23 13:00:28 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) May 23 13:00:28.417691 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c9e6036-3874-4967-a0bc-c32eb0a0484f None None] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:00:28.421215 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.fernet_utils [None req-6c9e6036-3874-4967-a0bc-c32eb0a0484f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92057) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:00:28.572773 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 872/1748] 10.222.0.30 () {58 vars in 957 bytes} [Mon May 23 13:00:28 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 294 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) May 23 13:00:28.578522 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-345ade78-42ff-4e83-be39-c5ab79f82df7 None None] REQUEST_METHOD: `POST` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:00:28.580983 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-345ade78-42ff-4e83-be39-c5ab79f82df7 None None] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:00:28.581487 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-345ade78-42ff-4e83-be39-c5ab79f82df7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:00:28.621182 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c9e6036-3874-4967-a0bc-c32eb0a0484f None None] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:00:28.633271 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab9eafe4-8c88-4821-9d56-27467749cc7b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:00:28.636680 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab9eafe4-8c88-4821-9d56-27467749cc7b admin admin] REQUEST_METHOD: `DELETE` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:00:28.639534 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab9eafe4-8c88-4821-9d56-27467749cc7b admin admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:00:28.639534 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab9eafe4-8c88-4821-9d56-27467749cc7b admin admin] PATH_INFO: `/v3/users/89a816fe27194bf38144a959521f3338` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:00:28.645102 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab9eafe4-8c88-4821-9d56-27467749cc7b admin admin] RBAC: Authorizing `identity:delete_user(user_id=89a816fe27194bf38144a959521f3338)` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:00:28.647431 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab9eafe4-8c88-4821-9d56-27467749cc7b admin admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:00:28.652878 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.auth.core [None req-345ade78-42ff-4e83-be39-c5ab79f82df7 None None] MFA Rules not processed for user `0f7119f8fa5c4445977671140ded91a4`. Rule list: `[]` (Enabled: `True`). {{(pid=92056) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 13:00:28.668464 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.fernet_utils [None req-345ade78-42ff-4e83-be39-c5ab79f82df7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92056) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:00:28.696032 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 873/1749] 10.222.0.30 () {58 vars in 957 bytes} [Mon May 23 13:00:28 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 123 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) May 23 13:00:28.699582 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-230a718a-d442-4f31-a6a3-94096ffe9315 None None] REQUEST_METHOD: `POST` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:00:28.699582 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-230a718a-d442-4f31-a6a3-94096ffe9315 None None] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:00:28.699582 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-230a718a-d442-4f31-a6a3-94096ffe9315 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:00:28.747734 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.auth.core [None req-230a718a-d442-4f31-a6a3-94096ffe9315 None None] MFA Rules not processed for user `cbe9931d94d44cc7ace0e83a7e5015fc`. Rule list: `[]` (Enabled: `True`). {{(pid=92056) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 13:00:28.765197 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.fernet_utils [None req-230a718a-d442-4f31-a6a3-94096ffe9315 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92056) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:00:28.880154 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:DELETE=10,keystone:SELECT=88,keystone:INSERT=2 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 13:00:28.904404 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [None req-ab9eafe4-8c88-4821-9d56-27467749cc7b admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '89a816fe27194bf38144a959521f3338'} {{(pid=92057) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:00:28.940455 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 13:00:28.943483 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 874/1750] 10.222.0.30 () {58 vars in 1018 bytes} [Mon May 23 13:00:28 2022] POST /identity/v3/auth/tokens => generated 3362 bytes in 247 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) May 23 13:00:28.944434 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-230a718a-d442-4f31-a6a3-94096ffe9315 None None] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:00:28.955623 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-230a718a-d442-4f31-a6a3-94096ffe9315 None None] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:00:28.960840 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d001a5b-cbc8-4576-9b57-e80f5e017b36 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:00:28.961612 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d001a5b-cbc8-4576-9b57-e80f5e017b36 admin admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:00:28.962015 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d001a5b-cbc8-4576-9b57-e80f5e017b36 admin admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:00:28.968794 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d001a5b-cbc8-4576-9b57-e80f5e017b36 admin admin] PATH_INFO: `/v3/domains` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:00:28.968794 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d001a5b-cbc8-4576-9b57-e80f5e017b36 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=92056) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} May 23 13:00:28.968794 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d001a5b-cbc8-4576-9b57-e80f5e017b36 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:00:28.968794 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d001a5b-cbc8-4576-9b57-e80f5e017b36 admin admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:00:28.985578 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 875/1751] 10.222.0.30 () {58 vars in 1152 bytes} [Mon May 23 13:00:28 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 39 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) May 23 13:00:28.985578 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43b909cf-8772-4f17-b6e5-026602ee0e74 None None] REQUEST_METHOD: `POST` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:00:28.985578 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43b909cf-8772-4f17-b6e5-026602ee0e74 None None] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:00:28.985578 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43b909cf-8772-4f17-b6e5-026602ee0e74 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:00:28.996857 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [None req-ab9eafe4-8c88-4821-9d56-27467749cc7b admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '89a816fe27194bf38144a959521f3338'} {{(pid=92057) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:00:29.015428 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [None req-ab9eafe4-8c88-4821-9d56-27467749cc7b admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '89a816fe27194bf38144a959521f3338'} {{(pid=92057) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:00:29.016341 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [None req-ab9eafe4-8c88-4821-9d56-27467749cc7b admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '89a816fe27194bf38144a959521f3338'} {{(pid=92057) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:00:29.048757 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.auth.core [None req-43b909cf-8772-4f17-b6e5-026602ee0e74 None None] MFA Rules not processed for user `0f7119f8fa5c4445977671140ded91a4`. Rule list: `[]` (Enabled: `True`). {{(pid=92056) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 13:00:29.051277 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.fernet_utils [None req-43b909cf-8772-4f17-b6e5-026602ee0e74 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92056) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:00:29.051966 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [None req-ab9eafe4-8c88-4821-9d56-27467749cc7b admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '89a816fe27194bf38144a959521f3338'} {{(pid=92057) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:00:29.071941 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 877/1752] 10.222.0.30 () {58 vars in 1223 bytes} [Mon May 23 13:00:28 2022] DELETE /identity/v3/users/89a816fe27194bf38144a959521f3338 => generated 0 bytes in 652 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) May 23 13:00:29.071941 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab9eafe4-8c88-4821-9d56-27467749cc7b admin admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:00:29.079243 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 876/1753] 10.222.0.30 () {58 vars in 957 bytes} [Mon May 23 13:00:28 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 96 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) May 23 13:00:29.083287 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43b909cf-8772-4f17-b6e5-026602ee0e74 None None] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:00:29.085630 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.fernet_utils [None req-ab9eafe4-8c88-4821-9d56-27467749cc7b admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92057) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:00:29.095650 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43b909cf-8772-4f17-b6e5-026602ee0e74 None None] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:00:29.206531 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b83fa81-dfa5-4bab-8b15-7a52bf3fd7ca admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:00:29.208207 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b83fa81-dfa5-4bab-8b15-7a52bf3fd7ca admin admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:00:29.208499 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b83fa81-dfa5-4bab-8b15-7a52bf3fd7ca admin admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:00:29.208781 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b83fa81-dfa5-4bab-8b15-7a52bf3fd7ca admin admin] PATH_INFO: `/v3/domains` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:00:29.209758 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b83fa81-dfa5-4bab-8b15-7a52bf3fd7ca admin admin] RBAC: Adding query filter params (name=Default) {{(pid=92056) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} May 23 13:00:29.210040 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b83fa81-dfa5-4bab-8b15-7a52bf3fd7ca admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:00:29.211682 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b83fa81-dfa5-4bab-8b15-7a52bf3fd7ca admin admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:00:29.225368 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 877/1754] 10.222.0.30 () {58 vars in 1152 bytes} [Mon May 23 13:00:29 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 143 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) May 23 13:00:29.226476 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b83fa81-dfa5-4bab-8b15-7a52bf3fd7ca admin admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:00:29.229925 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.fernet_utils [None req-4b83fa81-dfa5-4bab-8b15-7a52bf3fd7ca admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92056) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:00:29.235404 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab9eafe4-8c88-4821-9d56-27467749cc7b admin admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:00:29.241837 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19c52826-8396-45e7-90f9-1181adc62372 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:00:29.242570 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19c52826-8396-45e7-90f9-1181adc62372 admin admin] REQUEST_METHOD: `POST` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:00:29.242984 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19c52826-8396-45e7-90f9-1181adc62372 admin admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:00:29.243097 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b83fa81-dfa5-4bab-8b15-7a52bf3fd7ca admin admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:00:29.243267 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19c52826-8396-45e7-90f9-1181adc62372 admin admin] PATH_INFO: `/v3/projects` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:00:29.244911 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19c52826-8396-45e7-90f9-1181adc62372 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:00:29.247160 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19c52826-8396-45e7-90f9-1181adc62372 admin admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:00:29.248719 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a4e1e1f-19b4-4be7-ba9c-093b181b9c3e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:00:29.249635 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a4e1e1f-19b4-4be7-ba9c-093b181b9c3e admin admin] REQUEST_METHOD: `DELETE` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:00:29.249946 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a4e1e1f-19b4-4be7-ba9c-093b181b9c3e admin admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:00:29.250229 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a4e1e1f-19b4-4be7-ba9c-093b181b9c3e admin admin] PATH_INFO: `/v3/projects/ac05364258c444ae9ce20b6d509f7ed0` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:00:29.251430 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a4e1e1f-19b4-4be7-ba9c-093b181b9c3e admin admin] RBAC: Authorizing `identity:delete_project(project_id=ac05364258c444ae9ce20b6d509f7ed0)` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:00:29.253643 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a4e1e1f-19b4-4be7-ba9c-093b181b9c3e admin admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:00:29.350380 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 878/1755] 10.222.0.30 () {60 vars in 1152 bytes} [Mon May 23 13:00:29 2022] POST /identity/v3/projects => generated 373 bytes in 278 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) May 23 13:00:29.350380 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19c52826-8396-45e7-90f9-1181adc62372 admin admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:00:29.355947 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.fernet_utils [None req-19c52826-8396-45e7-90f9-1181adc62372 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92057) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:00:29.402717 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-4a4e1e1f-19b4-4be7-ba9c-093b181b9c3e admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'ac05364258c444ae9ce20b6d509f7ed0', 'request_id': 'req-4a4e1e1f-19b4-4be7-ba9c-093b181b9c3e'} {{(pid=92056) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:00:29.439144 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19c52826-8396-45e7-90f9-1181adc62372 admin admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:00:29.503816 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49a109ef-548e-434b-8151-9240b7a2ea2e service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cbe9931d94d44cc7ace0e83a7e5015fc', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9c389ae65aaa4542bb5a5c4fd0bacf5b', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'member', 'reader', 'service', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:00:29.504474 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49a109ef-548e-434b-8151-9240b7a2ea2e service nova] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:00:29.504757 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49a109ef-548e-434b-8151-9240b7a2ea2e service nova] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:00:29.505027 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49a109ef-548e-434b-8151-9240b7a2ea2e service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:00:29.506807 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.fernet_utils [None req-49a109ef-548e-434b-8151-9240b7a2ea2e service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92057) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:00:29.518768 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49a109ef-548e-434b-8151-9240b7a2ea2e service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:00:29.520335 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49a109ef-548e-434b-8151-9240b7a2ea2e service nova] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:00:29.522504 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-4a4e1e1f-19b4-4be7-ba9c-093b181b9c3e admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'ac05364258c444ae9ce20b6d509f7ed0', 'request_id': 'req-4a4e1e1f-19b4-4be7-ba9c-093b181b9c3e'} {{(pid=92056) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:00:29.530065 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-4a4e1e1f-19b4-4be7-ba9c-093b181b9c3e admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'ac05364258c444ae9ce20b6d509f7ed0', 'request_id': 'req-4a4e1e1f-19b4-4be7-ba9c-093b181b9c3e'} {{(pid=92056) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:00:29.530951 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 879/1756] 10.222.0.30 () {60 vars in 1365 bytes} [Mon May 23 13:00:29 2022] GET /identity/v3/auth/tokens => generated 3362 bytes in 183 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) May 23 13:00:29.532527 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49a109ef-548e-434b-8151-9240b7a2ea2e service nova] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:00:29.542703 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-4a4e1e1f-19b4-4be7-ba9c-093b181b9c3e admin admin] The token cache is being invalidate because project ac05364258c444ae9ce20b6d509f7ed0 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=92056) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} May 23 13:00:29.542991 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-4a4e1e1f-19b4-4be7-ba9c-093b181b9c3e admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=92056) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:00:29.543696 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-4a4e1e1f-19b4-4be7-ba9c-093b181b9c3e admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=92056) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:00:29.545638 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 878/1757] 10.222.0.30 () {58 vars in 1232 bytes} [Mon May 23 13:00:29 2022] DELETE /identity/v3/projects/ac05364258c444ae9ce20b6d509f7ed0 => generated 0 bytes in 320 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) May 23 13:00:29.546524 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a4e1e1f-19b4-4be7-ba9c-093b181b9c3e admin admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:00:29.549291 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.fernet_utils [None req-4a4e1e1f-19b4-4be7-ba9c-093b181b9c3e admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92056) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:00:29.597312 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49a109ef-548e-434b-8151-9240b7a2ea2e service nova] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:00:29.602534 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1835b6c9-78eb-464d-a1a1-facbcfa14343 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:00:29.603796 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1835b6c9-78eb-464d-a1a1-facbcfa14343 admin admin] REQUEST_METHOD: `POST` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:00:29.604152 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1835b6c9-78eb-464d-a1a1-facbcfa14343 admin admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:00:29.604499 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1835b6c9-78eb-464d-a1a1-facbcfa14343 admin admin] PATH_INFO: `/v3/projects` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:00:29.605199 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1835b6c9-78eb-464d-a1a1-facbcfa14343 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:00:29.607165 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1835b6c9-78eb-464d-a1a1-facbcfa14343 admin admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:00:29.629909 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a4e1e1f-19b4-4be7-ba9c-093b181b9c3e admin admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:00:29.634815 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39cc3148-6ef9-4274-95c2-3036796689ac admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:00:29.635505 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39cc3148-6ef9-4274-95c2-3036796689ac admin admin] REQUEST_METHOD: `POST` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:00:29.635786 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39cc3148-6ef9-4274-95c2-3036796689ac admin admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:00:29.636061 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39cc3148-6ef9-4274-95c2-3036796689ac admin admin] PATH_INFO: `/v3/users` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:00:29.636752 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39cc3148-6ef9-4274-95c2-3036796689ac admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:00:29.638967 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39cc3148-6ef9-4274-95c2-3036796689ac admin admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:00:29.707443 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 880/1758] 10.222.0.30 () {60 vars in 1152 bytes} [Mon May 23 13:00:29 2022] POST /identity/v3/projects => generated 403 bytes in 176 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) May 23 13:00:29.708293 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1835b6c9-78eb-464d-a1a1-facbcfa14343 admin admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:00:29.709870 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.fernet_utils [None req-1835b6c9-78eb-464d-a1a1-facbcfa14343 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92057) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:00:29.736094 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 879/1759] 10.222.0.30 () {60 vars in 1143 bytes} [Mon May 23 13:00:29 2022] POST /identity/v3/users => generated 287 bytes in 188 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) May 23 13:00:29.736094 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39cc3148-6ef9-4274-95c2-3036796689ac admin admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:00:29.737323 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.fernet_utils [None req-39cc3148-6ef9-4274-95c2-3036796689ac admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92056) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:00:29.750061 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39cc3148-6ef9-4274-95c2-3036796689ac admin admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:00:29.778283 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1835b6c9-78eb-464d-a1a1-facbcfa14343 admin admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:00:29.783099 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47a80aa6-ba85-4b1c-bff2-27eba60008be admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:00:29.784038 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47a80aa6-ba85-4b1c-bff2-27eba60008be admin admin] REQUEST_METHOD: `DELETE` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:00:29.784324 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47a80aa6-ba85-4b1c-bff2-27eba60008be admin admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:00:29.784602 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47a80aa6-ba85-4b1c-bff2-27eba60008be admin admin] PATH_INFO: `/v3/users/184466528299445ea47589748666f040` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:00:29.785783 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47a80aa6-ba85-4b1c-bff2-27eba60008be admin admin] RBAC: Authorizing `identity:delete_user(user_id=184466528299445ea47589748666f040)` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:00:29.787738 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47a80aa6-ba85-4b1c-bff2-27eba60008be admin admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:00:29.805938 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ed319b9-440c-4531-bb83-16cacb9b1440 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:00:29.807462 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ed319b9-440c-4531-bb83-16cacb9b1440 admin admin] REQUEST_METHOD: `DELETE` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:00:29.807722 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ed319b9-440c-4531-bb83-16cacb9b1440 admin admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:00:29.808003 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ed319b9-440c-4531-bb83-16cacb9b1440 admin admin] PATH_INFO: `/v3/projects/52d9418e818d4facaf28bd0ae70276fe` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:00:29.809213 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ed319b9-440c-4531-bb83-16cacb9b1440 admin admin] RBAC: Authorizing `identity:delete_project(project_id=52d9418e818d4facaf28bd0ae70276fe)` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:00:29.811161 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ed319b9-440c-4531-bb83-16cacb9b1440 admin admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:00:30.039093 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-4ed319b9-440c-4531-bb83-16cacb9b1440 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '52d9418e818d4facaf28bd0ae70276fe', 'request_id': 'req-4ed319b9-440c-4531-bb83-16cacb9b1440'} {{(pid=92056) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:00:30.039289 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=89,keystone:DELETE=9,keystone:INSERT=2 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 13:00:30.042559 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [None req-47a80aa6-ba85-4b1c-bff2-27eba60008be admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '184466528299445ea47589748666f040'} {{(pid=92057) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:00:30.065759 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 13:00:30.082642 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-4ed319b9-440c-4531-bb83-16cacb9b1440 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '52d9418e818d4facaf28bd0ae70276fe', 'request_id': 'req-4ed319b9-440c-4531-bb83-16cacb9b1440'} {{(pid=92056) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:00:30.089991 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-4ed319b9-440c-4531-bb83-16cacb9b1440 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '52d9418e818d4facaf28bd0ae70276fe', 'request_id': 'req-4ed319b9-440c-4531-bb83-16cacb9b1440'} {{(pid=92056) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:00:30.100692 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-4ed319b9-440c-4531-bb83-16cacb9b1440 admin admin] The token cache is being invalidate because project 52d9418e818d4facaf28bd0ae70276fe was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=92056) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} May 23 13:00:30.100978 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-4ed319b9-440c-4531-bb83-16cacb9b1440 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=92056) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:00:30.101645 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-4ed319b9-440c-4531-bb83-16cacb9b1440 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=92056) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:00:30.107598 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 880/1760] 10.222.0.30 () {58 vars in 1232 bytes} [Mon May 23 13:00:29 2022] DELETE /identity/v3/projects/52d9418e818d4facaf28bd0ae70276fe => generated 0 bytes in 369 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) May 23 13:00:30.107598 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ed319b9-440c-4531-bb83-16cacb9b1440 admin admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:00:30.107598 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.fernet_utils [None req-4ed319b9-440c-4531-bb83-16cacb9b1440 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92056) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:00:30.180189 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [None req-47a80aa6-ba85-4b1c-bff2-27eba60008be admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '184466528299445ea47589748666f040'} {{(pid=92057) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:00:30.184751 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ed319b9-440c-4531-bb83-16cacb9b1440 admin admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:00:30.190244 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [None req-47a80aa6-ba85-4b1c-bff2-27eba60008be admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '184466528299445ea47589748666f040'} {{(pid=92057) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:00:30.194695 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-560df738-9666-45b4-9206-251ac7b557d1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:00:30.194695 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-560df738-9666-45b4-9206-251ac7b557d1 admin admin] REQUEST_METHOD: `POST` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:00:30.194695 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-560df738-9666-45b4-9206-251ac7b557d1 admin admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:00:30.195076 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [None req-47a80aa6-ba85-4b1c-bff2-27eba60008be admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '184466528299445ea47589748666f040'} {{(pid=92057) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:00:30.195171 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-560df738-9666-45b4-9206-251ac7b557d1 admin admin] PATH_INFO: `/v3/users` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:00:30.195171 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-560df738-9666-45b4-9206-251ac7b557d1 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:00:30.196726 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-560df738-9666-45b4-9206-251ac7b557d1 admin admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:00:30.238904 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [None req-47a80aa6-ba85-4b1c-bff2-27eba60008be admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '184466528299445ea47589748666f040'} {{(pid=92057) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:00:30.247159 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 881/1761] 10.222.0.30 () {58 vars in 1223 bytes} [Mon May 23 13:00:29 2022] DELETE /identity/v3/users/184466528299445ea47589748666f040 => generated 0 bytes in 539 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) May 23 13:00:30.254491 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47a80aa6-ba85-4b1c-bff2-27eba60008be admin admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:00:30.254491 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.fernet_utils [None req-47a80aa6-ba85-4b1c-bff2-27eba60008be admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92057) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:00:30.284701 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 881/1762] 10.222.0.30 () {60 vars in 1143 bytes} [Mon May 23 13:00:30 2022] POST /identity/v3/users => generated 302 bytes in 177 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) May 23 13:00:30.284701 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23adc58e-803d-4e71-8fc4-3d151c928736 None None] REQUEST_METHOD: `POST` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:00:30.284701 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23adc58e-803d-4e71-8fc4-3d151c928736 None None] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:00:30.285559 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23adc58e-803d-4e71-8fc4-3d151c928736 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:00:30.315438 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.auth.core [None req-23adc58e-803d-4e71-8fc4-3d151c928736 None None] MFA Rules not processed for user `0f7119f8fa5c4445977671140ded91a4`. Rule list: `[]` (Enabled: `True`). {{(pid=92056) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 13:00:30.316835 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.fernet_utils [None req-23adc58e-803d-4e71-8fc4-3d151c928736 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92056) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:00:30.376221 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47a80aa6-ba85-4b1c-bff2-27eba60008be admin admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:00:30.381523 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22cbd267-e1b6-4971-885b-b9544010a104 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:00:30.382021 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22cbd267-e1b6-4971-885b-b9544010a104 admin admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:00:30.382313 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22cbd267-e1b6-4971-885b-b9544010a104 admin admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:00:30.382629 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22cbd267-e1b6-4971-885b-b9544010a104 admin admin] PATH_INFO: `/v3/roles` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:00:30.383324 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-22cbd267-e1b6-4971-885b-b9544010a104 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:00:30.385462 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-22cbd267-e1b6-4971-885b-b9544010a104 admin admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:00:30.401257 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 882/1763] 10.222.0.30 () {58 vars in 1121 bytes} [Mon May 23 13:00:30 2022] GET /identity/v3/roles => generated 1420 bytes in 153 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) May 23 13:00:30.402316 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22cbd267-e1b6-4971-885b-b9544010a104 admin admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:00:30.404317 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.fernet_utils [None req-22cbd267-e1b6-4971-885b-b9544010a104 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92057) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:00:30.419077 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22cbd267-e1b6-4971-885b-b9544010a104 admin admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:00:30.425822 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be25ebf2-e29b-4461-a0d7-d65f447f9e42 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:00:30.426531 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be25ebf2-e29b-4461-a0d7-d65f447f9e42 admin admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:00:30.426958 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be25ebf2-e29b-4461-a0d7-d65f447f9e42 admin admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:00:30.427244 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be25ebf2-e29b-4461-a0d7-d65f447f9e42 admin admin] PATH_INFO: `/v3/roles` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:00:30.427972 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be25ebf2-e29b-4461-a0d7-d65f447f9e42 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:00:30.429699 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be25ebf2-e29b-4461-a0d7-d65f447f9e42 admin admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:00:30.440690 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 883/1764] 10.222.0.30 () {58 vars in 1121 bytes} [Mon May 23 13:00:30 2022] GET /identity/v3/roles => generated 1420 bytes in 39 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) May 23 13:00:30.441346 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be25ebf2-e29b-4461-a0d7-d65f447f9e42 admin admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:00:30.448637 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 882/1765] 10.222.0.30 () {58 vars in 957 bytes} [Mon May 23 13:00:30 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 166 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) May 23 13:00:30.449511 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23adc58e-803d-4e71-8fc4-3d151c928736 None None] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:00:30.452107 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be25ebf2-e29b-4461-a0d7-d65f447f9e42 admin admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:00:30.457096 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d90c2325-de11-4a51-861a-8e776ea0a3a4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:00:30.457768 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d90c2325-de11-4a51-861a-8e776ea0a3a4 admin admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:00:30.458087 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d90c2325-de11-4a51-861a-8e776ea0a3a4 admin admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:00:30.458481 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d90c2325-de11-4a51-861a-8e776ea0a3a4 admin admin] PATH_INFO: `/v3/roles` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:00:30.459208 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d90c2325-de11-4a51-861a-8e776ea0a3a4 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:00:30.463362 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d90c2325-de11-4a51-861a-8e776ea0a3a4 admin admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:00:30.465393 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23adc58e-803d-4e71-8fc4-3d151c928736 None None] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:00:30.470810 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd20554e-e2ff-4f5c-a83e-77daaf3e7aa5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:00:30.471818 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd20554e-e2ff-4f5c-a83e-77daaf3e7aa5 admin admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:00:30.472391 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd20554e-e2ff-4f5c-a83e-77daaf3e7aa5 admin admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:00:30.472508 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd20554e-e2ff-4f5c-a83e-77daaf3e7aa5 admin admin] PATH_INFO: `/v3/roles` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:00:30.473330 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd20554e-e2ff-4f5c-a83e-77daaf3e7aa5 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:00:30.475098 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd20554e-e2ff-4f5c-a83e-77daaf3e7aa5 admin admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:00:30.487503 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 884/1766] 10.222.0.30 () {58 vars in 1121 bytes} [Mon May 23 13:00:30 2022] GET /identity/v3/roles => generated 1420 bytes in 38 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) May 23 13:00:30.487503 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ebf2e75-d28a-469a-bfbd-d7493e0990b7 None None] REQUEST_METHOD: `POST` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:00:30.487503 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ebf2e75-d28a-469a-bfbd-d7493e0990b7 None None] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:00:30.487503 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ebf2e75-d28a-469a-bfbd-d7493e0990b7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:00:30.489336 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 883/1767] 10.222.0.30 () {58 vars in 1121 bytes} [Mon May 23 13:00:30 2022] GET /identity/v3/roles => generated 1420 bytes in 40 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) May 23 13:00:30.490148 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd20554e-e2ff-4f5c-a83e-77daaf3e7aa5 admin admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:00:30.503362 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd20554e-e2ff-4f5c-a83e-77daaf3e7aa5 admin admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:00:30.507078 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=94,keystone:INSERT=1,keystone:DELETE=5 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 13:00:30.511539 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.auth.core [None req-1ebf2e75-d28a-469a-bfbd-d7493e0990b7 None None] MFA Rules not processed for user `0f7119f8fa5c4445977671140ded91a4`. Rule list: `[]` (Enabled: `True`). {{(pid=92057) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 13:00:30.513824 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.fernet_utils [None req-1ebf2e75-d28a-469a-bfbd-d7493e0990b7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92057) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:00:30.527922 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cebbbe22-ac99-487d-a637-63291ead42d9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:00:30.530503 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cebbbe22-ac99-487d-a637-63291ead42d9 admin admin] REQUEST_METHOD: `PUT` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:00:30.530887 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cebbbe22-ac99-487d-a637-63291ead42d9 admin admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:00:30.531168 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cebbbe22-ac99-487d-a637-63291ead42d9 admin admin] PATH_INFO: `/v3/projects/de4eed939156431fa99178951d5fcbec/users/7adac7fe4243402aa25085cacfe2b29d/roles/80fb22507c9a4ff09d44d8f12f6fff1a` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:00:30.537013 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 885/1768] 10.222.0.30 () {58 vars in 957 bytes} [Mon May 23 13:00:30 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 54 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) May 23 13:00:30.537013 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ebf2e75-d28a-469a-bfbd-d7493e0990b7 None None] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:00:30.552273 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ebf2e75-d28a-469a-bfbd-d7493e0990b7 None None] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:00:30.561432 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c3a1b74-8d88-4595-9871-1f73a13d20ee admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:00:30.562478 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c3a1b74-8d88-4595-9871-1f73a13d20ee admin admin] REQUEST_METHOD: `PUT` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:00:30.562832 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c3a1b74-8d88-4595-9871-1f73a13d20ee admin admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:00:30.563128 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c3a1b74-8d88-4595-9871-1f73a13d20ee admin admin] PATH_INFO: `/v3/projects/67bf533d20784c63aaff424cec03b974/users/f92e0a102a794ded99da9c9385266db1/roles/80fb22507c9a4ff09d44d8f12f6fff1a` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:00:30.571739 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cebbbe22-ac99-487d-a637-63291ead42d9 admin admin] RBAC: Authorizing `identity:create_grant(project_id=de4eed939156431fa99178951d5fcbec, user_id=7adac7fe4243402aa25085cacfe2b29d, role_id=80fb22507c9a4ff09d44d8f12f6fff1a)` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:00:30.576349 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cebbbe22-ac99-487d-a637-63291ead42d9 admin admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:00:30.579359 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c3a1b74-8d88-4595-9871-1f73a13d20ee admin admin] RBAC: Authorizing `identity:create_grant(project_id=67bf533d20784c63aaff424cec03b974, user_id=f92e0a102a794ded99da9c9385266db1, role_id=80fb22507c9a4ff09d44d8f12f6fff1a)` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:00:30.582038 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c3a1b74-8d88-4595-9871-1f73a13d20ee admin admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:00:30.600572 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=89,keystone:INSERT=4,keystone:DELETE=7 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 13:00:30.684193 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 884/1769] 10.222.0.30 () {60 vars in 1482 bytes} [Mon May 23 13:00:30 2022] PUT /identity/v3/projects/de4eed939156431fa99178951d5fcbec/users/7adac7fe4243402aa25085cacfe2b29d/roles/80fb22507c9a4ff09d44d8f12f6fff1a => generated 0 bytes in 194 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) May 23 13:00:30.685140 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cebbbe22-ac99-487d-a637-63291ead42d9 admin admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:00:30.715074 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 886/1770] 10.222.0.30 () {60 vars in 1482 bytes} [Mon May 23 13:00:30 2022] PUT /identity/v3/projects/67bf533d20784c63aaff424cec03b974/users/f92e0a102a794ded99da9c9385266db1/roles/80fb22507c9a4ff09d44d8f12f6fff1a => generated 0 bytes in 181 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) May 23 13:00:30.723691 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77ca2936-9962-42e7-bb17-e9a56b0fecb7 None None] REQUEST_METHOD: `POST` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:00:30.723691 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77ca2936-9962-42e7-bb17-e9a56b0fecb7 None None] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:00:30.723691 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77ca2936-9962-42e7-bb17-e9a56b0fecb7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:00:30.747209 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.auth.core [None req-77ca2936-9962-42e7-bb17-e9a56b0fecb7 None None] MFA Rules not processed for user `7adac7fe4243402aa25085cacfe2b29d`. Rule list: `[]` (Enabled: `True`). {{(pid=92057) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 13:00:30.748427 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.fernet_utils [None req-77ca2936-9962-42e7-bb17-e9a56b0fecb7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92057) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:00:30.761683 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cebbbe22-ac99-487d-a637-63291ead42d9 admin admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:00:30.821420 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 887/1771] 10.222.0.30 () {58 vars in 957 bytes} [Mon May 23 13:00:30 2022] POST /identity/v3/auth/tokens => generated 3240 bytes in 106 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) May 23 13:00:30.823268 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fe1bb83-2942-4e5a-aa9c-479c07082378 None None] REQUEST_METHOD: `POST` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:00:30.823618 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fe1bb83-2942-4e5a-aa9c-479c07082378 None None] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:00:30.823966 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fe1bb83-2942-4e5a-aa9c-479c07082378 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:00:30.825822 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-753b2af1-a016-4741-b324-8000ba227cea admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:00:30.826532 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-753b2af1-a016-4741-b324-8000ba227cea admin admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:00:30.826815 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-753b2af1-a016-4741-b324-8000ba227cea admin admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:00:30.827118 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-753b2af1-a016-4741-b324-8000ba227cea admin admin] PATH_INFO: `/v3/domains` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:00:30.827906 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-753b2af1-a016-4741-b324-8000ba227cea admin admin] RBAC: Adding query filter params (name=Default) {{(pid=92056) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} May 23 13:00:30.828226 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-753b2af1-a016-4741-b324-8000ba227cea admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:00:30.829830 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-753b2af1-a016-4741-b324-8000ba227cea admin admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:00:30.845211 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 885/1772] 10.222.0.30 () {58 vars in 1152 bytes} [Mon May 23 13:00:30 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 157 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) May 23 13:00:30.845211 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-753b2af1-a016-4741-b324-8000ba227cea admin admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:00:30.845211 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.fernet_utils [None req-753b2af1-a016-4741-b324-8000ba227cea admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92056) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:00:30.851881 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.auth.core [None req-5fe1bb83-2942-4e5a-aa9c-479c07082378 None None] MFA Rules not processed for user `f92e0a102a794ded99da9c9385266db1`. Rule list: `[]` (Enabled: `True`). {{(pid=92057) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 13:00:30.856396 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.fernet_utils [None req-5fe1bb83-2942-4e5a-aa9c-479c07082378 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92057) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:00:30.858268 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-753b2af1-a016-4741-b324-8000ba227cea admin admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:00:30.862852 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57a03956-e870-4022-be8f-892a206cd8c2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:00:30.863715 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57a03956-e870-4022-be8f-892a206cd8c2 admin admin] REQUEST_METHOD: `DELETE` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:00:30.863977 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57a03956-e870-4022-be8f-892a206cd8c2 admin admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:00:30.864250 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57a03956-e870-4022-be8f-892a206cd8c2 admin admin] PATH_INFO: `/v3/projects/7c8fe87d0aec4f5cacce879cc336d002` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:00:30.865462 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57a03956-e870-4022-be8f-892a206cd8c2 admin admin] RBAC: Authorizing `identity:delete_project(project_id=7c8fe87d0aec4f5cacce879cc336d002)` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:00:30.867404 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57a03956-e870-4022-be8f-892a206cd8c2 admin admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:00:30.919845 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 888/1773] 10.222.0.30 () {58 vars in 957 bytes} [Mon May 23 13:00:30 2022] POST /identity/v3/auth/tokens => generated 3270 bytes in 98 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) May 23 13:00:30.920692 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5fe1bb83-2942-4e5a-aa9c-479c07082378 None None] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:00:30.922188 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.fernet_utils [None req-5fe1bb83-2942-4e5a-aa9c-479c07082378 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92057) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:00:30.985891 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-57a03956-e870-4022-be8f-892a206cd8c2 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '7c8fe87d0aec4f5cacce879cc336d002', 'request_id': 'req-57a03956-e870-4022-be8f-892a206cd8c2'} {{(pid=92056) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:00:31.014810 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5fe1bb83-2942-4e5a-aa9c-479c07082378 None None] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:00:31.082660 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-57a03956-e870-4022-be8f-892a206cd8c2 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '7c8fe87d0aec4f5cacce879cc336d002', 'request_id': 'req-57a03956-e870-4022-be8f-892a206cd8c2'} {{(pid=92056) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:00:31.093392 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-57a03956-e870-4022-be8f-892a206cd8c2 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '7c8fe87d0aec4f5cacce879cc336d002', 'request_id': 'req-57a03956-e870-4022-be8f-892a206cd8c2'} {{(pid=92056) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:00:31.115286 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1851b60c-0231-42ff-ab45-72d094278655 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cbe9931d94d44cc7ace0e83a7e5015fc', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9c389ae65aaa4542bb5a5c4fd0bacf5b', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'member', 'reader', 'service', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:00:31.120382 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-57a03956-e870-4022-be8f-892a206cd8c2 admin admin] The token cache is being invalidate because project 7c8fe87d0aec4f5cacce879cc336d002 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=92056) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} May 23 13:00:31.120382 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-57a03956-e870-4022-be8f-892a206cd8c2 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=92056) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:00:31.121261 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1851b60c-0231-42ff-ab45-72d094278655 service nova] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:00:31.121261 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1851b60c-0231-42ff-ab45-72d094278655 service nova] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:00:31.121261 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1851b60c-0231-42ff-ab45-72d094278655 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:00:31.121462 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-57a03956-e870-4022-be8f-892a206cd8c2 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=92056) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:00:31.125587 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 886/1774] 10.222.0.30 () {58 vars in 1232 bytes} [Mon May 23 13:00:30 2022] DELETE /identity/v3/projects/7c8fe87d0aec4f5cacce879cc336d002 => generated 0 bytes in 280 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) May 23 13:00:31.126891 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.fernet_utils [None req-1851b60c-0231-42ff-ab45-72d094278655 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92057) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:00:31.128856 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57a03956-e870-4022-be8f-892a206cd8c2 admin admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:00:31.132169 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.fernet_utils [None req-57a03956-e870-4022-be8f-892a206cd8c2 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92056) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:00:31.198479 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=93,keystone:DELETE=5,keystone:INSERT=2 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 13:00:31.246016 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1851b60c-0231-42ff-ab45-72d094278655 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:00:31.247709 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1851b60c-0231-42ff-ab45-72d094278655 service nova] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:00:31.267237 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 13:00:31.276593 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57a03956-e870-4022-be8f-892a206cd8c2 admin admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:00:31.283567 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9544a71-b3db-45b7-bdb3-5926b58e2c33 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:00:31.283847 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 889/1775] 10.222.0.30 () {60 vars in 1365 bytes} [Mon May 23 13:00:30 2022] GET /identity/v3/auth/tokens => generated 3240 bytes in 363 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) May 23 13:00:31.284863 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1851b60c-0231-42ff-ab45-72d094278655 service nova] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:00:31.284976 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9544a71-b3db-45b7-bdb3-5926b58e2c33 admin admin] REQUEST_METHOD: `POST` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:00:31.285724 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9544a71-b3db-45b7-bdb3-5926b58e2c33 admin admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:00:31.286738 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9544a71-b3db-45b7-bdb3-5926b58e2c33 admin admin] PATH_INFO: `/v3/projects` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:00:31.287748 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9544a71-b3db-45b7-bdb3-5926b58e2c33 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:00:31.290523 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9544a71-b3db-45b7-bdb3-5926b58e2c33 admin admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:00:31.292051 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.fernet_utils [None req-1851b60c-0231-42ff-ab45-72d094278655 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92057) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:00:31.330684 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1851b60c-0231-42ff-ab45-72d094278655 service nova] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:00:31.339505 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0a241de-cdae-4cd3-81a9-cb829116df5b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:00:31.340181 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0a241de-cdae-4cd3-81a9-cb829116df5b admin admin] REQUEST_METHOD: `POST` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:00:31.340514 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0a241de-cdae-4cd3-81a9-cb829116df5b admin admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:00:31.340846 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0a241de-cdae-4cd3-81a9-cb829116df5b admin admin] PATH_INFO: `/v3/projects` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:00:31.342140 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0a241de-cdae-4cd3-81a9-cb829116df5b admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:00:31.344338 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0a241de-cdae-4cd3-81a9-cb829116df5b admin admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:00:31.427068 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 887/1776] 10.222.0.30 () {60 vars in 1152 bytes} [Mon May 23 13:00:31 2022] POST /identity/v3/projects => generated 385 bytes in 301 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) May 23 13:00:31.428472 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55c95dc4-2fc4-4515-bbb0-512c68f69062 None None] REQUEST_METHOD: `POST` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:00:31.428784 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55c95dc4-2fc4-4515-bbb0-512c68f69062 None None] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:00:31.429043 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55c95dc4-2fc4-4515-bbb0-512c68f69062 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:00:31.452325 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.auth.core [None req-55c95dc4-2fc4-4515-bbb0-512c68f69062 None None] MFA Rules not processed for user `0f7119f8fa5c4445977671140ded91a4`. Rule list: `[]` (Enabled: `True`). {{(pid=92056) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 13:00:31.454155 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.fernet_utils [None req-55c95dc4-2fc4-4515-bbb0-512c68f69062 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92056) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:00:31.463763 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 890/1777] 10.222.0.30 () {60 vars in 1152 bytes} [Mon May 23 13:00:31 2022] POST /identity/v3/projects => generated 405 bytes in 179 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) May 23 13:00:31.465176 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0a241de-cdae-4cd3-81a9-cb829116df5b admin admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:00:31.539160 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0a241de-cdae-4cd3-81a9-cb829116df5b admin admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:00:31.543276 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 888/1778] 10.222.0.30 () {58 vars in 957 bytes} [Mon May 23 13:00:31 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 113 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) May 23 13:00:31.543276 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55c95dc4-2fc4-4515-bbb0-512c68f69062 None None] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:00:31.551073 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95160131-d02e-4546-927e-879f881ba1b4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:00:31.551546 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55c95dc4-2fc4-4515-bbb0-512c68f69062 None None] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:00:31.551760 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95160131-d02e-4546-927e-879f881ba1b4 admin admin] REQUEST_METHOD: `POST` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:00:31.552051 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95160131-d02e-4546-927e-879f881ba1b4 admin admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:00:31.552521 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95160131-d02e-4546-927e-879f881ba1b4 admin admin] PATH_INFO: `/v3/users` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:00:31.553223 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95160131-d02e-4546-927e-879f881ba1b4 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:00:31.555642 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95160131-d02e-4546-927e-879f881ba1b4 admin admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:00:31.557754 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d123954b-a1ab-4e7b-abf7-775057e0ba67 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:00:31.558549 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d123954b-a1ab-4e7b-abf7-775057e0ba67 admin admin] REQUEST_METHOD: `POST` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:00:31.559376 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d123954b-a1ab-4e7b-abf7-775057e0ba67 admin admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:00:31.559878 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d123954b-a1ab-4e7b-abf7-775057e0ba67 admin admin] PATH_INFO: `/v3/users` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:00:31.560687 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d123954b-a1ab-4e7b-abf7-775057e0ba67 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:00:31.564802 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d123954b-a1ab-4e7b-abf7-775057e0ba67 admin admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:00:31.702909 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 891/1779] 10.222.0.30 () {60 vars in 1143 bytes} [Mon May 23 13:00:31 2022] POST /identity/v3/users => generated 293 bytes in 234 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) May 23 13:00:31.702909 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d0207df-731e-4400-bb52-bd530011321f None None] REQUEST_METHOD: `POST` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:00:31.702909 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d0207df-731e-4400-bb52-bd530011321f None None] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:00:31.702909 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d0207df-731e-4400-bb52-bd530011321f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:00:31.719340 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.auth.core [None req-5d0207df-731e-4400-bb52-bd530011321f None None] MFA Rules not processed for user `0f7119f8fa5c4445977671140ded91a4`. Rule list: `[]` (Enabled: `True`). {{(pid=92057) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 13:00:31.720545 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.fernet_utils [None req-5d0207df-731e-4400-bb52-bd530011321f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92057) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:00:31.730039 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 892/1780] 10.222.0.30 () {58 vars in 957 bytes} [Mon May 23 13:00:31 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) May 23 13:00:31.731633 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d0207df-731e-4400-bb52-bd530011321f None None] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:00:31.740349 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d0207df-731e-4400-bb52-bd530011321f None None] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:00:31.746541 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91456357-55a4-43c7-a452-42dd78c4bd0c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:00:31.747209 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91456357-55a4-43c7-a452-42dd78c4bd0c admin admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:00:31.747494 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91456357-55a4-43c7-a452-42dd78c4bd0c admin admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:00:31.747775 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91456357-55a4-43c7-a452-42dd78c4bd0c admin admin] PATH_INFO: `/v3/roles` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:00:31.748510 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-91456357-55a4-43c7-a452-42dd78c4bd0c admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:00:31.750602 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-91456357-55a4-43c7-a452-42dd78c4bd0c admin admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:00:31.753979 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=91,keystone:DELETE=7,keystone:INSERT=2 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 13:00:31.760076 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 889/1781] 10.222.0.30 () {60 vars in 1143 bytes} [Mon May 23 13:00:31 2022] POST /identity/v3/users => generated 309 bytes in 219 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) May 23 13:00:31.763996 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d123954b-a1ab-4e7b-abf7-775057e0ba67 admin admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:00:31.767484 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 893/1782] 10.222.0.30 () {58 vars in 1121 bytes} [Mon May 23 13:00:31 2022] GET /identity/v3/roles => generated 1420 bytes in 37 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) May 23 13:00:31.773760 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91456357-55a4-43c7-a452-42dd78c4bd0c admin admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:00:31.775361 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d123954b-a1ab-4e7b-abf7-775057e0ba67 admin admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:00:31.780042 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea479e7d-ebdf-4566-b39f-3ddd53322201 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:00:31.780713 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea479e7d-ebdf-4566-b39f-3ddd53322201 admin admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:00:31.780997 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea479e7d-ebdf-4566-b39f-3ddd53322201 admin admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:00:31.781274 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea479e7d-ebdf-4566-b39f-3ddd53322201 admin admin] PATH_INFO: `/v3/domains` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:00:31.782037 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea479e7d-ebdf-4566-b39f-3ddd53322201 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=92056) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} May 23 13:00:31.782342 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea479e7d-ebdf-4566-b39f-3ddd53322201 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:00:31.783947 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea479e7d-ebdf-4566-b39f-3ddd53322201 admin admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:00:31.785706 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91456357-55a4-43c7-a452-42dd78c4bd0c admin admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:00:31.791351 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6d781bb-ca39-40cf-ae11-09ed04e17f52 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:00:31.791996 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6d781bb-ca39-40cf-ae11-09ed04e17f52 admin admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:00:31.792288 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6d781bb-ca39-40cf-ae11-09ed04e17f52 admin admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:00:31.792569 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6d781bb-ca39-40cf-ae11-09ed04e17f52 admin admin] PATH_INFO: `/v3/roles` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:00:31.793595 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6d781bb-ca39-40cf-ae11-09ed04e17f52 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:00:31.794974 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6d781bb-ca39-40cf-ae11-09ed04e17f52 admin admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:00:31.811751 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 894/1783] 10.222.0.30 () {58 vars in 1121 bytes} [Mon May 23 13:00:31 2022] GET /identity/v3/roles => generated 1420 bytes in 41 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) May 23 13:00:31.811751 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6d781bb-ca39-40cf-ae11-09ed04e17f52 admin admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:00:31.815582 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 890/1784] 10.222.0.30 () {58 vars in 1152 bytes} [Mon May 23 13:00:31 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 55 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) May 23 13:00:31.816550 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea479e7d-ebdf-4566-b39f-3ddd53322201 admin admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:00:31.823389 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6d781bb-ca39-40cf-ae11-09ed04e17f52 admin admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:00:31.827332 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea479e7d-ebdf-4566-b39f-3ddd53322201 admin admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:00:31.829230 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-301f65df-4145-40cf-85b2-9aeb01be6729 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:00:31.830060 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-301f65df-4145-40cf-85b2-9aeb01be6729 admin admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:00:31.830679 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-301f65df-4145-40cf-85b2-9aeb01be6729 admin admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:00:31.830679 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-301f65df-4145-40cf-85b2-9aeb01be6729 admin admin] PATH_INFO: `/v3/roles` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:00:31.831560 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-301f65df-4145-40cf-85b2-9aeb01be6729 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:00:31.832226 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f29f5800-a01c-499b-a109-6c0ba9763ab4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:00:31.833117 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f29f5800-a01c-499b-a109-6c0ba9763ab4 admin admin] REQUEST_METHOD: `PUT` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:00:31.833345 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-301f65df-4145-40cf-85b2-9aeb01be6729 admin admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:00:31.833528 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f29f5800-a01c-499b-a109-6c0ba9763ab4 admin admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:00:31.833866 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f29f5800-a01c-499b-a109-6c0ba9763ab4 admin admin] PATH_INFO: `/v3/projects/dea2ad48ebf846b7ada5b6e1b69c7467/users/78921079f3b04f61a23d24810a00cf66/roles/bc3c3c9d654940cbaa5177a6be23d1bc` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:00:31.845055 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=100 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 13:00:31.849436 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 895/1785] 10.222.0.30 () {58 vars in 1121 bytes} [Mon May 23 13:00:31 2022] GET /identity/v3/roles => generated 1420 bytes in 40 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) May 23 13:00:31.850453 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-301f65df-4145-40cf-85b2-9aeb01be6729 admin admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:00:31.852620 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f29f5800-a01c-499b-a109-6c0ba9763ab4 admin admin] RBAC: Authorizing `identity:create_grant(project_id=dea2ad48ebf846b7ada5b6e1b69c7467, user_id=78921079f3b04f61a23d24810a00cf66, role_id=bc3c3c9d654940cbaa5177a6be23d1bc)` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:00:31.857440 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f29f5800-a01c-499b-a109-6c0ba9763ab4 admin admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:00:31.867876 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-301f65df-4145-40cf-85b2-9aeb01be6729 admin admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:00:31.867876 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39c4d6b6-4716-4746-81c8-333fe0bc4062 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:00:31.868359 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39c4d6b6-4716-4746-81c8-333fe0bc4062 admin admin] REQUEST_METHOD: `POST` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:00:31.868692 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39c4d6b6-4716-4746-81c8-333fe0bc4062 admin admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:00:31.868996 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39c4d6b6-4716-4746-81c8-333fe0bc4062 admin admin] PATH_INFO: `/v3/projects` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:00:31.869729 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39c4d6b6-4716-4746-81c8-333fe0bc4062 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:00:31.871686 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39c4d6b6-4716-4746-81c8-333fe0bc4062 admin admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:00:31.960269 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 891/1786] 10.222.0.30 () {60 vars in 1482 bytes} [Mon May 23 13:00:31 2022] PUT /identity/v3/projects/dea2ad48ebf846b7ada5b6e1b69c7467/users/78921079f3b04f61a23d24810a00cf66/roles/bc3c3c9d654940cbaa5177a6be23d1bc => generated 0 bytes in 144 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) May 23 13:00:31.961744 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f29f5800-a01c-499b-a109-6c0ba9763ab4 admin admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:00:31.989111 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 896/1787] 10.222.0.30 () {60 vars in 1152 bytes} [Mon May 23 13:00:31 2022] POST /identity/v3/projects => generated 373 bytes in 136 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) May 23 13:00:31.989111 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39c4d6b6-4716-4746-81c8-333fe0bc4062 admin admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:00:32.026350 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f29f5800-a01c-499b-a109-6c0ba9763ab4 admin admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:00:32.049335 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39c4d6b6-4716-4746-81c8-333fe0bc4062 admin admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:00:32.054149 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56b9d075-218b-427f-94d2-eae1811f83ea admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:00:32.054822 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56b9d075-218b-427f-94d2-eae1811f83ea admin admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:00:32.055111 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56b9d075-218b-427f-94d2-eae1811f83ea admin admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:00:32.055395 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56b9d075-218b-427f-94d2-eae1811f83ea admin admin] PATH_INFO: `/v3/roles` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:00:32.056166 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56b9d075-218b-427f-94d2-eae1811f83ea admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:00:32.057805 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56b9d075-218b-427f-94d2-eae1811f83ea admin admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:00:32.069964 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 897/1788] 10.222.0.30 () {58 vars in 1121 bytes} [Mon May 23 13:00:31 2022] GET /identity/v3/roles => generated 1420 bytes in 83 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) May 23 13:00:32.070855 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56b9d075-218b-427f-94d2-eae1811f83ea admin admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:00:32.082359 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52914046-5c02-4235-bfc1-51af799504ef admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:00:32.083196 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52914046-5c02-4235-bfc1-51af799504ef admin admin] REQUEST_METHOD: `PUT` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:00:32.083481 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52914046-5c02-4235-bfc1-51af799504ef admin admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:00:32.083759 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52914046-5c02-4235-bfc1-51af799504ef admin admin] PATH_INFO: `/v3/projects/003c58a0a07c4c349e26c20f5f9a68cc/users/3981e49dab1c46669478d9e34cfb4e03/roles/80fb22507c9a4ff09d44d8f12f6fff1a` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:00:32.086507 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56b9d075-218b-427f-94d2-eae1811f83ea admin admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:00:32.089742 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 13:00:32.094691 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27e74350-cc31-4a83-90b1-18b193dea317 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:00:32.095367 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27e74350-cc31-4a83-90b1-18b193dea317 admin admin] REQUEST_METHOD: `POST` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:00:32.095652 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27e74350-cc31-4a83-90b1-18b193dea317 admin admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:00:32.096509 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27e74350-cc31-4a83-90b1-18b193dea317 admin admin] PATH_INFO: `/v3/users` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:00:32.096776 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-27e74350-cc31-4a83-90b1-18b193dea317 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:00:32.098880 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-27e74350-cc31-4a83-90b1-18b193dea317 admin admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:00:32.100122 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52914046-5c02-4235-bfc1-51af799504ef admin admin] RBAC: Authorizing `identity:create_grant(project_id=003c58a0a07c4c349e26c20f5f9a68cc, user_id=3981e49dab1c46669478d9e34cfb4e03, role_id=80fb22507c9a4ff09d44d8f12f6fff1a)` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:00:32.102888 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52914046-5c02-4235-bfc1-51af799504ef admin admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:00:32.191390 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 898/1789] 10.222.0.30 () {60 vars in 1143 bytes} [Mon May 23 13:00:32 2022] POST /identity/v3/users => generated 287 bytes in 121 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) May 23 13:00:32.192335 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 892/1790] 10.222.0.30 () {60 vars in 1482 bytes} [Mon May 23 13:00:31 2022] PUT /identity/v3/projects/003c58a0a07c4c349e26c20f5f9a68cc/users/3981e49dab1c46669478d9e34cfb4e03/roles/80fb22507c9a4ff09d44d8f12f6fff1a => generated 0 bytes in 231 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) May 23 13:00:32.193792 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27e74350-cc31-4a83-90b1-18b193dea317 admin admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:00:32.195699 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52914046-5c02-4235-bfc1-51af799504ef admin admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:00:32.206015 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27e74350-cc31-4a83-90b1-18b193dea317 admin admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:00:32.266679 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52914046-5c02-4235-bfc1-51af799504ef admin admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:00:32.271279 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6e66d74-ded4-484c-9cb9-9281dab5133f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:00:32.272233 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6e66d74-ded4-484c-9cb9-9281dab5133f admin admin] REQUEST_METHOD: `PUT` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:00:32.272640 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6e66d74-ded4-484c-9cb9-9281dab5133f admin admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:00:32.273043 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6e66d74-ded4-484c-9cb9-9281dab5133f admin admin] PATH_INFO: `/v3/projects/dea2ad48ebf846b7ada5b6e1b69c7467/users/78921079f3b04f61a23d24810a00cf66/roles/bc3c3c9d654940cbaa5177a6be23d1bc` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:00:32.275910 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6e66d74-ded4-484c-9cb9-9281dab5133f admin admin] RBAC: Authorizing `identity:create_grant(project_id=dea2ad48ebf846b7ada5b6e1b69c7467, user_id=78921079f3b04f61a23d24810a00cf66, role_id=bc3c3c9d654940cbaa5177a6be23d1bc)` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:00:32.279244 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6e66d74-ded4-484c-9cb9-9281dab5133f admin admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:00:32.285387 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf886d9d-4de1-49d7-95ee-2168bfa0dd77 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:00:32.286709 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf886d9d-4de1-49d7-95ee-2168bfa0dd77 admin admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:00:32.287566 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf886d9d-4de1-49d7-95ee-2168bfa0dd77 admin admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:00:32.288448 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf886d9d-4de1-49d7-95ee-2168bfa0dd77 admin admin] PATH_INFO: `/v3/roles` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:00:32.301745 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf886d9d-4de1-49d7-95ee-2168bfa0dd77 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:00:32.301745 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf886d9d-4de1-49d7-95ee-2168bfa0dd77 admin admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:00:32.302187 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 899/1791] 10.222.0.30 () {60 vars in 1482 bytes} [Mon May 23 13:00:32 2022] PUT /identity/v3/projects/dea2ad48ebf846b7ada5b6e1b69c7467/users/78921079f3b04f61a23d24810a00cf66/roles/bc3c3c9d654940cbaa5177a6be23d1bc => generated 0 bytes in 107 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) May 23 13:00:32.303693 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95359486-6ff6-4967-a3fa-d7d8ae6f2714 None None] REQUEST_METHOD: `POST` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:00:32.306481 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95359486-6ff6-4967-a3fa-d7d8ae6f2714 None None] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:00:32.307488 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95359486-6ff6-4967-a3fa-d7d8ae6f2714 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:00:32.316792 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 893/1792] 10.222.0.30 () {58 vars in 1121 bytes} [Mon May 23 13:00:32 2022] GET /identity/v3/roles => generated 1420 bytes in 121 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) May 23 13:00:32.318305 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57d3afd6-abdb-4ded-91d1-c16a0b907f94 None None] REQUEST_METHOD: `POST` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:00:32.318684 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57d3afd6-abdb-4ded-91d1-c16a0b907f94 None None] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:00:32.319055 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57d3afd6-abdb-4ded-91d1-c16a0b907f94 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:00:32.351722 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.auth.core [None req-57d3afd6-abdb-4ded-91d1-c16a0b907f94 None None] MFA Rules not processed for user `78921079f3b04f61a23d24810a00cf66`. Rule list: `[]` (Enabled: `True`). {{(pid=92056) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 13:00:32.357842 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.fernet_utils [None req-57d3afd6-abdb-4ded-91d1-c16a0b907f94 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92056) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:00:32.358436 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.auth.core [None req-95359486-6ff6-4967-a3fa-d7d8ae6f2714 None None] MFA Rules not processed for user `3981e49dab1c46669478d9e34cfb4e03`. Rule list: `[]` (Enabled: `True`). {{(pid=92057) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 13:00:32.358436 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.fernet_utils [None req-95359486-6ff6-4967-a3fa-d7d8ae6f2714 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92057) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:00:32.382981 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 13:00:32.454974 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 894/1793] 10.222.0.30 () {58 vars in 957 bytes} [Mon May 23 13:00:32 2022] POST /identity/v3/auth/tokens => generated 3339 bytes in 138 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) May 23 13:00:32.455995 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57d3afd6-abdb-4ded-91d1-c16a0b907f94 None None] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:00:32.461334 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 900/1794] 10.222.0.30 () {58 vars in 957 bytes} [Mon May 23 13:00:32 2022] POST /identity/v3/auth/tokens => generated 3252 bytes in 160 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) May 23 13:00:32.468184 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95359486-6ff6-4967-a3fa-d7d8ae6f2714 None None] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:00:32.482034 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 13:00:32.539555 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57d3afd6-abdb-4ded-91d1-c16a0b907f94 None None] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:00:32.545724 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3752a988-7d40-45f0-bdc9-b45e2cbde341 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:00:32.546659 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3752a988-7d40-45f0-bdc9-b45e2cbde341 admin admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:00:32.547303 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3752a988-7d40-45f0-bdc9-b45e2cbde341 admin admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:00:32.547303 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3752a988-7d40-45f0-bdc9-b45e2cbde341 admin admin] PATH_INFO: `/v3/roles` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:00:32.548364 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3752a988-7d40-45f0-bdc9-b45e2cbde341 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:00:32.550181 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3752a988-7d40-45f0-bdc9-b45e2cbde341 admin admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:00:32.571263 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 895/1795] 10.222.0.30 () {58 vars in 1121 bytes} [Mon May 23 13:00:32 2022] GET /identity/v3/roles => generated 1420 bytes in 116 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) May 23 13:00:32.572438 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3752a988-7d40-45f0-bdc9-b45e2cbde341 admin admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:00:32.579617 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.fernet_utils [None req-3752a988-7d40-45f0-bdc9-b45e2cbde341 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92056) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:00:32.620958 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=100 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 13:00:32.655423 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3752a988-7d40-45f0-bdc9-b45e2cbde341 admin admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:00:32.660033 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a09a0d1-1002-4253-bfb5-693b2520d716 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1666097f8ed545c6b0e4809c93e4775a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9c389ae65aaa4542bb5a5c4fd0bacf5b', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:00:32.660867 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a09a0d1-1002-4253-bfb5-693b2520d716 service cinder] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:00:32.661148 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a09a0d1-1002-4253-bfb5-693b2520d716 service cinder] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:00:32.661423 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a09a0d1-1002-4253-bfb5-693b2520d716 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:00:32.671497 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a09a0d1-1002-4253-bfb5-693b2520d716 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:00:32.672932 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a09a0d1-1002-4253-bfb5-693b2520d716 service cinder] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:00:32.685030 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 896/1796] 10.222.0.30 () {60 vars in 1365 bytes} [Mon May 23 13:00:32 2022] GET /identity/v3/auth/tokens => generated 3339 bytes in 113 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) May 23 13:00:32.686122 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a09a0d1-1002-4253-bfb5-693b2520d716 service cinder] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:00:32.696092 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95359486-6ff6-4967-a3fa-d7d8ae6f2714 None None] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:00:32.702671 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e34ac59-9680-4817-9a87-655cc5b1d802 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:00:32.702895 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a09a0d1-1002-4253-bfb5-693b2520d716 service cinder] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:00:32.703406 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e34ac59-9680-4817-9a87-655cc5b1d802 admin admin] REQUEST_METHOD: `POST` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:00:32.704083 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e34ac59-9680-4817-9a87-655cc5b1d802 admin admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:00:32.704503 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e34ac59-9680-4817-9a87-655cc5b1d802 admin admin] PATH_INFO: `/v3/projects` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:00:32.705339 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e34ac59-9680-4817-9a87-655cc5b1d802 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:00:32.707721 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e34ac59-9680-4817-9a87-655cc5b1d802 admin admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:00:32.707850 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93b4cd94-2c2c-4944-aaef-4468c55e54ff admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:00:32.708746 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93b4cd94-2c2c-4944-aaef-4468c55e54ff admin admin] REQUEST_METHOD: `PUT` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:00:32.709022 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93b4cd94-2c2c-4944-aaef-4468c55e54ff admin admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:00:32.709300 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93b4cd94-2c2c-4944-aaef-4468c55e54ff admin admin] PATH_INFO: `/v3/projects/e0304d5b3acb4f47b03e048b9b82dc3d/users/352102dbcd4d4614be1c236842b63d96/roles/80fb22507c9a4ff09d44d8f12f6fff1a` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:00:32.735513 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93b4cd94-2c2c-4944-aaef-4468c55e54ff admin admin] RBAC: Authorizing `identity:create_grant(project_id=e0304d5b3acb4f47b03e048b9b82dc3d, user_id=352102dbcd4d4614be1c236842b63d96, role_id=80fb22507c9a4ff09d44d8f12f6fff1a)` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:00:32.739400 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93b4cd94-2c2c-4944-aaef-4468c55e54ff admin admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:00:33.361736 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 897/1797] 10.222.0.30 () {60 vars in 1482 bytes} [Mon May 23 13:00:32 2022] PUT /identity/v3/projects/e0304d5b3acb4f47b03e048b9b82dc3d/users/352102dbcd4d4614be1c236842b63d96/roles/80fb22507c9a4ff09d44d8f12f6fff1a => generated 0 bytes in 587 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) May 23 13:00:33.362329 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 901/1798] 10.222.0.30 () {60 vars in 1152 bytes} [Mon May 23 13:00:32 2022] POST /identity/v3/projects => generated 383 bytes in 805 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) May 23 13:00:33.362426 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93b4cd94-2c2c-4944-aaef-4468c55e54ff admin admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:00:33.362426 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93b4cd94-2c2c-4944-aaef-4468c55e54ff admin admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:00:33.371938 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e34ac59-9680-4817-9a87-655cc5b1d802 admin admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:00:33.410730 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b61331a1-0a28-42ae-b947-051d9bb13390 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:00:33.411611 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b61331a1-0a28-42ae-b947-051d9bb13390 admin admin] REQUEST_METHOD: `DELETE` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:00:33.411905 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b61331a1-0a28-42ae-b947-051d9bb13390 admin admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:00:33.412193 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b61331a1-0a28-42ae-b947-051d9bb13390 admin admin] PATH_INFO: `/v3/users/f92e0a102a794ded99da9c9385266db1` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:00:33.413400 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b61331a1-0a28-42ae-b947-051d9bb13390 admin admin] RBAC: Authorizing `identity:delete_user(user_id=f92e0a102a794ded99da9c9385266db1)` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:00:33.415513 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b61331a1-0a28-42ae-b947-051d9bb13390 admin admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:00:33.459687 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e34ac59-9680-4817-9a87-655cc5b1d802 admin admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:00:33.464750 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f63ad5b4-e8fa-4e75-bfac-9776979dcbbc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:00:33.465304 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f63ad5b4-e8fa-4e75-bfac-9776979dcbbc admin admin] REQUEST_METHOD: `POST` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:00:33.465592 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f63ad5b4-e8fa-4e75-bfac-9776979dcbbc admin admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:00:33.465871 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f63ad5b4-e8fa-4e75-bfac-9776979dcbbc admin admin] PATH_INFO: `/v3/users` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:00:33.466597 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f63ad5b4-e8fa-4e75-bfac-9776979dcbbc admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:00:33.468520 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f63ad5b4-e8fa-4e75-bfac-9776979dcbbc admin admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:00:33.810822 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 902/1799] 10.222.0.30 () {60 vars in 1143 bytes} [Mon May 23 13:00:33 2022] POST /identity/v3/users => generated 298 bytes in 439 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) May 23 13:00:33.812094 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e688bed9-40a3-4a75-a072-30b3f314b2c0 None None] REQUEST_METHOD: `POST` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:00:33.812477 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e688bed9-40a3-4a75-a072-30b3f314b2c0 None None] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:00:33.812816 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e688bed9-40a3-4a75-a072-30b3f314b2c0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:00:33.863412 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.auth.core [None req-e688bed9-40a3-4a75-a072-30b3f314b2c0 None None] MFA Rules not processed for user `352102dbcd4d4614be1c236842b63d96`. Rule list: `[]` (Enabled: `True`). {{(pid=92057) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 13:00:33.864765 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.fernet_utils [None req-e688bed9-40a3-4a75-a072-30b3f314b2c0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92057) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:00:33.929487 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-b61331a1-0a28-42ae-b947-051d9bb13390 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'f92e0a102a794ded99da9c9385266db1'} {{(pid=92056) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:00:33.930919 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-b61331a1-0a28-42ae-b947-051d9bb13390 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'f92e0a102a794ded99da9c9385266db1'} {{(pid=92056) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:00:33.935977 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=90,keystone:INSERT=2,keystone:DELETE=8 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 13:00:33.959655 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 903/1800] 10.222.0.30 () {58 vars in 957 bytes} [Mon May 23 13:00:33 2022] POST /identity/v3/auth/tokens => generated 3240 bytes in 149 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) May 23 13:00:33.960419 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e688bed9-40a3-4a75-a072-30b3f314b2c0 None None] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:00:33.961714 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.fernet_utils [None req-e688bed9-40a3-4a75-a072-30b3f314b2c0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92057) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:00:33.976228 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e688bed9-40a3-4a75-a072-30b3f314b2c0 None None] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:00:33.981755 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2fb81308-aef2-4de5-9225-b0a24e5783b9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:00:33.982554 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fb81308-aef2-4de5-9225-b0a24e5783b9 admin admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:00:33.982844 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fb81308-aef2-4de5-9225-b0a24e5783b9 admin admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:00:33.983128 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fb81308-aef2-4de5-9225-b0a24e5783b9 admin admin] PATH_INFO: `/v3/roles` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:00:33.983855 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2fb81308-aef2-4de5-9225-b0a24e5783b9 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:00:33.985468 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2fb81308-aef2-4de5-9225-b0a24e5783b9 admin admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:00:33.997815 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 904/1801] 10.222.0.30 () {58 vars in 1121 bytes} [Mon May 23 13:00:33 2022] GET /identity/v3/roles => generated 1420 bytes in 38 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) May 23 13:00:34.005775 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2fb81308-aef2-4de5-9225-b0a24e5783b9 admin admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:00:34.005775 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.fernet_utils [None req-2fb81308-aef2-4de5-9225-b0a24e5783b9 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92057) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:00:34.011661 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 13:00:34.027192 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-b61331a1-0a28-42ae-b947-051d9bb13390 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'f92e0a102a794ded99da9c9385266db1'} {{(pid=92056) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:00:34.049493 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-b61331a1-0a28-42ae-b947-051d9bb13390 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'f92e0a102a794ded99da9c9385266db1'} {{(pid=92056) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:00:34.057353 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-b61331a1-0a28-42ae-b947-051d9bb13390 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'f92e0a102a794ded99da9c9385266db1'} {{(pid=92056) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:00:34.077622 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 898/1802] 10.222.0.30 () {58 vars in 1223 bytes} [Mon May 23 13:00:33 2022] DELETE /identity/v3/users/f92e0a102a794ded99da9c9385266db1 => generated 0 bytes in 803 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) May 23 13:00:34.082770 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b61331a1-0a28-42ae-b947-051d9bb13390 admin admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:00:34.101678 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2fb81308-aef2-4de5-9225-b0a24e5783b9 admin admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:00:34.154981 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-155df7c9-b1ec-417d-9a41-871f51524474 service glance] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '5cea0feed4254ec085854986f99acf30', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9c389ae65aaa4542bb5a5c4fd0bacf5b', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:00:34.155680 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-155df7c9-b1ec-417d-9a41-871f51524474 service glance] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:00:34.155983 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-155df7c9-b1ec-417d-9a41-871f51524474 service glance] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:00:34.156249 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-155df7c9-b1ec-417d-9a41-871f51524474 service glance] PATH_INFO: `/v3/auth/tokens` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:00:34.159054 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b61331a1-0a28-42ae-b947-051d9bb13390 admin admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:00:34.164545 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf5c9ecd-47ee-4591-a4c7-0c0fbc459319 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:00:34.165582 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf5c9ecd-47ee-4591-a4c7-0c0fbc459319 admin admin] REQUEST_METHOD: `PUT` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:00:34.165977 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf5c9ecd-47ee-4591-a4c7-0c0fbc459319 admin admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:00:34.166320 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf5c9ecd-47ee-4591-a4c7-0c0fbc459319 admin admin] PATH_INFO: `/v3/projects/d28f430177ad4819a02bc61616ff5fc1/users/4b1c8706939346ee842d6d6c893ba333/roles/bc3c3c9d654940cbaa5177a6be23d1bc` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:00:34.183850 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf5c9ecd-47ee-4591-a4c7-0c0fbc459319 admin admin] RBAC: Authorizing `identity:create_grant(project_id=d28f430177ad4819a02bc61616ff5fc1, user_id=4b1c8706939346ee842d6d6c893ba333, role_id=bc3c3c9d654940cbaa5177a6be23d1bc)` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:00:34.186985 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf5c9ecd-47ee-4591-a4c7-0c0fbc459319 admin admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:00:34.219727 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-155df7c9-b1ec-417d-9a41-871f51524474 service glance] RBAC: Authorizing `identity:validate_token()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:00:34.221274 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-155df7c9-b1ec-417d-9a41-871f51524474 service glance] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:00:34.241601 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 905/1803] 10.222.0.30 () {60 vars in 1365 bytes} [Mon May 23 13:00:34 2022] GET /identity/v3/auth/tokens => generated 3240 bytes in 239 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) May 23 13:00:34.242768 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f4cb247-26df-45cc-a231-c3708a44cccd None None] REQUEST_METHOD: `POST` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:00:34.243128 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f4cb247-26df-45cc-a231-c3708a44cccd None None] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:00:34.243557 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f4cb247-26df-45cc-a231-c3708a44cccd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:00:34.272207 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.auth.core [None req-3f4cb247-26df-45cc-a231-c3708a44cccd None None] MFA Rules not processed for user `0f7119f8fa5c4445977671140ded91a4`. Rule list: `[]` (Enabled: `True`). {{(pid=92057) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 13:00:34.274411 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.fernet_utils [None req-3f4cb247-26df-45cc-a231-c3708a44cccd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92057) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:00:34.281510 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 899/1804] 10.222.0.30 () {60 vars in 1482 bytes} [Mon May 23 13:00:34 2022] PUT /identity/v3/projects/d28f430177ad4819a02bc61616ff5fc1/users/4b1c8706939346ee842d6d6c893ba333/roles/bc3c3c9d654940cbaa5177a6be23d1bc => generated 0 bytes in 200 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) May 23 13:00:34.282738 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf5c9ecd-47ee-4591-a4c7-0c0fbc459319 admin admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:00:34.371096 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 906/1805] 10.222.0.30 () {58 vars in 957 bytes} [Mon May 23 13:00:34 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 126 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) May 23 13:00:34.385271 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f4cb247-26df-45cc-a231-c3708a44cccd None None] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:00:34.388461 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.fernet_utils [None req-3f4cb247-26df-45cc-a231-c3708a44cccd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92057) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:00:34.393044 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf5c9ecd-47ee-4591-a4c7-0c0fbc459319 admin admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:00:34.400131 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4484422-024a-4655-8ae2-35bdf1941838 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:00:34.400921 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4484422-024a-4655-8ae2-35bdf1941838 admin admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:00:34.401953 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4484422-024a-4655-8ae2-35bdf1941838 admin admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:00:34.402689 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4484422-024a-4655-8ae2-35bdf1941838 admin admin] PATH_INFO: `/v3/roles` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:00:34.403621 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4484422-024a-4655-8ae2-35bdf1941838 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:00:34.406359 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4484422-024a-4655-8ae2-35bdf1941838 admin admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:00:34.416968 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=100 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 13:00:34.426199 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 900/1806] 10.222.0.30 () {58 vars in 1121 bytes} [Mon May 23 13:00:34 2022] GET /identity/v3/roles => generated 1420 bytes in 144 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) May 23 13:00:34.429665 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4484422-024a-4655-8ae2-35bdf1941838 admin admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:00:34.441690 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4484422-024a-4655-8ae2-35bdf1941838 admin admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:00:34.449644 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38d01acc-5102-4d54-9090-98741f01068c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:00:34.450850 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38d01acc-5102-4d54-9090-98741f01068c admin admin] REQUEST_METHOD: `PUT` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:00:34.451198 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38d01acc-5102-4d54-9090-98741f01068c admin admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:00:34.451551 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38d01acc-5102-4d54-9090-98741f01068c admin admin] PATH_INFO: `/v3/projects/d28f430177ad4819a02bc61616ff5fc1/users/4b1c8706939346ee842d6d6c893ba333/roles/bc3c3c9d654940cbaa5177a6be23d1bc` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:00:34.454605 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38d01acc-5102-4d54-9090-98741f01068c admin admin] RBAC: Authorizing `identity:create_grant(project_id=d28f430177ad4819a02bc61616ff5fc1, user_id=4b1c8706939346ee842d6d6c893ba333, role_id=bc3c3c9d654940cbaa5177a6be23d1bc)` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:00:34.458711 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38d01acc-5102-4d54-9090-98741f01068c admin admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:00:34.469632 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f4cb247-26df-45cc-a231-c3708a44cccd None None] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:00:34.472856 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 901/1807] 10.222.0.30 () {60 vars in 1482 bytes} [Mon May 23 13:00:34 2022] PUT /identity/v3/projects/d28f430177ad4819a02bc61616ff5fc1/users/4b1c8706939346ee842d6d6c893ba333/roles/bc3c3c9d654940cbaa5177a6be23d1bc => generated 0 bytes in 46 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) May 23 13:00:34.487821 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48a8ebf4-9e67-4d4e-915a-171641a1c794 None None] REQUEST_METHOD: `POST` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:00:34.492721 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48a8ebf4-9e67-4d4e-915a-171641a1c794 None None] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:00:34.492721 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48a8ebf4-9e67-4d4e-915a-171641a1c794 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:00:34.531012 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.auth.core [None req-48a8ebf4-9e67-4d4e-915a-171641a1c794 None None] MFA Rules not processed for user `4b1c8706939346ee842d6d6c893ba333`. Rule list: `[]` (Enabled: `True`). {{(pid=92056) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 13:00:34.532861 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.fernet_utils [None req-48a8ebf4-9e67-4d4e-915a-171641a1c794 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92056) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:00:34.552241 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-010135c1-f395-402a-8bbd-4bc8ce0c24d1 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2eb83ca38e79449095f3a2f947bbb06a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9c389ae65aaa4542bb5a5c4fd0bacf5b', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:00:34.564839 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-010135c1-f395-402a-8bbd-4bc8ce0c24d1 service neutron] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:00:34.565142 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-010135c1-f395-402a-8bbd-4bc8ce0c24d1 service neutron] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:00:34.565416 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-010135c1-f395-402a-8bbd-4bc8ce0c24d1 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:00:34.576552 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=96,keystone:DELETE=2,keystone:INSERT=2 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 13:00:34.655998 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 902/1808] 10.222.0.30 () {58 vars in 957 bytes} [Mon May 23 13:00:34 2022] POST /identity/v3/auth/tokens => generated 3317 bytes in 170 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) May 23 13:00:34.670289 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48a8ebf4-9e67-4d4e-915a-171641a1c794 None None] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:00:34.675510 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.fernet_utils [None req-48a8ebf4-9e67-4d4e-915a-171641a1c794 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92056) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:00:34.676491 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-010135c1-f395-402a-8bbd-4bc8ce0c24d1 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:00:34.677977 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-010135c1-f395-402a-8bbd-4bc8ce0c24d1 service neutron] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:00:34.700041 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 907/1809] 10.222.0.30 () {60 vars in 1365 bytes} [Mon May 23 13:00:34 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 312 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) May 23 13:00:34.754463 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48a8ebf4-9e67-4d4e-915a-171641a1c794 None None] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:00:34.760533 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1aff320f-ba03-4315-a783-bacac6aef819 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1666097f8ed545c6b0e4809c93e4775a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9c389ae65aaa4542bb5a5c4fd0bacf5b', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:00:34.761653 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1aff320f-ba03-4315-a783-bacac6aef819 service cinder] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:00:34.762387 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1aff320f-ba03-4315-a783-bacac6aef819 service cinder] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:00:34.763131 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1aff320f-ba03-4315-a783-bacac6aef819 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:00:34.774990 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1aff320f-ba03-4315-a783-bacac6aef819 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:00:34.777187 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1aff320f-ba03-4315-a783-bacac6aef819 service cinder] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:00:34.788857 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 903/1810] 10.222.0.30 () {60 vars in 1365 bytes} [Mon May 23 13:00:34 2022] GET /identity/v3/auth/tokens => generated 3317 bytes in 119 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) May 23 13:00:35.050656 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-010135c1-f395-402a-8bbd-4bc8ce0c24d1 service neutron] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:00:35.066822 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-010135c1-f395-402a-8bbd-4bc8ce0c24d1 service neutron] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:00:35.081803 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5c39c30-5d92-463d-9109-5c391d56a6c2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:00:35.082748 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5c39c30-5d92-463d-9109-5c391d56a6c2 admin admin] REQUEST_METHOD: `DELETE` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:00:35.086798 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5c39c30-5d92-463d-9109-5c391d56a6c2 admin admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:00:35.086798 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5c39c30-5d92-463d-9109-5c391d56a6c2 admin admin] PATH_INFO: `/v3/users/3981e49dab1c46669478d9e34cfb4e03` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:00:35.091291 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a5c39c30-5d92-463d-9109-5c391d56a6c2 admin admin] RBAC: Authorizing `identity:delete_user(user_id=3981e49dab1c46669478d9e34cfb4e03)` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:00:35.093718 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a5c39c30-5d92-463d-9109-5c391d56a6c2 admin admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:00:35.188274 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1aff320f-ba03-4315-a783-bacac6aef819 service cinder] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:00:35.190123 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.fernet_utils [None req-1aff320f-ba03-4315-a783-bacac6aef819 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92056) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:00:35.213125 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1aff320f-ba03-4315-a783-bacac6aef819 service cinder] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:00:35.218807 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd0fdde3-707e-46a9-b611-0b9e52f72539 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:00:35.219709 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd0fdde3-707e-46a9-b611-0b9e52f72539 admin admin] REQUEST_METHOD: `DELETE` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:00:35.219991 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd0fdde3-707e-46a9-b611-0b9e52f72539 admin admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:00:35.220293 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd0fdde3-707e-46a9-b611-0b9e52f72539 admin admin] PATH_INFO: `/v3/projects/67bf533d20784c63aaff424cec03b974` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:00:35.221665 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd0fdde3-707e-46a9-b611-0b9e52f72539 admin admin] RBAC: Authorizing `identity:delete_project(project_id=67bf533d20784c63aaff424cec03b974)` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:00:35.223734 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd0fdde3-707e-46a9-b611-0b9e52f72539 admin admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:00:35.312486 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [None req-a5c39c30-5d92-463d-9109-5c391d56a6c2 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '3981e49dab1c46669478d9e34cfb4e03'} {{(pid=92057) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:00:35.357578 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-bd0fdde3-707e-46a9-b611-0b9e52f72539 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '67bf533d20784c63aaff424cec03b974', 'request_id': 'req-bd0fdde3-707e-46a9-b611-0b9e52f72539'} {{(pid=92056) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:00:35.405917 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [None req-a5c39c30-5d92-463d-9109-5c391d56a6c2 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '3981e49dab1c46669478d9e34cfb4e03'} {{(pid=92057) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:00:35.414647 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [None req-a5c39c30-5d92-463d-9109-5c391d56a6c2 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '3981e49dab1c46669478d9e34cfb4e03'} {{(pid=92057) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:00:35.425997 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [None req-a5c39c30-5d92-463d-9109-5c391d56a6c2 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '3981e49dab1c46669478d9e34cfb4e03'} {{(pid=92057) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:00:35.429011 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=90,keystone:DELETE=9,keystone:INSERT=1 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 13:00:35.432541 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [None req-a5c39c30-5d92-463d-9109-5c391d56a6c2 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '3981e49dab1c46669478d9e34cfb4e03'} {{(pid=92057) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:00:35.441279 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 908/1811] 10.222.0.30 () {58 vars in 1223 bytes} [Mon May 23 13:00:35 2022] DELETE /identity/v3/users/3981e49dab1c46669478d9e34cfb4e03 => generated 0 bytes in 391 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) May 23 13:00:35.445769 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a3dd160-2ec5-483c-9723-4ec386d75c4e None None] REQUEST_METHOD: `POST` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:00:35.446103 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a3dd160-2ec5-483c-9723-4ec386d75c4e None None] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:00:35.446393 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a3dd160-2ec5-483c-9723-4ec386d75c4e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:00:35.465174 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.auth.core [None req-1a3dd160-2ec5-483c-9723-4ec386d75c4e None None] MFA Rules not processed for user `0f7119f8fa5c4445977671140ded91a4`. Rule list: `[]` (Enabled: `True`). {{(pid=92057) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 13:00:35.466466 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.fernet_utils [None req-1a3dd160-2ec5-483c-9723-4ec386d75c4e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92057) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:00:35.508577 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-bd0fdde3-707e-46a9-b611-0b9e52f72539 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '67bf533d20784c63aaff424cec03b974', 'request_id': 'req-bd0fdde3-707e-46a9-b611-0b9e52f72539'} {{(pid=92056) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:00:35.517399 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-bd0fdde3-707e-46a9-b611-0b9e52f72539 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '67bf533d20784c63aaff424cec03b974', 'request_id': 'req-bd0fdde3-707e-46a9-b611-0b9e52f72539'} {{(pid=92056) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:00:35.530612 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 909/1812] 10.222.0.30 () {58 vars in 957 bytes} [Mon May 23 13:00:35 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) May 23 13:00:35.540226 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a3dd160-2ec5-483c-9723-4ec386d75c4e None None] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:00:35.542641 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.fernet_utils [None req-1a3dd160-2ec5-483c-9723-4ec386d75c4e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92057) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:00:35.548797 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-bd0fdde3-707e-46a9-b611-0b9e52f72539 admin admin] The token cache is being invalidate because project 67bf533d20784c63aaff424cec03b974 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=92056) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} May 23 13:00:35.549196 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-bd0fdde3-707e-46a9-b611-0b9e52f72539 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=92056) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:00:35.549969 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-bd0fdde3-707e-46a9-b611-0b9e52f72539 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=92056) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:00:35.551994 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 904/1813] 10.222.0.30 () {58 vars in 1232 bytes} [Mon May 23 13:00:35 2022] DELETE /identity/v3/projects/67bf533d20784c63aaff424cec03b974 => generated 0 bytes in 365 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) May 23 13:00:35.555816 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd0fdde3-707e-46a9-b611-0b9e52f72539 admin admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:00:35.558002 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.fernet_utils [None req-bd0fdde3-707e-46a9-b611-0b9e52f72539 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92056) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:00:35.623207 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a3dd160-2ec5-483c-9723-4ec386d75c4e None None] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:00:35.631408 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68849139-ca1f-4f2b-9ee1-689d695600b5 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2eb83ca38e79449095f3a2f947bbb06a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9c389ae65aaa4542bb5a5c4fd0bacf5b', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:00:35.632109 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68849139-ca1f-4f2b-9ee1-689d695600b5 service neutron] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:00:35.632389 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68849139-ca1f-4f2b-9ee1-689d695600b5 service neutron] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:00:35.632661 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68849139-ca1f-4f2b-9ee1-689d695600b5 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:00:35.634534 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.fernet_utils [None req-68849139-ca1f-4f2b-9ee1-689d695600b5 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92057) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:00:35.637424 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 13:00:35.653345 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68849139-ca1f-4f2b-9ee1-689d695600b5 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:00:35.654906 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68849139-ca1f-4f2b-9ee1-689d695600b5 service neutron] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:00:35.658392 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd0fdde3-707e-46a9-b611-0b9e52f72539 admin admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:00:35.663640 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3ba83e5-f9f9-41a2-b3af-fa3daa93da24 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:00:35.664521 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3ba83e5-f9f9-41a2-b3af-fa3daa93da24 admin admin] REQUEST_METHOD: `DELETE` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:00:35.664799 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3ba83e5-f9f9-41a2-b3af-fa3daa93da24 admin admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:00:35.665066 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3ba83e5-f9f9-41a2-b3af-fa3daa93da24 admin admin] PATH_INFO: `/v3/users/78921079f3b04f61a23d24810a00cf66` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:00:35.666298 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3ba83e5-f9f9-41a2-b3af-fa3daa93da24 admin admin] RBAC: Authorizing `identity:delete_user(user_id=78921079f3b04f61a23d24810a00cf66)` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:00:35.668390 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3ba83e5-f9f9-41a2-b3af-fa3daa93da24 admin admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:00:35.669163 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 910/1814] 10.222.0.30 () {60 vars in 1365 bytes} [Mon May 23 13:00:35 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 129 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) May 23 13:00:35.892766 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-f3ba83e5-f9f9-41a2-b3af-fa3daa93da24 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '78921079f3b04f61a23d24810a00cf66'} {{(pid=92056) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:00:35.893913 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-f3ba83e5-f9f9-41a2-b3af-fa3daa93da24 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '78921079f3b04f61a23d24810a00cf66'} {{(pid=92056) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:00:35.990298 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-f3ba83e5-f9f9-41a2-b3af-fa3daa93da24 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '78921079f3b04f61a23d24810a00cf66'} {{(pid=92056) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:00:36.007318 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-f3ba83e5-f9f9-41a2-b3af-fa3daa93da24 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '78921079f3b04f61a23d24810a00cf66'} {{(pid=92056) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:00:36.016436 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-f3ba83e5-f9f9-41a2-b3af-fa3daa93da24 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '78921079f3b04f61a23d24810a00cf66'} {{(pid=92056) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:00:36.022866 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 905/1815] 10.222.0.30 () {58 vars in 1223 bytes} [Mon May 23 13:00:35 2022] DELETE /identity/v3/users/78921079f3b04f61a23d24810a00cf66 => generated 0 bytes in 467 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) May 23 13:00:36.175049 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68849139-ca1f-4f2b-9ee1-689d695600b5 service neutron] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:00:36.179881 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.fernet_utils [None req-68849139-ca1f-4f2b-9ee1-689d695600b5 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92057) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:00:36.201101 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68849139-ca1f-4f2b-9ee1-689d695600b5 service neutron] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:00:36.268644 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3ba83e5-f9f9-41a2-b3af-fa3daa93da24 admin admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:00:36.268835 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.fernet_utils [None req-f3ba83e5-f9f9-41a2-b3af-fa3daa93da24 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92056) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:00:36.300489 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef82f0df-4239-4638-8137-371651cee4d8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:00:36.300905 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef82f0df-4239-4638-8137-371651cee4d8 admin admin] REQUEST_METHOD: `DELETE` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:00:36.301197 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef82f0df-4239-4638-8137-371651cee4d8 admin admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:00:36.301470 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef82f0df-4239-4638-8137-371651cee4d8 admin admin] PATH_INFO: `/v3/projects/003c58a0a07c4c349e26c20f5f9a68cc` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:00:36.302784 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef82f0df-4239-4638-8137-371651cee4d8 admin admin] RBAC: Authorizing `identity:delete_project(project_id=003c58a0a07c4c349e26c20f5f9a68cc)` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:00:36.304818 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef82f0df-4239-4638-8137-371651cee4d8 admin admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:00:36.373950 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3ba83e5-f9f9-41a2-b3af-fa3daa93da24 admin admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:00:36.383280 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21264a99-c631-4cc1-8675-c83eaa8cd2c7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:00:36.385490 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21264a99-c631-4cc1-8675-c83eaa8cd2c7 admin admin] REQUEST_METHOD: `DELETE` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:00:36.385490 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21264a99-c631-4cc1-8675-c83eaa8cd2c7 admin admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:00:36.385490 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21264a99-c631-4cc1-8675-c83eaa8cd2c7 admin admin] PATH_INFO: `/v3/users/7adac7fe4243402aa25085cacfe2b29d` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:00:36.386281 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-21264a99-c631-4cc1-8675-c83eaa8cd2c7 admin admin] RBAC: Authorizing `identity:delete_user(user_id=7adac7fe4243402aa25085cacfe2b29d)` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:00:36.388805 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-21264a99-c631-4cc1-8675-c83eaa8cd2c7 admin admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:00:36.466275 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:DELETE=4,keystone:SELECT=96 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 13:00:36.486353 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [None req-ef82f0df-4239-4638-8137-371651cee4d8 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '003c58a0a07c4c349e26c20f5f9a68cc', 'request_id': 'req-ef82f0df-4239-4638-8137-371651cee4d8'} {{(pid=92057) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:00:36.497325 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [None req-ef82f0df-4239-4638-8137-371651cee4d8 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '003c58a0a07c4c349e26c20f5f9a68cc', 'request_id': 'req-ef82f0df-4239-4638-8137-371651cee4d8'} {{(pid=92057) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:00:36.603009 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-21264a99-c631-4cc1-8675-c83eaa8cd2c7 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '7adac7fe4243402aa25085cacfe2b29d'} {{(pid=92056) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:00:36.603741 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-21264a99-c631-4cc1-8675-c83eaa8cd2c7 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '7adac7fe4243402aa25085cacfe2b29d'} {{(pid=92056) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:00:36.613739 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [None req-ef82f0df-4239-4638-8137-371651cee4d8 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '003c58a0a07c4c349e26c20f5f9a68cc', 'request_id': 'req-ef82f0df-4239-4638-8137-371651cee4d8'} {{(pid=92057) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:00:36.626438 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [None req-ef82f0df-4239-4638-8137-371651cee4d8 admin admin] The token cache is being invalidate because project 003c58a0a07c4c349e26c20f5f9a68cc was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=92057) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} May 23 13:00:36.626810 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [None req-ef82f0df-4239-4638-8137-371651cee4d8 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=92057) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:00:36.627763 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [None req-ef82f0df-4239-4638-8137-371651cee4d8 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=92057) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:00:36.632865 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 911/1816] 10.222.0.30 () {58 vars in 1232 bytes} [Mon May 23 13:00:36 2022] DELETE /identity/v3/projects/003c58a0a07c4c349e26c20f5f9a68cc => generated 0 bytes in 455 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) May 23 13:00:36.633800 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef82f0df-4239-4638-8137-371651cee4d8 admin admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:00:36.635724 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.fernet_utils [None req-ef82f0df-4239-4638-8137-371651cee4d8 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92057) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:00:36.700311 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-21264a99-c631-4cc1-8675-c83eaa8cd2c7 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '7adac7fe4243402aa25085cacfe2b29d'} {{(pid=92056) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:00:36.714169 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef82f0df-4239-4638-8137-371651cee4d8 admin admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:00:36.717067 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=79,keystone:DELETE=19,keystone:INSERT=2 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 13:00:36.719627 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75442599-0c27-4b23-82b1-859543383bf0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:00:36.720362 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-21264a99-c631-4cc1-8675-c83eaa8cd2c7 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '7adac7fe4243402aa25085cacfe2b29d'} {{(pid=92056) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:00:36.720468 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75442599-0c27-4b23-82b1-859543383bf0 admin admin] REQUEST_METHOD: `DELETE` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:00:36.720799 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75442599-0c27-4b23-82b1-859543383bf0 admin admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:00:36.721018 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75442599-0c27-4b23-82b1-859543383bf0 admin admin] PATH_INFO: `/v3/users/4b1c8706939346ee842d6d6c893ba333` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:00:36.722422 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75442599-0c27-4b23-82b1-859543383bf0 admin admin] RBAC: Authorizing `identity:delete_user(user_id=4b1c8706939346ee842d6d6c893ba333)` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:00:36.725051 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75442599-0c27-4b23-82b1-859543383bf0 admin admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:00:36.734069 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-21264a99-c631-4cc1-8675-c83eaa8cd2c7 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '7adac7fe4243402aa25085cacfe2b29d'} {{(pid=92056) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:00:36.742803 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 906/1817] 10.222.0.30 () {58 vars in 1223 bytes} [Mon May 23 13:00:36 2022] DELETE /identity/v3/users/7adac7fe4243402aa25085cacfe2b29d => generated 0 bytes in 476 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) May 23 13:00:36.744010 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21264a99-c631-4cc1-8675-c83eaa8cd2c7 admin admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:00:36.746331 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.fernet_utils [None req-21264a99-c631-4cc1-8675-c83eaa8cd2c7 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92056) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:00:36.826359 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21264a99-c631-4cc1-8675-c83eaa8cd2c7 admin admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:00:36.834548 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c3e4731-c123-4a1c-9209-777eb694220c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:00:36.835472 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c3e4731-c123-4a1c-9209-777eb694220c admin admin] REQUEST_METHOD: `DELETE` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:00:36.835750 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c3e4731-c123-4a1c-9209-777eb694220c admin admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:00:36.836025 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c3e4731-c123-4a1c-9209-777eb694220c admin admin] PATH_INFO: `/v3/projects/dea2ad48ebf846b7ada5b6e1b69c7467` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:00:36.837451 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c3e4731-c123-4a1c-9209-777eb694220c admin admin] RBAC: Authorizing `identity:delete_project(project_id=dea2ad48ebf846b7ada5b6e1b69c7467)` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:00:36.840255 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c3e4731-c123-4a1c-9209-777eb694220c admin admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:00:36.953630 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [None req-75442599-0c27-4b23-82b1-859543383bf0 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '4b1c8706939346ee842d6d6c893ba333'} {{(pid=92057) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:00:36.991944 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-5c3e4731-c123-4a1c-9209-777eb694220c admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'dea2ad48ebf846b7ada5b6e1b69c7467', 'request_id': 'req-5c3e4731-c123-4a1c-9209-777eb694220c'} {{(pid=92056) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:00:37.045544 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [None req-75442599-0c27-4b23-82b1-859543383bf0 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '4b1c8706939346ee842d6d6c893ba333'} {{(pid=92057) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:00:37.055141 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [None req-75442599-0c27-4b23-82b1-859543383bf0 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '4b1c8706939346ee842d6d6c893ba333'} {{(pid=92057) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:00:37.055141 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [None req-75442599-0c27-4b23-82b1-859543383bf0 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '4b1c8706939346ee842d6d6c893ba333'} {{(pid=92057) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:00:37.067564 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [None req-75442599-0c27-4b23-82b1-859543383bf0 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '4b1c8706939346ee842d6d6c893ba333'} {{(pid=92057) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:00:37.074725 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 912/1818] 10.222.0.30 () {58 vars in 1223 bytes} [Mon May 23 13:00:36 2022] DELETE /identity/v3/users/4b1c8706939346ee842d6d6c893ba333 => generated 0 bytes in 441 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) May 23 13:00:37.076140 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b085e01-b5af-4157-b723-d148023e9dd3 None None] REQUEST_METHOD: `POST` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:00:37.076437 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b085e01-b5af-4157-b723-d148023e9dd3 None None] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:00:37.076722 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b085e01-b5af-4157-b723-d148023e9dd3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:00:37.097171 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.auth.core [None req-2b085e01-b5af-4157-b723-d148023e9dd3 None None] MFA Rules not processed for user `0f7119f8fa5c4445977671140ded91a4`. Rule list: `[]` (Enabled: `True`). {{(pid=92057) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 13:00:37.098457 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.fernet_utils [None req-2b085e01-b5af-4157-b723-d148023e9dd3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92057) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:00:37.102653 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-5c3e4731-c123-4a1c-9209-777eb694220c admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'dea2ad48ebf846b7ada5b6e1b69c7467', 'request_id': 'req-5c3e4731-c123-4a1c-9209-777eb694220c'} {{(pid=92056) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:00:37.110633 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-5c3e4731-c123-4a1c-9209-777eb694220c admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'dea2ad48ebf846b7ada5b6e1b69c7467', 'request_id': 'req-5c3e4731-c123-4a1c-9209-777eb694220c'} {{(pid=92056) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:00:37.123680 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-5c3e4731-c123-4a1c-9209-777eb694220c admin admin] The token cache is being invalidate because project dea2ad48ebf846b7ada5b6e1b69c7467 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=92056) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} May 23 13:00:37.124282 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-5c3e4731-c123-4a1c-9209-777eb694220c admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=92056) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:00:37.126621 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-5c3e4731-c123-4a1c-9209-777eb694220c admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=92056) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:00:37.129916 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 907/1819] 10.222.0.30 () {58 vars in 1232 bytes} [Mon May 23 13:00:36 2022] DELETE /identity/v3/projects/dea2ad48ebf846b7ada5b6e1b69c7467 => generated 0 bytes in 386 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) May 23 13:00:37.136740 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab2e215c-206a-4846-953a-f093fd115050 None None] REQUEST_METHOD: `POST` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:00:37.136958 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab2e215c-206a-4846-953a-f093fd115050 None None] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:00:37.137307 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab2e215c-206a-4846-953a-f093fd115050 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:00:37.154969 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=84,keystone:DELETE=14,keystone:INSERT=2 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 13:00:37.161236 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.auth.core [None req-ab2e215c-206a-4846-953a-f093fd115050 None None] MFA Rules not processed for user `0f7119f8fa5c4445977671140ded91a4`. Rule list: `[]` (Enabled: `True`). {{(pid=92056) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 13:00:37.162668 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.fernet_utils [None req-ab2e215c-206a-4846-953a-f093fd115050 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92056) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:00:37.181413 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 913/1820] 10.222.0.30 () {58 vars in 957 bytes} [Mon May 23 13:00:37 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 103 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) May 23 13:00:37.188298 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b085e01-b5af-4157-b723-d148023e9dd3 None None] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:00:37.189820 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.fernet_utils [None req-2b085e01-b5af-4157-b723-d148023e9dd3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92057) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:00:37.247804 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 908/1821] 10.222.0.30 () {58 vars in 957 bytes} [Mon May 23 13:00:37 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 114 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) May 23 13:00:37.257164 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94158926-eaf4-4b01-84e7-923278b6fd6c None None] REQUEST_METHOD: `POST` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:00:37.257672 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94158926-eaf4-4b01-84e7-923278b6fd6c None None] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:00:37.258197 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94158926-eaf4-4b01-84e7-923278b6fd6c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:00:37.264611 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b085e01-b5af-4157-b723-d148023e9dd3 None None] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:00:37.272209 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-267fe687-792b-4ed4-b1f5-3905841fe62b service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2eb83ca38e79449095f3a2f947bbb06a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9c389ae65aaa4542bb5a5c4fd0bacf5b', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:00:37.273980 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-267fe687-792b-4ed4-b1f5-3905841fe62b service neutron] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:00:37.275072 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-267fe687-792b-4ed4-b1f5-3905841fe62b service neutron] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:00:37.275773 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-267fe687-792b-4ed4-b1f5-3905841fe62b service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:00:37.279467 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.auth.core [None req-94158926-eaf4-4b01-84e7-923278b6fd6c None None] MFA Rules not processed for user `0f7119f8fa5c4445977671140ded91a4`. Rule list: `[]` (Enabled: `True`). {{(pid=92056) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 13:00:37.280724 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.fernet_utils [None req-94158926-eaf4-4b01-84e7-923278b6fd6c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92056) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:00:37.290270 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 909/1822] 10.222.0.30 () {58 vars in 957 bytes} [Mon May 23 13:00:37 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 34 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) May 23 13:00:37.294514 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94158926-eaf4-4b01-84e7-923278b6fd6c None None] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:00:37.294649 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-267fe687-792b-4ed4-b1f5-3905841fe62b service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:00:37.296796 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-267fe687-792b-4ed4-b1f5-3905841fe62b service neutron] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:00:37.313174 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94158926-eaf4-4b01-84e7-923278b6fd6c None None] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:00:37.313383 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 914/1823] 10.222.0.30 () {60 vars in 1365 bytes} [Mon May 23 13:00:37 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 124 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) May 23 13:00:37.315730 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1411dbc0-fcbb-4ae1-b608-868b5aa3a67b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:00:37.316545 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1411dbc0-fcbb-4ae1-b608-868b5aa3a67b admin admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:00:37.316835 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1411dbc0-fcbb-4ae1-b608-868b5aa3a67b admin admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:00:37.317134 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1411dbc0-fcbb-4ae1-b608-868b5aa3a67b admin admin] PATH_INFO: `/v3/domains` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:00:37.317932 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1411dbc0-fcbb-4ae1-b608-868b5aa3a67b admin admin] RBAC: Adding query filter params (name=Default) {{(pid=92056) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} May 23 13:00:37.318391 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1411dbc0-fcbb-4ae1-b608-868b5aa3a67b admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:00:37.322242 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1411dbc0-fcbb-4ae1-b608-868b5aa3a67b admin admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:00:37.335376 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 910/1824] 10.222.0.30 () {58 vars in 1152 bytes} [Mon May 23 13:00:37 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 41 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) May 23 13:00:37.340688 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-267fe687-792b-4ed4-b1f5-3905841fe62b service neutron] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:00:37.356573 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-267fe687-792b-4ed4-b1f5-3905841fe62b service neutron] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:00:37.362169 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7cc74ba8-5b1b-4662-b340-53e6567af880 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:00:37.365261 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7cc74ba8-5b1b-4662-b340-53e6567af880 admin admin] REQUEST_METHOD: `POST` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:00:37.365776 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7cc74ba8-5b1b-4662-b340-53e6567af880 admin admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:00:37.366239 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7cc74ba8-5b1b-4662-b340-53e6567af880 admin admin] PATH_INFO: `/v3/projects` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:00:37.367140 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7cc74ba8-5b1b-4662-b340-53e6567af880 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:00:37.369544 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7cc74ba8-5b1b-4662-b340-53e6567af880 admin admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:00:37.497537 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04e47649-72e8-4f56-9775-4a0320670dd9 None None] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:00:37.497889 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04e47649-72e8-4f56-9775-4a0320670dd9 None None] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:00:37.498159 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04e47649-72e8-4f56-9775-4a0320670dd9 None None] PATH_INFO: `/` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:00:37.502078 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 911/1825] 10.222.0.30 () {54 vars in 921 bytes} [Mon May 23 13:00:37 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) May 23 13:00:37.512211 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e7139da-a155-44a0-9371-104c172c3af7 None None] REQUEST_METHOD: `POST` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:00:37.512344 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 915/1826] 10.222.0.30 () {60 vars in 1152 bytes} [Mon May 23 13:00:37 2022] POST /identity/v3/projects => generated 365 bytes in 169 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) May 23 13:00:37.512701 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e7139da-a155-44a0-9371-104c172c3af7 None None] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:00:37.512701 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e7139da-a155-44a0-9371-104c172c3af7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:00:37.524773 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7cc74ba8-5b1b-4662-b340-53e6567af880 admin admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:00:37.547761 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.auth.core [None req-3e7139da-a155-44a0-9371-104c172c3af7 None None] MFA Rules not processed for user `cbe9931d94d44cc7ace0e83a7e5015fc`. Rule list: `[]` (Enabled: `True`). {{(pid=92056) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 13:00:37.549053 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.fernet_utils [None req-3e7139da-a155-44a0-9371-104c172c3af7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92056) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:00:37.618498 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7cc74ba8-5b1b-4662-b340-53e6567af880 admin admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:00:37.623457 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62ac993a-e3bd-4978-9da6-e579939b1f14 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:00:37.624139 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62ac993a-e3bd-4978-9da6-e579939b1f14 admin admin] REQUEST_METHOD: `POST` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:00:37.624424 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62ac993a-e3bd-4978-9da6-e579939b1f14 admin admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:00:37.624710 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62ac993a-e3bd-4978-9da6-e579939b1f14 admin admin] PATH_INFO: `/v3/users` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:00:37.625419 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-62ac993a-e3bd-4978-9da6-e579939b1f14 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:00:37.627427 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-62ac993a-e3bd-4978-9da6-e579939b1f14 admin admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:00:37.628659 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 912/1827] 10.222.0.30 () {58 vars in 1018 bytes} [Mon May 23 13:00:37 2022] POST /identity/v3/auth/tokens => generated 3362 bytes in 121 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) May 23 13:00:37.644302 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e7139da-a155-44a0-9371-104c172c3af7 None None] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:00:37.645652 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.fernet_utils [None req-3e7139da-a155-44a0-9371-104c172c3af7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92056) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:00:37.668890 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e7139da-a155-44a0-9371-104c172c3af7 None None] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:00:37.704156 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f2944c8-f385-457e-ad4c-2cb002a099a9 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cbe9931d94d44cc7ace0e83a7e5015fc', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9c389ae65aaa4542bb5a5c4fd0bacf5b', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'member', 'reader', 'service', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:00:37.705147 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f2944c8-f385-457e-ad4c-2cb002a099a9 service nova] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:00:37.705638 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f2944c8-f385-457e-ad4c-2cb002a099a9 service nova] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:00:37.706145 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f2944c8-f385-457e-ad4c-2cb002a099a9 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:00:37.736229 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f2944c8-f385-457e-ad4c-2cb002a099a9 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:00:37.738054 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f2944c8-f385-457e-ad4c-2cb002a099a9 service nova] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:00:37.753541 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 913/1828] 10.222.0.30 () {60 vars in 1365 bytes} [Mon May 23 13:00:37 2022] GET /identity/v3/auth/tokens => generated 3362 bytes in 109 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) May 23 13:00:37.754688 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f2944c8-f385-457e-ad4c-2cb002a099a9 service nova] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:00:37.756193 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.fernet_utils [None req-4f2944c8-f385-457e-ad4c-2cb002a099a9 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92056) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:00:37.762737 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 916/1829] 10.222.0.30 () {60 vars in 1143 bytes} [Mon May 23 13:00:37 2022] POST /identity/v3/users => generated 283 bytes in 244 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) May 23 13:00:37.764254 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62ac993a-e3bd-4978-9da6-e579939b1f14 admin admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:00:37.778131 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f2944c8-f385-457e-ad4c-2cb002a099a9 service nova] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:00:37.780338 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62ac993a-e3bd-4978-9da6-e579939b1f14 admin admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:00:37.792533 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9364bc24-da15-4b1a-84cb-c44afb54b4ac admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:00:37.793532 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9364bc24-da15-4b1a-84cb-c44afb54b4ac admin admin] REQUEST_METHOD: `DELETE` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:00:37.794687 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9364bc24-da15-4b1a-84cb-c44afb54b4ac admin admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:00:37.795023 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9364bc24-da15-4b1a-84cb-c44afb54b4ac admin admin] PATH_INFO: `/v3/projects/d28f430177ad4819a02bc61616ff5fc1` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:00:37.795660 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0f8cf64-ee13-4ee1-851e-e2031b4e7e60 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:00:37.796301 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9364bc24-da15-4b1a-84cb-c44afb54b4ac admin admin] RBAC: Authorizing `identity:delete_project(project_id=d28f430177ad4819a02bc61616ff5fc1)` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:00:37.796455 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0f8cf64-ee13-4ee1-851e-e2031b4e7e60 admin admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:00:37.798219 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0f8cf64-ee13-4ee1-851e-e2031b4e7e60 admin admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:00:37.798336 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9364bc24-da15-4b1a-84cb-c44afb54b4ac admin admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:00:37.798604 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0f8cf64-ee13-4ee1-851e-e2031b4e7e60 admin admin] PATH_INFO: `/v3/roles` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:00:37.799443 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0f8cf64-ee13-4ee1-851e-e2031b4e7e60 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:00:37.805386 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0f8cf64-ee13-4ee1-851e-e2031b4e7e60 admin admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:00:37.831516 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 917/1830] 10.222.0.30 () {58 vars in 1121 bytes} [Mon May 23 13:00:37 2022] GET /identity/v3/roles => generated 1420 bytes in 63 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) May 23 13:00:37.831516 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0f8cf64-ee13-4ee1-851e-e2031b4e7e60 admin admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:00:37.842658 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0f8cf64-ee13-4ee1-851e-e2031b4e7e60 admin admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:00:37.848208 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-796f5d4f-16be-4785-8593-f37a184dcf5a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:00:37.848958 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-796f5d4f-16be-4785-8593-f37a184dcf5a admin admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:00:37.854733 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-796f5d4f-16be-4785-8593-f37a184dcf5a admin admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:00:37.854733 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-796f5d4f-16be-4785-8593-f37a184dcf5a admin admin] PATH_INFO: `/v3/roles` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:00:37.854733 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-796f5d4f-16be-4785-8593-f37a184dcf5a admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:00:37.854733 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-796f5d4f-16be-4785-8593-f37a184dcf5a admin admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:00:37.868313 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 918/1831] 10.222.0.30 () {58 vars in 1121 bytes} [Mon May 23 13:00:37 2022] GET /identity/v3/roles => generated 1420 bytes in 36 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) May 23 13:00:37.869543 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-796f5d4f-16be-4785-8593-f37a184dcf5a admin admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:00:37.880215 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-796f5d4f-16be-4785-8593-f37a184dcf5a admin admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:00:37.893897 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7084288-53f1-4846-b053-cdd83b93516d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:00:37.893897 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7084288-53f1-4846-b053-cdd83b93516d admin admin] REQUEST_METHOD: `PUT` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:00:37.893897 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7084288-53f1-4846-b053-cdd83b93516d admin admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:00:37.893897 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7084288-53f1-4846-b053-cdd83b93516d admin admin] PATH_INFO: `/v3/projects/65d681ef71ba4ac4af80d806a3076cbd/users/4777a2b9585c446fad7630ad914716a0/roles/80fb22507c9a4ff09d44d8f12f6fff1a` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:00:37.910889 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7084288-53f1-4846-b053-cdd83b93516d admin admin] RBAC: Authorizing `identity:create_grant(project_id=65d681ef71ba4ac4af80d806a3076cbd, user_id=4777a2b9585c446fad7630ad914716a0, role_id=80fb22507c9a4ff09d44d8f12f6fff1a)` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:00:37.914180 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7084288-53f1-4846-b053-cdd83b93516d admin admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:00:38.062244 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 919/1832] 10.222.0.30 () {60 vars in 1482 bytes} [Mon May 23 13:00:37 2022] PUT /identity/v3/projects/65d681ef71ba4ac4af80d806a3076cbd/users/4777a2b9585c446fad7630ad914716a0/roles/80fb22507c9a4ff09d44d8f12f6fff1a => generated 0 bytes in 193 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) May 23 13:00:38.069220 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-9364bc24-da15-4b1a-84cb-c44afb54b4ac admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'd28f430177ad4819a02bc61616ff5fc1', 'request_id': 'req-9364bc24-da15-4b1a-84cb-c44afb54b4ac'} {{(pid=92056) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:00:38.073641 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8be8e9f-a3c3-4593-8065-fd23cd4503bd None None] REQUEST_METHOD: `POST` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:00:38.074990 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8be8e9f-a3c3-4593-8065-fd23cd4503bd None None] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:00:38.077241 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8be8e9f-a3c3-4593-8065-fd23cd4503bd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:00:38.081212 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:DELETE=8,keystone:SELECT=91,keystone:INSERT=1 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 13:00:38.106239 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.auth.core [None req-c8be8e9f-a3c3-4593-8065-fd23cd4503bd None None] MFA Rules not processed for user `4777a2b9585c446fad7630ad914716a0`. Rule list: `[]` (Enabled: `True`). {{(pid=92057) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 13:00:38.107487 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.fernet_utils [None req-c8be8e9f-a3c3-4593-8065-fd23cd4503bd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92057) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:00:38.177294 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-9364bc24-da15-4b1a-84cb-c44afb54b4ac admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'd28f430177ad4819a02bc61616ff5fc1', 'request_id': 'req-9364bc24-da15-4b1a-84cb-c44afb54b4ac'} {{(pid=92056) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:00:38.185365 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-9364bc24-da15-4b1a-84cb-c44afb54b4ac admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'd28f430177ad4819a02bc61616ff5fc1', 'request_id': 'req-9364bc24-da15-4b1a-84cb-c44afb54b4ac'} {{(pid=92056) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:00:38.195728 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 920/1833] 10.222.0.30 () {58 vars in 957 bytes} [Mon May 23 13:00:38 2022] POST /identity/v3/auth/tokens => generated 3232 bytes in 124 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) May 23 13:00:38.197012 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8be8e9f-a3c3-4593-8065-fd23cd4503bd None None] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:00:38.197454 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-9364bc24-da15-4b1a-84cb-c44afb54b4ac admin admin] The token cache is being invalidate because project d28f430177ad4819a02bc61616ff5fc1 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=92056) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} May 23 13:00:38.197814 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-9364bc24-da15-4b1a-84cb-c44afb54b4ac admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=92056) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:00:38.199303 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-9364bc24-da15-4b1a-84cb-c44afb54b4ac admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=92056) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:00:38.200357 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.fernet_utils [None req-c8be8e9f-a3c3-4593-8065-fd23cd4503bd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92057) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:00:38.201085 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 914/1834] 10.222.0.30 () {58 vars in 1232 bytes} [Mon May 23 13:00:37 2022] DELETE /identity/v3/projects/d28f430177ad4819a02bc61616ff5fc1 => generated 0 bytes in 447 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) May 23 13:00:38.201685 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9364bc24-da15-4b1a-84cb-c44afb54b4ac admin admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:00:38.203140 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.fernet_utils [None req-9364bc24-da15-4b1a-84cb-c44afb54b4ac admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92056) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:00:38.278690 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8be8e9f-a3c3-4593-8065-fd23cd4503bd None None] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:00:38.282071 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9364bc24-da15-4b1a-84cb-c44afb54b4ac admin admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:00:38.283458 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de094bcd-dacd-48fa-b06e-78c70e6eb376 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:00:38.284343 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de094bcd-dacd-48fa-b06e-78c70e6eb376 admin admin] REQUEST_METHOD: `DELETE` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:00:38.284648 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de094bcd-dacd-48fa-b06e-78c70e6eb376 admin admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:00:38.284929 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de094bcd-dacd-48fa-b06e-78c70e6eb376 admin admin] PATH_INFO: `/v3/projects/de4eed939156431fa99178951d5fcbec` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:00:38.286301 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de094bcd-dacd-48fa-b06e-78c70e6eb376 admin admin] RBAC: Authorizing `identity:delete_project(project_id=de4eed939156431fa99178951d5fcbec)` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:00:38.287821 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94cd7b99-b8a7-4833-90ee-5bd3cd0ad0da admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:00:38.288258 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de094bcd-dacd-48fa-b06e-78c70e6eb376 admin admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:00:38.289154 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94cd7b99-b8a7-4833-90ee-5bd3cd0ad0da admin admin] REQUEST_METHOD: `POST` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:00:38.289446 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94cd7b99-b8a7-4833-90ee-5bd3cd0ad0da admin admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:00:38.289731 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94cd7b99-b8a7-4833-90ee-5bd3cd0ad0da admin admin] PATH_INFO: `/v3/projects` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:00:38.290521 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94cd7b99-b8a7-4833-90ee-5bd3cd0ad0da admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:00:38.293386 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94cd7b99-b8a7-4833-90ee-5bd3cd0ad0da admin admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:00:38.401815 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 915/1835] 10.222.0.30 () {60 vars in 1152 bytes} [Mon May 23 13:00:38 2022] POST /identity/v3/projects => generated 365 bytes in 197 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) May 23 13:00:38.401815 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-054a8a7f-67d4-47e0-a822-ecec3296347a None None] REQUEST_METHOD: `POST` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:00:38.401815 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-054a8a7f-67d4-47e0-a822-ecec3296347a None None] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:00:38.401815 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-054a8a7f-67d4-47e0-a822-ecec3296347a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:00:38.422389 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.auth.core [None req-054a8a7f-67d4-47e0-a822-ecec3296347a None None] MFA Rules not processed for user `0f7119f8fa5c4445977671140ded91a4`. Rule list: `[]` (Enabled: `True`). {{(pid=92056) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 13:00:38.423668 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.fernet_utils [None req-054a8a7f-67d4-47e0-a822-ecec3296347a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92056) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:00:38.437661 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [None req-de094bcd-dacd-48fa-b06e-78c70e6eb376 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'de4eed939156431fa99178951d5fcbec', 'request_id': 'req-de094bcd-dacd-48fa-b06e-78c70e6eb376'} {{(pid=92057) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:00:38.443101 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [None req-de094bcd-dacd-48fa-b06e-78c70e6eb376 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'de4eed939156431fa99178951d5fcbec', 'request_id': 'req-de094bcd-dacd-48fa-b06e-78c70e6eb376'} {{(pid=92057) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:00:38.485469 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 916/1836] 10.222.0.30 () {58 vars in 957 bytes} [Mon May 23 13:00:38 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 87 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) May 23 13:00:38.486431 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-054a8a7f-67d4-47e0-a822-ecec3296347a None None] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:00:38.494086 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 13:00:38.495913 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-054a8a7f-67d4-47e0-a822-ecec3296347a None None] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:00:38.500549 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40de8d77-a45c-442f-987d-c38bc7eb9974 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:00:38.501206 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40de8d77-a45c-442f-987d-c38bc7eb9974 admin admin] REQUEST_METHOD: `POST` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:00:38.501494 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40de8d77-a45c-442f-987d-c38bc7eb9974 admin admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:00:38.501777 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40de8d77-a45c-442f-987d-c38bc7eb9974 admin admin] PATH_INFO: `/v3/users` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:00:38.502505 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40de8d77-a45c-442f-987d-c38bc7eb9974 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:00:38.504497 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40de8d77-a45c-442f-987d-c38bc7eb9974 admin admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:00:38.516374 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [None req-de094bcd-dacd-48fa-b06e-78c70e6eb376 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'de4eed939156431fa99178951d5fcbec', 'request_id': 'req-de094bcd-dacd-48fa-b06e-78c70e6eb376'} {{(pid=92057) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:00:38.528804 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [None req-de094bcd-dacd-48fa-b06e-78c70e6eb376 admin admin] The token cache is being invalidate because project de4eed939156431fa99178951d5fcbec was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=92057) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} May 23 13:00:38.529087 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [None req-de094bcd-dacd-48fa-b06e-78c70e6eb376 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=92057) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:00:38.529736 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [None req-de094bcd-dacd-48fa-b06e-78c70e6eb376 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=92057) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:00:38.531557 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 921/1837] 10.222.0.30 () {58 vars in 1232 bytes} [Mon May 23 13:00:38 2022] DELETE /identity/v3/projects/de4eed939156431fa99178951d5fcbec => generated 0 bytes in 335 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) May 23 13:00:38.532644 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d09df6ef-c46f-4f50-9801-f0a5bd24b373 None None] REQUEST_METHOD: `POST` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:00:38.532933 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d09df6ef-c46f-4f50-9801-f0a5bd24b373 None None] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:00:38.533212 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d09df6ef-c46f-4f50-9801-f0a5bd24b373 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:00:38.552830 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.auth.core [None req-d09df6ef-c46f-4f50-9801-f0a5bd24b373 None None] MFA Rules not processed for user `0f7119f8fa5c4445977671140ded91a4`. Rule list: `[]` (Enabled: `True`). {{(pid=92057) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 13:00:38.558703 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.fernet_utils [None req-d09df6ef-c46f-4f50-9801-f0a5bd24b373 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92057) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:00:38.567500 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 922/1838] 10.222.0.30 () {58 vars in 957 bytes} [Mon May 23 13:00:38 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 36 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) May 23 13:00:38.568926 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f419fbea-5761-489c-8c0b-9e149e71f10c None None] REQUEST_METHOD: `POST` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:00:38.572154 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f419fbea-5761-489c-8c0b-9e149e71f10c None None] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:00:38.572154 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f419fbea-5761-489c-8c0b-9e149e71f10c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:00:38.589632 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.auth.core [None req-f419fbea-5761-489c-8c0b-9e149e71f10c None None] MFA Rules not processed for user `0f7119f8fa5c4445977671140ded91a4`. Rule list: `[]` (Enabled: `True`). {{(pid=92057) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 13:00:38.590923 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.fernet_utils [None req-f419fbea-5761-489c-8c0b-9e149e71f10c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92057) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:00:38.598190 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 917/1839] 10.222.0.30 () {60 vars in 1143 bytes} [Mon May 23 13:00:38 2022] POST /identity/v3/users => generated 289 bytes in 112 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) May 23 13:00:38.601312 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 923/1840] 10.222.0.30 () {58 vars in 957 bytes} [Mon May 23 13:00:38 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) May 23 13:00:38.601436 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40de8d77-a45c-442f-987d-c38bc7eb9974 admin admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:00:38.608192 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ade155bc-dd63-453b-89dc-ca59ef7efba7 None None] REQUEST_METHOD: `POST` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:00:38.608484 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ade155bc-dd63-453b-89dc-ca59ef7efba7 None None] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:00:38.608756 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ade155bc-dd63-453b-89dc-ca59ef7efba7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:00:38.615892 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40de8d77-a45c-442f-987d-c38bc7eb9974 admin admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:00:38.622943 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c266561-aaf5-4dae-a83f-33b9fc3a244b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:00:38.623967 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c266561-aaf5-4dae-a83f-33b9fc3a244b admin admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:00:38.624258 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c266561-aaf5-4dae-a83f-33b9fc3a244b admin admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:00:38.624541 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c266561-aaf5-4dae-a83f-33b9fc3a244b admin admin] PATH_INFO: `/v3/domains` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:00:38.625280 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c266561-aaf5-4dae-a83f-33b9fc3a244b admin admin] RBAC: Adding query filter params (name=Default) {{(pid=92056) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} May 23 13:00:38.625589 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c266561-aaf5-4dae-a83f-33b9fc3a244b admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:00:38.627240 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c266561-aaf5-4dae-a83f-33b9fc3a244b admin admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:00:38.628945 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.auth.core [None req-ade155bc-dd63-453b-89dc-ca59ef7efba7 None None] MFA Rules not processed for user `0f7119f8fa5c4445977671140ded91a4`. Rule list: `[]` (Enabled: `True`). {{(pid=92057) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 13:00:38.630204 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.fernet_utils [None req-ade155bc-dd63-453b-89dc-ca59ef7efba7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92057) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:00:38.639278 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 924/1841] 10.222.0.30 () {58 vars in 957 bytes} [Mon May 23 13:00:38 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) May 23 13:00:38.640003 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ade155bc-dd63-453b-89dc-ca59ef7efba7 None None] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:00:38.641289 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.fernet_utils [None req-ade155bc-dd63-453b-89dc-ca59ef7efba7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92057) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:00:38.641796 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 918/1842] 10.222.0.30 () {58 vars in 1152 bytes} [Mon May 23 13:00:38 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 43 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) May 23 13:00:38.647183 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c266561-aaf5-4dae-a83f-33b9fc3a244b admin admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:00:38.656149 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ade155bc-dd63-453b-89dc-ca59ef7efba7 None None] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:00:38.658873 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c266561-aaf5-4dae-a83f-33b9fc3a244b admin admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:00:38.660833 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a2598db-4f21-4bc6-bcc8-3f47d610ee88 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:00:38.661588 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a2598db-4f21-4bc6-bcc8-3f47d610ee88 admin admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:00:38.661871 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a2598db-4f21-4bc6-bcc8-3f47d610ee88 admin admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:00:38.662200 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a2598db-4f21-4bc6-bcc8-3f47d610ee88 admin admin] PATH_INFO: `/v3/roles` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:00:38.663049 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a2598db-4f21-4bc6-bcc8-3f47d610ee88 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:00:38.663498 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cbadcb5c-af3e-4e79-ac5a-043a2a78d390 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:00:38.664146 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbadcb5c-af3e-4e79-ac5a-043a2a78d390 admin admin] REQUEST_METHOD: `POST` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:00:38.664457 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbadcb5c-af3e-4e79-ac5a-043a2a78d390 admin admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:00:38.664734 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbadcb5c-af3e-4e79-ac5a-043a2a78d390 admin admin] PATH_INFO: `/v3/projects` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:00:38.664976 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a2598db-4f21-4bc6-bcc8-3f47d610ee88 admin admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:00:38.665477 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cbadcb5c-af3e-4e79-ac5a-043a2a78d390 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:00:38.668120 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cbadcb5c-af3e-4e79-ac5a-043a2a78d390 admin admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:00:38.675586 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 925/1843] 10.222.0.30 () {58 vars in 1121 bytes} [Mon May 23 13:00:38 2022] GET /identity/v3/roles => generated 1420 bytes in 36 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) May 23 13:00:38.676817 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a2598db-4f21-4bc6-bcc8-3f47d610ee88 admin admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:00:38.687157 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a2598db-4f21-4bc6-bcc8-3f47d610ee88 admin admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:00:38.691940 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56bf71ef-92da-4357-b17d-368586b30ef0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:00:38.692638 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56bf71ef-92da-4357-b17d-368586b30ef0 admin admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:00:38.692919 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56bf71ef-92da-4357-b17d-368586b30ef0 admin admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:00:38.693213 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56bf71ef-92da-4357-b17d-368586b30ef0 admin admin] PATH_INFO: `/v3/domains` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:00:38.693950 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56bf71ef-92da-4357-b17d-368586b30ef0 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=92057) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} May 23 13:00:38.694261 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56bf71ef-92da-4357-b17d-368586b30ef0 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:00:38.695863 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56bf71ef-92da-4357-b17d-368586b30ef0 admin admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:00:38.708765 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 926/1844] 10.222.0.30 () {58 vars in 1152 bytes} [Mon May 23 13:00:38 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) May 23 13:00:38.709611 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56bf71ef-92da-4357-b17d-368586b30ef0 admin admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:00:38.720357 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56bf71ef-92da-4357-b17d-368586b30ef0 admin admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:00:38.724974 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af75d8ab-1c8e-4f3a-a1e6-bc2552a44cab admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:00:38.725844 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af75d8ab-1c8e-4f3a-a1e6-bc2552a44cab admin admin] REQUEST_METHOD: `PUT` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:00:38.726159 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af75d8ab-1c8e-4f3a-a1e6-bc2552a44cab admin admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:00:38.726461 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af75d8ab-1c8e-4f3a-a1e6-bc2552a44cab admin admin] PATH_INFO: `/v3/projects/b0e5a990b2584c4db0a06a4308eb0864/users/3321e1372371457c9f19cdde83bb1ccc/roles/bc3c3c9d654940cbaa5177a6be23d1bc` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:00:38.742152 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af75d8ab-1c8e-4f3a-a1e6-bc2552a44cab admin admin] RBAC: Authorizing `identity:create_grant(project_id=b0e5a990b2584c4db0a06a4308eb0864, user_id=3321e1372371457c9f19cdde83bb1ccc, role_id=bc3c3c9d654940cbaa5177a6be23d1bc)` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:00:38.745500 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af75d8ab-1c8e-4f3a-a1e6-bc2552a44cab admin admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:00:38.751240 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 919/1845] 10.222.0.30 () {60 vars in 1152 bytes} [Mon May 23 13:00:38 2022] POST /identity/v3/projects => generated 387 bytes in 104 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) May 23 13:00:38.752176 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cbadcb5c-af3e-4e79-ac5a-043a2a78d390 admin admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:00:38.836236 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 927/1846] 10.222.0.30 () {60 vars in 1482 bytes} [Mon May 23 13:00:38 2022] PUT /identity/v3/projects/b0e5a990b2584c4db0a06a4308eb0864/users/3321e1372371457c9f19cdde83bb1ccc/roles/bc3c3c9d654940cbaa5177a6be23d1bc => generated 0 bytes in 123 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) May 23 13:00:38.836236 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af75d8ab-1c8e-4f3a-a1e6-bc2552a44cab admin admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:00:38.861740 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cbadcb5c-af3e-4e79-ac5a-043a2a78d390 admin admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:00:38.914952 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af75d8ab-1c8e-4f3a-a1e6-bc2552a44cab admin admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:00:38.920085 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5aef6bd3-3706-42f0-8cb8-3e5038896cbb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:00:38.920844 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5aef6bd3-3706-42f0-8cb8-3e5038896cbb admin admin] REQUEST_METHOD: `POST` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:00:38.921126 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5aef6bd3-3706-42f0-8cb8-3e5038896cbb admin admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:00:38.921410 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5aef6bd3-3706-42f0-8cb8-3e5038896cbb admin admin] PATH_INFO: `/v3/users` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:00:38.922144 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5aef6bd3-3706-42f0-8cb8-3e5038896cbb admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:00:38.924152 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5aef6bd3-3706-42f0-8cb8-3e5038896cbb admin admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:00:38.942766 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-430b83b2-2b0c-4dd7-a37e-a0cbd6b3212b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:00:38.943573 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-430b83b2-2b0c-4dd7-a37e-a0cbd6b3212b admin admin] REQUEST_METHOD: `POST` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:00:38.943745 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-430b83b2-2b0c-4dd7-a37e-a0cbd6b3212b admin admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:00:38.944027 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-430b83b2-2b0c-4dd7-a37e-a0cbd6b3212b admin admin] PATH_INFO: `/v3/projects` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:00:38.944800 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-430b83b2-2b0c-4dd7-a37e-a0cbd6b3212b admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:00:38.946852 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-430b83b2-2b0c-4dd7-a37e-a0cbd6b3212b admin admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:00:39.033992 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 928/1847] 10.222.0.30 () {60 vars in 1143 bytes} [Mon May 23 13:00:38 2022] POST /identity/v3/users => generated 294 bytes in 198 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) May 23 13:00:39.033992 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5aef6bd3-3706-42f0-8cb8-3e5038896cbb admin admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:00:39.046528 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5aef6bd3-3706-42f0-8cb8-3e5038896cbb admin admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:00:39.051971 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f0d902a-d400-4a24-b5f5-f8d3ba3805e5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:00:39.052713 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f0d902a-d400-4a24-b5f5-f8d3ba3805e5 admin admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:00:39.053000 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f0d902a-d400-4a24-b5f5-f8d3ba3805e5 admin admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:00:39.053280 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f0d902a-d400-4a24-b5f5-f8d3ba3805e5 admin admin] PATH_INFO: `/v3/roles` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:00:39.054063 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f0d902a-d400-4a24-b5f5-f8d3ba3805e5 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:00:39.055689 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f0d902a-d400-4a24-b5f5-f8d3ba3805e5 admin admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:00:39.070326 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 929/1848] 10.222.0.30 () {58 vars in 1121 bytes} [Mon May 23 13:00:39 2022] GET /identity/v3/roles => generated 1420 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) May 23 13:00:39.070326 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f0d902a-d400-4a24-b5f5-f8d3ba3805e5 admin admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:00:39.080598 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f0d902a-d400-4a24-b5f5-f8d3ba3805e5 admin admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:00:39.085919 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc14ec9f-7fae-4450-85cd-3ed97ff81856 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:00:39.089981 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc14ec9f-7fae-4450-85cd-3ed97ff81856 admin admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:00:39.089981 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc14ec9f-7fae-4450-85cd-3ed97ff81856 admin admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:00:39.089981 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc14ec9f-7fae-4450-85cd-3ed97ff81856 admin admin] PATH_INFO: `/v3/roles` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:00:39.089981 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc14ec9f-7fae-4450-85cd-3ed97ff81856 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:00:39.091619 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc14ec9f-7fae-4450-85cd-3ed97ff81856 admin admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:00:39.093916 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 920/1849] 10.222.0.30 () {60 vars in 1152 bytes} [Mon May 23 13:00:38 2022] POST /identity/v3/projects => generated 387 bytes in 342 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) May 23 13:00:39.095123 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-430b83b2-2b0c-4dd7-a37e-a0cbd6b3212b admin admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:00:39.109994 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 930/1850] 10.222.0.30 () {58 vars in 1121 bytes} [Mon May 23 13:00:39 2022] GET /identity/v3/roles => generated 1420 bytes in 42 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) May 23 13:00:39.111243 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc14ec9f-7fae-4450-85cd-3ed97ff81856 admin admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:00:39.127457 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc14ec9f-7fae-4450-85cd-3ed97ff81856 admin admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:00:39.186029 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-430b83b2-2b0c-4dd7-a37e-a0cbd6b3212b admin admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:00:39.193013 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d8c5a96-a8c1-4347-9b3c-00ad108edbe2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:00:39.194124 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d8c5a96-a8c1-4347-9b3c-00ad108edbe2 admin admin] REQUEST_METHOD: `PUT` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:00:39.194463 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d8c5a96-a8c1-4347-9b3c-00ad108edbe2 admin admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:00:39.194747 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d8c5a96-a8c1-4347-9b3c-00ad108edbe2 admin admin] PATH_INFO: `/v3/projects/b0e5a990b2584c4db0a06a4308eb0864/users/3321e1372371457c9f19cdde83bb1ccc/roles/bc3c3c9d654940cbaa5177a6be23d1bc` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:00:39.197010 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d8c5a96-a8c1-4347-9b3c-00ad108edbe2 admin admin] RBAC: Authorizing `identity:create_grant(project_id=b0e5a990b2584c4db0a06a4308eb0864, user_id=3321e1372371457c9f19cdde83bb1ccc, role_id=bc3c3c9d654940cbaa5177a6be23d1bc)` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:00:39.199764 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d8c5a96-a8c1-4347-9b3c-00ad108edbe2 admin admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:00:39.217936 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 921/1851] 10.222.0.30 () {60 vars in 1482 bytes} [Mon May 23 13:00:39 2022] PUT /identity/v3/projects/b0e5a990b2584c4db0a06a4308eb0864/users/3321e1372371457c9f19cdde83bb1ccc/roles/bc3c3c9d654940cbaa5177a6be23d1bc => generated 0 bytes in 119 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) May 23 13:00:39.217936 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d8c5a96-a8c1-4347-9b3c-00ad108edbe2 admin admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:00:39.218231 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2039982-e4c3-40df-ac2e-19b264001efd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:00:39.218231 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2039982-e4c3-40df-ac2e-19b264001efd admin admin] REQUEST_METHOD: `POST` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:00:39.218231 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2039982-e4c3-40df-ac2e-19b264001efd admin admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:00:39.218231 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2039982-e4c3-40df-ac2e-19b264001efd admin admin] PATH_INFO: `/v3/users` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:00:39.218500 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2039982-e4c3-40df-ac2e-19b264001efd admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:00:39.220472 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2039982-e4c3-40df-ac2e-19b264001efd admin admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:00:39.316504 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d8c5a96-a8c1-4347-9b3c-00ad108edbe2 admin admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:00:39.322450 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35cf839a-f664-4002-9ae2-8a66702c1c19 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:00:39.323706 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35cf839a-f664-4002-9ae2-8a66702c1c19 admin admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:00:39.323985 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35cf839a-f664-4002-9ae2-8a66702c1c19 admin admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:00:39.324258 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35cf839a-f664-4002-9ae2-8a66702c1c19 admin admin] PATH_INFO: `/v3/roles` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:00:39.324996 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35cf839a-f664-4002-9ae2-8a66702c1c19 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:00:39.326955 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35cf839a-f664-4002-9ae2-8a66702c1c19 admin admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:00:39.346013 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 922/1852] 10.222.0.30 () {58 vars in 1121 bytes} [Mon May 23 13:00:39 2022] GET /identity/v3/roles => generated 1420 bytes in 128 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) May 23 13:00:39.346013 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9761225a-04d0-410d-befe-6d8db0bb6179 None None] REQUEST_METHOD: `POST` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:00:39.346013 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9761225a-04d0-410d-befe-6d8db0bb6179 None None] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:00:39.346013 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9761225a-04d0-410d-befe-6d8db0bb6179 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:00:39.379795 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.auth.core [None req-9761225a-04d0-410d-befe-6d8db0bb6179 None None] MFA Rules not processed for user `3321e1372371457c9f19cdde83bb1ccc`. Rule list: `[]` (Enabled: `True`). {{(pid=92056) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 13:00:39.381078 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.fernet_utils [None req-9761225a-04d0-410d-befe-6d8db0bb6179 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92056) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:00:39.382923 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 931/1853] 10.222.0.30 () {60 vars in 1143 bytes} [Mon May 23 13:00:39 2022] POST /identity/v3/users => generated 294 bytes in 272 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) May 23 13:00:39.384048 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2039982-e4c3-40df-ac2e-19b264001efd admin admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:00:39.404998 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2039982-e4c3-40df-ac2e-19b264001efd admin admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:00:39.412835 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-739563cf-634e-47e3-9583-223317230710 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:00:39.413968 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-739563cf-634e-47e3-9583-223317230710 admin admin] REQUEST_METHOD: `PUT` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:00:39.414390 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-739563cf-634e-47e3-9583-223317230710 admin admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:00:39.414767 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-739563cf-634e-47e3-9583-223317230710 admin admin] PATH_INFO: `/v3/projects/fbf2a131f924445c8c8651592f8db51e/users/d1086d05b1ed42b08853995f88f645bf/roles/80fb22507c9a4ff09d44d8f12f6fff1a` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:00:39.440774 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-739563cf-634e-47e3-9583-223317230710 admin admin] RBAC: Authorizing `identity:create_grant(project_id=fbf2a131f924445c8c8651592f8db51e, user_id=d1086d05b1ed42b08853995f88f645bf, role_id=80fb22507c9a4ff09d44d8f12f6fff1a)` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:00:39.443478 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-739563cf-634e-47e3-9583-223317230710 admin admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:00:39.483795 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 923/1854] 10.222.0.30 () {58 vars in 957 bytes} [Mon May 23 13:00:39 2022] POST /identity/v3/auth/tokens => generated 3299 bytes in 141 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) May 23 13:00:39.485021 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9761225a-04d0-410d-befe-6d8db0bb6179 None None] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:00:39.498479 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9761225a-04d0-410d-befe-6d8db0bb6179 None None] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:00:39.503142 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e111d3f-65f0-4915-ba2f-5b580e0b5a62 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:00:39.503835 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e111d3f-65f0-4915-ba2f-5b580e0b5a62 admin admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:00:39.504113 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e111d3f-65f0-4915-ba2f-5b580e0b5a62 admin admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:00:39.504398 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e111d3f-65f0-4915-ba2f-5b580e0b5a62 admin admin] PATH_INFO: `/v3/roles` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:00:39.505503 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e111d3f-65f0-4915-ba2f-5b580e0b5a62 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:00:39.506911 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e111d3f-65f0-4915-ba2f-5b580e0b5a62 admin admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:00:39.519499 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 924/1855] 10.222.0.30 () {58 vars in 1121 bytes} [Mon May 23 13:00:39 2022] GET /identity/v3/roles => generated 1420 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) May 23 13:00:39.520347 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e111d3f-65f0-4915-ba2f-5b580e0b5a62 admin admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:00:39.522111 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.fernet_utils [None req-7e111d3f-65f0-4915-ba2f-5b580e0b5a62 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92056) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:00:39.531035 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 13:00:39.540996 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 932/1856] 10.222.0.30 () {60 vars in 1482 bytes} [Mon May 23 13:00:39 2022] PUT /identity/v3/projects/fbf2a131f924445c8c8651592f8db51e/users/d1086d05b1ed42b08853995f88f645bf/roles/80fb22507c9a4ff09d44d8f12f6fff1a => generated 0 bytes in 157 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) May 23 13:00:39.541866 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-739563cf-634e-47e3-9583-223317230710 admin admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:00:39.593918 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e111d3f-65f0-4915-ba2f-5b580e0b5a62 admin admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:00:39.605933 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 13:00:39.615115 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-739563cf-634e-47e3-9583-223317230710 admin admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:00:39.625973 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eff3f40c-883b-463a-a552-b6a9a84108ea admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:00:39.626692 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eff3f40c-883b-463a-a552-b6a9a84108ea admin admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:00:39.627200 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eff3f40c-883b-463a-a552-b6a9a84108ea admin admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:00:39.627485 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eff3f40c-883b-463a-a552-b6a9a84108ea admin admin] PATH_INFO: `/v3/roles` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:00:39.628720 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eff3f40c-883b-463a-a552-b6a9a84108ea admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:00:39.630331 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eff3f40c-883b-463a-a552-b6a9a84108ea admin admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:00:39.643891 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 933/1857] 10.222.0.30 () {58 vars in 1121 bytes} [Mon May 23 13:00:39 2022] GET /identity/v3/roles => generated 1420 bytes in 102 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) May 23 13:00:39.645285 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-328e14d7-0c2b-4eb0-bc6f-f9d4d72e71e0 None None] REQUEST_METHOD: `POST` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:00:39.645585 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-328e14d7-0c2b-4eb0-bc6f-f9d4d72e71e0 None None] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:00:39.645870 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-328e14d7-0c2b-4eb0-bc6f-f9d4d72e71e0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:00:39.658575 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9450bab-bc1f-4905-ab6a-455530c41738 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1666097f8ed545c6b0e4809c93e4775a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9c389ae65aaa4542bb5a5c4fd0bacf5b', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:00:39.659212 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9450bab-bc1f-4905-ab6a-455530c41738 service cinder] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:00:39.659489 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9450bab-bc1f-4905-ab6a-455530c41738 service cinder] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:00:39.659768 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9450bab-bc1f-4905-ab6a-455530c41738 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:00:39.674451 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.auth.core [None req-328e14d7-0c2b-4eb0-bc6f-f9d4d72e71e0 None None] MFA Rules not processed for user `d1086d05b1ed42b08853995f88f645bf`. Rule list: `[]` (Enabled: `True`). {{(pid=92057) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 13:00:39.675588 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.fernet_utils [None req-328e14d7-0c2b-4eb0-bc6f-f9d4d72e71e0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92057) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:00:39.739577 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9450bab-bc1f-4905-ab6a-455530c41738 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:00:39.741074 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9450bab-bc1f-4905-ab6a-455530c41738 service cinder] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:00:39.753486 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 934/1858] 10.222.0.30 () {58 vars in 957 bytes} [Mon May 23 13:00:39 2022] POST /identity/v3/auth/tokens => generated 3254 bytes in 109 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) May 23 13:00:39.753863 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 925/1859] 10.222.0.30 () {60 vars in 1365 bytes} [Mon May 23 13:00:39 2022] GET /identity/v3/auth/tokens => generated 3299 bytes in 234 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) May 23 13:00:39.754366 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-328e14d7-0c2b-4eb0-bc6f-f9d4d72e71e0 None None] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:00:39.758818 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9450bab-bc1f-4905-ab6a-455530c41738 service cinder] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:00:39.766605 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-328e14d7-0c2b-4eb0-bc6f-f9d4d72e71e0 None None] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:00:39.771837 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c719334-63ca-464b-86a5-abbbf5fe4d2b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:00:39.772276 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9450bab-bc1f-4905-ab6a-455530c41738 service cinder] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:00:39.776736 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c719334-63ca-464b-86a5-abbbf5fe4d2b admin admin] REQUEST_METHOD: `PUT` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:00:39.776736 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c719334-63ca-464b-86a5-abbbf5fe4d2b admin admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:00:39.776736 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c719334-63ca-464b-86a5-abbbf5fe4d2b admin admin] PATH_INFO: `/v3/projects/1f7aeeb0a04e43e7a2d5f10bade807d0/users/c66bce3e7b31423ca3cdc8bf15713e60/roles/80fb22507c9a4ff09d44d8f12f6fff1a` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:00:39.779482 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79523fcb-1549-484e-ae82-12763a1e86a3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:00:39.780221 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79523fcb-1549-484e-ae82-12763a1e86a3 admin admin] REQUEST_METHOD: `POST` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:00:39.780507 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79523fcb-1549-484e-ae82-12763a1e86a3 admin admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:00:39.780783 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79523fcb-1549-484e-ae82-12763a1e86a3 admin admin] PATH_INFO: `/v3/projects` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:00:39.781474 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79523fcb-1549-484e-ae82-12763a1e86a3 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:00:39.783481 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79523fcb-1549-484e-ae82-12763a1e86a3 admin admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:00:39.789894 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c719334-63ca-464b-86a5-abbbf5fe4d2b admin admin] RBAC: Authorizing `identity:create_grant(project_id=1f7aeeb0a04e43e7a2d5f10bade807d0, user_id=c66bce3e7b31423ca3cdc8bf15713e60, role_id=80fb22507c9a4ff09d44d8f12f6fff1a)` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:00:39.794652 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c719334-63ca-464b-86a5-abbbf5fe4d2b admin admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:00:39.911038 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 935/1860] 10.222.0.30 () {60 vars in 1482 bytes} [Mon May 23 13:00:39 2022] PUT /identity/v3/projects/1f7aeeb0a04e43e7a2d5f10bade807d0/users/c66bce3e7b31423ca3cdc8bf15713e60/roles/80fb22507c9a4ff09d44d8f12f6fff1a => generated 0 bytes in 157 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) May 23 13:00:39.916336 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 926/1861] 10.222.0.30 () {60 vars in 1152 bytes} [Mon May 23 13:00:39 2022] POST /identity/v3/projects => generated 389 bytes in 155 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) May 23 13:00:39.921574 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c719334-63ca-464b-86a5-abbbf5fe4d2b admin admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:00:39.922668 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c4fea6d-40ff-4fbb-94c4-122848255ba6 None None] REQUEST_METHOD: `POST` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:00:39.922964 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c4fea6d-40ff-4fbb-94c4-122848255ba6 None None] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:00:39.923252 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c4fea6d-40ff-4fbb-94c4-122848255ba6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:00:39.953434 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.auth.core [None req-0c4fea6d-40ff-4fbb-94c4-122848255ba6 None None] MFA Rules not processed for user `c66bce3e7b31423ca3cdc8bf15713e60`. Rule list: `[]` (Enabled: `True`). {{(pid=92056) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 13:00:39.954604 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.fernet_utils [None req-0c4fea6d-40ff-4fbb-94c4-122848255ba6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92056) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:00:39.989227 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c719334-63ca-464b-86a5-abbbf5fe4d2b admin admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:00:40.023837 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 927/1862] 10.222.0.30 () {58 vars in 957 bytes} [Mon May 23 13:00:39 2022] POST /identity/v3/auth/tokens => generated 3254 bytes in 102 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) May 23 13:00:40.029291 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c4fea6d-40ff-4fbb-94c4-122848255ba6 None None] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:00:40.057156 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de8bf520-0d34-4b85-a4d2-8ef7bc5e6fe9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:00:40.059716 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de8bf520-0d34-4b85-a4d2-8ef7bc5e6fe9 admin admin] REQUEST_METHOD: `POST` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:00:40.060008 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de8bf520-0d34-4b85-a4d2-8ef7bc5e6fe9 admin admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:00:40.064210 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de8bf520-0d34-4b85-a4d2-8ef7bc5e6fe9 admin admin] PATH_INFO: `/v3/users` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:00:40.064210 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de8bf520-0d34-4b85-a4d2-8ef7bc5e6fe9 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:00:40.064210 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de8bf520-0d34-4b85-a4d2-8ef7bc5e6fe9 admin admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:00:40.115987 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c4fea6d-40ff-4fbb-94c4-122848255ba6 None None] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:00:40.121989 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5dc155e8-3eb0-4295-8a4b-346c09772e40 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:00:40.123395 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5dc155e8-3eb0-4295-8a4b-346c09772e40 admin admin] REQUEST_METHOD: `POST` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:00:40.123689 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5dc155e8-3eb0-4295-8a4b-346c09772e40 admin admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:00:40.123976 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5dc155e8-3eb0-4295-8a4b-346c09772e40 admin admin] PATH_INFO: `/v3/projects` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:00:40.124691 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5dc155e8-3eb0-4295-8a4b-346c09772e40 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:00:40.126727 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5dc155e8-3eb0-4295-8a4b-346c09772e40 admin admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:00:40.148473 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=92,keystone:INSERT=8 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 13:00:40.155537 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 936/1863] 10.222.0.30 () {60 vars in 1143 bytes} [Mon May 23 13:00:39 2022] POST /identity/v3/users => generated 301 bytes in 235 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) May 23 13:00:40.156413 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de8bf520-0d34-4b85-a4d2-8ef7bc5e6fe9 admin admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:00:40.197342 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 13:00:40.239700 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 928/1864] 10.222.0.30 () {60 vars in 1152 bytes} [Mon May 23 13:00:40 2022] POST /identity/v3/projects => generated 385 bytes in 208 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) May 23 13:00:40.239700 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5dc155e8-3eb0-4295-8a4b-346c09772e40 admin admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:00:40.243760 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de8bf520-0d34-4b85-a4d2-8ef7bc5e6fe9 admin admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:00:40.324038 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 13:00:40.335480 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3dde39e4-8466-43f6-ae0b-8e0c28e2be71 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1666097f8ed545c6b0e4809c93e4775a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9c389ae65aaa4542bb5a5c4fd0bacf5b', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:00:40.336722 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3dde39e4-8466-43f6-ae0b-8e0c28e2be71 service cinder] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:00:40.337385 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3dde39e4-8466-43f6-ae0b-8e0c28e2be71 service cinder] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:00:40.337503 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3dde39e4-8466-43f6-ae0b-8e0c28e2be71 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:00:40.339447 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5dc155e8-3eb0-4295-8a4b-346c09772e40 admin admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:00:40.339639 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.fernet_utils [None req-3dde39e4-8466-43f6-ae0b-8e0c28e2be71 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92057) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:00:40.347300 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ec69727-ced2-4b49-bbb7-2f86c148c75b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:00:40.348106 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ec69727-ced2-4b49-bbb7-2f86c148c75b admin admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:00:40.348534 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ec69727-ced2-4b49-bbb7-2f86c148c75b admin admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:00:40.349003 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ec69727-ced2-4b49-bbb7-2f86c148c75b admin admin] PATH_INFO: `/v3/roles` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:00:40.349818 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ec69727-ced2-4b49-bbb7-2f86c148c75b admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:00:40.351532 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ec69727-ced2-4b49-bbb7-2f86c148c75b admin admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:00:40.360639 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 13:00:40.371004 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 929/1865] 10.222.0.30 () {58 vars in 1121 bytes} [Mon May 23 13:00:40 2022] GET /identity/v3/roles => generated 1420 bytes in 132 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) May 23 13:00:40.371915 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ec69727-ced2-4b49-bbb7-2f86c148c75b admin admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:00:40.382213 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ec69727-ced2-4b49-bbb7-2f86c148c75b admin admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:00:40.390664 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4494e337-9d43-4f2c-b09e-f567e2c2058c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:00:40.391357 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4494e337-9d43-4f2c-b09e-f567e2c2058c admin admin] REQUEST_METHOD: `POST` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:00:40.391732 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4494e337-9d43-4f2c-b09e-f567e2c2058c admin admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:00:40.392072 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4494e337-9d43-4f2c-b09e-f567e2c2058c admin admin] PATH_INFO: `/v3/users` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:00:40.393161 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4494e337-9d43-4f2c-b09e-f567e2c2058c admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:00:40.395906 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4494e337-9d43-4f2c-b09e-f567e2c2058c admin admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:00:40.440596 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3dde39e4-8466-43f6-ae0b-8e0c28e2be71 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:00:40.442197 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3dde39e4-8466-43f6-ae0b-8e0c28e2be71 service cinder] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:00:40.452812 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 937/1866] 10.222.0.30 () {60 vars in 1365 bytes} [Mon May 23 13:00:40 2022] GET /identity/v3/auth/tokens => generated 3232 bytes in 297 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) May 23 13:00:40.453855 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3dde39e4-8466-43f6-ae0b-8e0c28e2be71 service cinder] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:00:40.463498 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 13:00:40.469409 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3dde39e4-8466-43f6-ae0b-8e0c28e2be71 service cinder] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:00:40.474937 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a96374ca-149c-4cc8-a220-c8e72a83e8b9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:00:40.475840 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a96374ca-149c-4cc8-a220-c8e72a83e8b9 admin admin] REQUEST_METHOD: `PUT` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:00:40.476197 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a96374ca-149c-4cc8-a220-c8e72a83e8b9 admin admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:00:40.476647 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a96374ca-149c-4cc8-a220-c8e72a83e8b9 admin admin] PATH_INFO: `/v3/projects/4981e1e0af024a17b76a1136c7973976/users/3ba8699f28764bd5af2d07b226d0e3a5/roles/bc3c3c9d654940cbaa5177a6be23d1bc` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:00:40.494539 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a96374ca-149c-4cc8-a220-c8e72a83e8b9 admin admin] RBAC: Authorizing `identity:create_grant(project_id=4981e1e0af024a17b76a1136c7973976, user_id=3ba8699f28764bd5af2d07b226d0e3a5, role_id=bc3c3c9d654940cbaa5177a6be23d1bc)` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:00:40.498175 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a96374ca-149c-4cc8-a220-c8e72a83e8b9 admin admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:00:40.513950 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 930/1867] 10.222.0.30 () {60 vars in 1143 bytes} [Mon May 23 13:00:40 2022] POST /identity/v3/users => generated 293 bytes in 142 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) May 23 13:00:40.517549 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4494e337-9d43-4f2c-b09e-f567e2c2058c admin admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:00:40.541023 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 13:00:40.543360 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4494e337-9d43-4f2c-b09e-f567e2c2058c admin admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:00:40.551820 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e294e878-9c85-444c-abff-bcfe07aeb613 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:00:40.552575 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e294e878-9c85-444c-abff-bcfe07aeb613 admin admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:00:40.552927 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e294e878-9c85-444c-abff-bcfe07aeb613 admin admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:00:40.553276 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e294e878-9c85-444c-abff-bcfe07aeb613 admin admin] PATH_INFO: `/v3/roles` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:00:40.554116 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e294e878-9c85-444c-abff-bcfe07aeb613 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:00:40.555807 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e294e878-9c85-444c-abff-bcfe07aeb613 admin admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:00:40.571764 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 931/1868] 10.222.0.30 () {58 vars in 1121 bytes} [Mon May 23 13:00:40 2022] GET /identity/v3/roles => generated 1420 bytes in 54 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) May 23 13:00:40.576504 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e294e878-9c85-444c-abff-bcfe07aeb613 admin admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:00:40.587713 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e294e878-9c85-444c-abff-bcfe07aeb613 admin admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:00:40.601464 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 938/1869] 10.222.0.30 () {60 vars in 1482 bytes} [Mon May 23 13:00:40 2022] PUT /identity/v3/projects/4981e1e0af024a17b76a1136c7973976/users/3ba8699f28764bd5af2d07b226d0e3a5/roles/bc3c3c9d654940cbaa5177a6be23d1bc => generated 0 bytes in 148 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) May 23 13:00:40.605197 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a96374ca-149c-4cc8-a220-c8e72a83e8b9 admin admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:00:40.673512 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5049afeb-22d5-40be-9d5a-1e71718120a3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:00:40.674317 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5049afeb-22d5-40be-9d5a-1e71718120a3 admin admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:00:40.674784 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5049afeb-22d5-40be-9d5a-1e71718120a3 admin admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:00:40.675312 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5049afeb-22d5-40be-9d5a-1e71718120a3 admin admin] PATH_INFO: `/v3/roles` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:00:40.676051 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5049afeb-22d5-40be-9d5a-1e71718120a3 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:00:40.677710 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5049afeb-22d5-40be-9d5a-1e71718120a3 admin admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:00:40.702926 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 932/1870] 10.222.0.30 () {58 vars in 1121 bytes} [Mon May 23 13:00:40 2022] GET /identity/v3/roles => generated 1420 bytes in 127 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) May 23 13:00:40.710062 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5049afeb-22d5-40be-9d5a-1e71718120a3 admin admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:00:40.718424 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a96374ca-149c-4cc8-a220-c8e72a83e8b9 admin admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:00:40.723841 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a285a7b8-c7c9-4134-8ea9-f957ec720319 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:00:40.724475 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a285a7b8-c7c9-4134-8ea9-f957ec720319 admin admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:00:40.724828 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a285a7b8-c7c9-4134-8ea9-f957ec720319 admin admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:00:40.725318 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a285a7b8-c7c9-4134-8ea9-f957ec720319 admin admin] PATH_INFO: `/v3/roles` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:00:40.725692 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5049afeb-22d5-40be-9d5a-1e71718120a3 admin admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:00:40.726088 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a285a7b8-c7c9-4134-8ea9-f957ec720319 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:00:40.727892 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a285a7b8-c7c9-4134-8ea9-f957ec720319 admin admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:00:40.736155 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da6e1aa6-c635-4643-993a-c6392afc8967 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:00:40.737792 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da6e1aa6-c635-4643-993a-c6392afc8967 admin admin] REQUEST_METHOD: `PUT` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:00:40.738119 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da6e1aa6-c635-4643-993a-c6392afc8967 admin admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:00:40.738403 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da6e1aa6-c635-4643-993a-c6392afc8967 admin admin] PATH_INFO: `/v3/projects/cd1c87d63b9d408daec92fa22bdf787f/users/2ba91aaaa18a48de964d9079b1df1182/roles/80fb22507c9a4ff09d44d8f12f6fff1a` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:00:40.740741 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 939/1871] 10.222.0.30 () {58 vars in 1121 bytes} [Mon May 23 13:00:40 2022] GET /identity/v3/roles => generated 1420 bytes in 136 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) May 23 13:00:40.744659 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a285a7b8-c7c9-4134-8ea9-f957ec720319 admin admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:00:40.758115 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a285a7b8-c7c9-4134-8ea9-f957ec720319 admin admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:00:40.758391 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da6e1aa6-c635-4643-993a-c6392afc8967 admin admin] RBAC: Authorizing `identity:create_grant(project_id=cd1c87d63b9d408daec92fa22bdf787f, user_id=2ba91aaaa18a48de964d9079b1df1182, role_id=80fb22507c9a4ff09d44d8f12f6fff1a)` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:00:40.761958 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da6e1aa6-c635-4643-993a-c6392afc8967 admin admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:00:40.763930 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96c4f61d-ed3b-495e-8c18-637825683e98 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:00:40.764700 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96c4f61d-ed3b-495e-8c18-637825683e98 admin admin] REQUEST_METHOD: `PUT` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:00:40.764980 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96c4f61d-ed3b-495e-8c18-637825683e98 admin admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:00:40.765250 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96c4f61d-ed3b-495e-8c18-637825683e98 admin admin] PATH_INFO: `/v3/projects/4981e1e0af024a17b76a1136c7973976/users/3ba8699f28764bd5af2d07b226d0e3a5/roles/bc3c3c9d654940cbaa5177a6be23d1bc` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:00:40.767271 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96c4f61d-ed3b-495e-8c18-637825683e98 admin admin] RBAC: Authorizing `identity:create_grant(project_id=4981e1e0af024a17b76a1136c7973976, user_id=3ba8699f28764bd5af2d07b226d0e3a5, role_id=bc3c3c9d654940cbaa5177a6be23d1bc)` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:00:40.769938 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96c4f61d-ed3b-495e-8c18-637825683e98 admin admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:00:40.782054 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 940/1872] 10.222.0.30 () {60 vars in 1482 bytes} [Mon May 23 13:00:40 2022] PUT /identity/v3/projects/4981e1e0af024a17b76a1136c7973976/users/3ba8699f28764bd5af2d07b226d0e3a5/roles/bc3c3c9d654940cbaa5177a6be23d1bc => generated 0 bytes in 38 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) May 23 13:00:40.793559 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb47cb26-da6a-4f88-8219-e80f3e97c222 None None] REQUEST_METHOD: `POST` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:00:40.793987 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb47cb26-da6a-4f88-8219-e80f3e97c222 None None] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:00:40.794276 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb47cb26-da6a-4f88-8219-e80f3e97c222 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:00:40.823601 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.auth.core [None req-bb47cb26-da6a-4f88-8219-e80f3e97c222 None None] MFA Rules not processed for user `3ba8699f28764bd5af2d07b226d0e3a5`. Rule list: `[]` (Enabled: `True`). {{(pid=92057) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 13:00:40.824797 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.fernet_utils [None req-bb47cb26-da6a-4f88-8219-e80f3e97c222 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92057) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:00:40.880797 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 933/1873] 10.222.0.30 () {60 vars in 1482 bytes} [Mon May 23 13:00:40 2022] PUT /identity/v3/projects/cd1c87d63b9d408daec92fa22bdf787f/users/2ba91aaaa18a48de964d9079b1df1182/roles/80fb22507c9a4ff09d44d8f12f6fff1a => generated 0 bytes in 170 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) May 23 13:00:40.889969 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bedec7c-53d5-49ab-b62b-1afd84e79213 None None] REQUEST_METHOD: `POST` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:00:40.890269 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bedec7c-53d5-49ab-b62b-1afd84e79213 None None] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:00:40.890582 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bedec7c-53d5-49ab-b62b-1afd84e79213 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:00:40.903168 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 941/1874] 10.222.0.30 () {58 vars in 957 bytes} [Mon May 23 13:00:40 2022] POST /identity/v3/auth/tokens => generated 3323 bytes in 111 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) May 23 13:00:40.918498 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.auth.core [None req-6bedec7c-53d5-49ab-b62b-1afd84e79213 None None] MFA Rules not processed for user `2ba91aaaa18a48de964d9079b1df1182`. Rule list: `[]` (Enabled: `True`). {{(pid=92056) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 13:00:40.919727 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.fernet_utils [None req-6bedec7c-53d5-49ab-b62b-1afd84e79213 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92056) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:00:40.988102 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 934/1875] 10.222.0.30 () {58 vars in 957 bytes} [Mon May 23 13:00:40 2022] POST /identity/v3/auth/tokens => generated 3252 bytes in 99 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) May 23 13:00:40.992594 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb47cb26-da6a-4f88-8219-e80f3e97c222 None None] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:00:41.014656 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 13:00:41.044481 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6bedec7c-53d5-49ab-b62b-1afd84e79213 None None] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:00:41.068054 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 13:00:41.093220 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb47cb26-da6a-4f88-8219-e80f3e97c222 None None] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:00:41.109704 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6bedec7c-53d5-49ab-b62b-1afd84e79213 None None] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:00:41.113518 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9a0aae6-5a20-48a0-9e26-26c8de274241 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1666097f8ed545c6b0e4809c93e4775a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9c389ae65aaa4542bb5a5c4fd0bacf5b', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:00:41.114197 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9a0aae6-5a20-48a0-9e26-26c8de274241 service cinder] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:00:41.114477 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9a0aae6-5a20-48a0-9e26-26c8de274241 service cinder] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:00:41.114815 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9a0aae6-5a20-48a0-9e26-26c8de274241 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:00:41.169151 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf5ccee7-1514-4583-b5cd-327297609d21 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:00:41.169307 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf5ccee7-1514-4583-b5cd-327297609d21 admin admin] REQUEST_METHOD: `POST` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:00:41.169598 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf5ccee7-1514-4583-b5cd-327297609d21 admin admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:00:41.169901 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf5ccee7-1514-4583-b5cd-327297609d21 admin admin] PATH_INFO: `/v3/projects` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:00:41.170724 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf5ccee7-1514-4583-b5cd-327297609d21 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:00:41.173272 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf5ccee7-1514-4583-b5cd-327297609d21 admin admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:00:41.193288 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9a0aae6-5a20-48a0-9e26-26c8de274241 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:00:41.194790 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9a0aae6-5a20-48a0-9e26-26c8de274241 service cinder] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:00:41.206920 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 935/1876] 10.222.0.30 () {60 vars in 1365 bytes} [Mon May 23 13:00:41 2022] GET /identity/v3/auth/tokens => generated 3323 bytes in 163 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) May 23 13:00:41.208325 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 13:00:41.254146 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 942/1877] 10.222.0.30 () {60 vars in 1152 bytes} [Mon May 23 13:00:40 2022] POST /identity/v3/projects => generated 385 bytes in 262 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) May 23 13:00:41.258239 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9a0aae6-5a20-48a0-9e26-26c8de274241 service cinder] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:00:41.273197 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9a0aae6-5a20-48a0-9e26-26c8de274241 service cinder] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:00:41.347856 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf5ccee7-1514-4583-b5cd-327297609d21 admin admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:00:41.368044 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe07c726-07da-49d2-a617-199f753e5b3e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:00:41.368417 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe07c726-07da-49d2-a617-199f753e5b3e admin admin] REQUEST_METHOD: `POST` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:00:41.368699 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe07c726-07da-49d2-a617-199f753e5b3e admin admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:00:41.368975 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe07c726-07da-49d2-a617-199f753e5b3e admin admin] PATH_INFO: `/v3/users` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:00:41.369712 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe07c726-07da-49d2-a617-199f753e5b3e admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:00:41.372108 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe07c726-07da-49d2-a617-199f753e5b3e admin admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:00:41.438051 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf5ccee7-1514-4583-b5cd-327297609d21 admin admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:00:41.446289 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-572672ba-c18b-4a1a-86a7-ee106767cf33 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:00:41.447538 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-572672ba-c18b-4a1a-86a7-ee106767cf33 admin admin] REQUEST_METHOD: `DELETE` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:00:41.447835 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-572672ba-c18b-4a1a-86a7-ee106767cf33 admin admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:00:41.448954 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-572672ba-c18b-4a1a-86a7-ee106767cf33 admin admin] PATH_INFO: `/v3/users/d1086d05b1ed42b08853995f88f645bf` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:00:41.451361 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-572672ba-c18b-4a1a-86a7-ee106767cf33 admin admin] RBAC: Authorizing `identity:delete_user(user_id=d1086d05b1ed42b08853995f88f645bf)` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:00:41.452467 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 936/1878] 10.222.0.30 () {60 vars in 1143 bytes} [Mon May 23 13:00:41 2022] POST /identity/v3/users => generated 299 bytes in 195 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) May 23 13:00:41.452854 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-572672ba-c18b-4a1a-86a7-ee106767cf33 admin admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:00:41.453569 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe07c726-07da-49d2-a617-199f753e5b3e admin admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:00:41.467716 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe07c726-07da-49d2-a617-199f753e5b3e admin admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:00:41.475325 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec319ef2-21c0-4e2e-86bf-34a63e4e039b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:00:41.476149 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec319ef2-21c0-4e2e-86bf-34a63e4e039b admin admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:00:41.476504 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec319ef2-21c0-4e2e-86bf-34a63e4e039b admin admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:00:41.476886 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec319ef2-21c0-4e2e-86bf-34a63e4e039b admin admin] PATH_INFO: `/v3/roles` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:00:41.477735 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec319ef2-21c0-4e2e-86bf-34a63e4e039b admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:00:41.479882 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec319ef2-21c0-4e2e-86bf-34a63e4e039b admin admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:00:41.493060 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 937/1879] 10.222.0.30 () {58 vars in 1121 bytes} [Mon May 23 13:00:41 2022] GET /identity/v3/roles => generated 1420 bytes in 40 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) May 23 13:00:41.498705 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec319ef2-21c0-4e2e-86bf-34a63e4e039b admin admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:00:41.513353 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec319ef2-21c0-4e2e-86bf-34a63e4e039b admin admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:00:41.520402 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1791ff7-92aa-4800-ac54-3aeaf2ea299f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:00:41.521641 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1791ff7-92aa-4800-ac54-3aeaf2ea299f admin admin] REQUEST_METHOD: `PUT` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:00:41.521988 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1791ff7-92aa-4800-ac54-3aeaf2ea299f admin admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:00:41.522274 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1791ff7-92aa-4800-ac54-3aeaf2ea299f admin admin] PATH_INFO: `/v3/projects/f622a4cb39a14cc789f66eef0a5bf0d1/users/a4cade4474ff444d9734a0b688dd2a83/roles/bc3c3c9d654940cbaa5177a6be23d1bc` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:00:41.556940 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1791ff7-92aa-4800-ac54-3aeaf2ea299f admin admin] RBAC: Authorizing `identity:create_grant(project_id=f622a4cb39a14cc789f66eef0a5bf0d1, user_id=a4cade4474ff444d9734a0b688dd2a83, role_id=bc3c3c9d654940cbaa5177a6be23d1bc)` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:00:41.561314 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1791ff7-92aa-4800-ac54-3aeaf2ea299f admin admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:00:41.642362 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [None req-572672ba-c18b-4a1a-86a7-ee106767cf33 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'd1086d05b1ed42b08853995f88f645bf'} {{(pid=92057) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:00:41.684906 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 938/1880] 10.222.0.30 () {60 vars in 1482 bytes} [Mon May 23 13:00:41 2022] PUT /identity/v3/projects/f622a4cb39a14cc789f66eef0a5bf0d1/users/a4cade4474ff444d9734a0b688dd2a83/roles/bc3c3c9d654940cbaa5177a6be23d1bc => generated 0 bytes in 186 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) May 23 13:00:41.688188 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1791ff7-92aa-4800-ac54-3aeaf2ea299f admin admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:00:41.770652 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [None req-572672ba-c18b-4a1a-86a7-ee106767cf33 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'd1086d05b1ed42b08853995f88f645bf'} {{(pid=92057) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:00:41.783733 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [None req-572672ba-c18b-4a1a-86a7-ee106767cf33 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'd1086d05b1ed42b08853995f88f645bf'} {{(pid=92057) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:00:41.784763 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [None req-572672ba-c18b-4a1a-86a7-ee106767cf33 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'd1086d05b1ed42b08853995f88f645bf'} {{(pid=92057) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:00:41.812792 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1791ff7-92aa-4800-ac54-3aeaf2ea299f admin admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:00:41.815533 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [None req-572672ba-c18b-4a1a-86a7-ee106767cf33 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'd1086d05b1ed42b08853995f88f645bf'} {{(pid=92057) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:00:41.819021 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=88,keystone:INSERT=2,keystone:DELETE=10 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 13:00:41.825189 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5a89ca8-e4a7-4ac9-99f2-580812e82c88 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:00:41.825326 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 943/1881] 10.222.0.30 () {58 vars in 1223 bytes} [Mon May 23 13:00:41 2022] DELETE /identity/v3/users/d1086d05b1ed42b08853995f88f645bf => generated 0 bytes in 478 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) May 23 13:00:41.826985 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5a89ca8-e4a7-4ac9-99f2-580812e82c88 admin admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:00:41.827302 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5a89ca8-e4a7-4ac9-99f2-580812e82c88 admin admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:00:41.827583 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5a89ca8-e4a7-4ac9-99f2-580812e82c88 admin admin] PATH_INFO: `/v3/roles` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:00:41.829184 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5a89ca8-e4a7-4ac9-99f2-580812e82c88 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:00:41.830903 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5a89ca8-e4a7-4ac9-99f2-580812e82c88 admin admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:00:41.834371 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a5ccfb3-2d4f-4658-b051-7cce8a2e3eb9 None None] REQUEST_METHOD: `POST` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:00:41.834371 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a5ccfb3-2d4f-4658-b051-7cce8a2e3eb9 None None] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:00:41.834371 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a5ccfb3-2d4f-4658-b051-7cce8a2e3eb9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:00:41.839719 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 13:00:41.861597 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 939/1882] 10.222.0.30 () {58 vars in 1121 bytes} [Mon May 23 13:00:41 2022] GET /identity/v3/roles => generated 1420 bytes in 173 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) May 23 13:00:41.862834 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.auth.core [None req-7a5ccfb3-2d4f-4658-b051-7cce8a2e3eb9 None None] MFA Rules not processed for user `0f7119f8fa5c4445977671140ded91a4`. Rule list: `[]` (Enabled: `True`). {{(pid=92057) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 13:00:41.864866 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5a89ca8-e4a7-4ac9-99f2-580812e82c88 admin admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:00:41.865763 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.fernet_utils [None req-7a5ccfb3-2d4f-4658-b051-7cce8a2e3eb9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92057) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:00:41.866902 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.fernet_utils [None req-d5a89ca8-e4a7-4ac9-99f2-580812e82c88 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92056) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:00:41.883362 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5a89ca8-e4a7-4ac9-99f2-580812e82c88 admin admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:00:41.949306 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 944/1883] 10.222.0.30 () {58 vars in 957 bytes} [Mon May 23 13:00:41 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 119 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) May 23 13:00:41.959859 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a5ccfb3-2d4f-4658-b051-7cce8a2e3eb9 None None] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:00:41.961687 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.fernet_utils [None req-7a5ccfb3-2d4f-4658-b051-7cce8a2e3eb9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92057) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:00:41.966827 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f218bdf3-b4ab-4ae2-88af-b4e1386be85a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:00:41.967735 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f218bdf3-b4ab-4ae2-88af-b4e1386be85a admin admin] REQUEST_METHOD: `PUT` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:00:41.968021 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f218bdf3-b4ab-4ae2-88af-b4e1386be85a admin admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:00:41.968302 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f218bdf3-b4ab-4ae2-88af-b4e1386be85a admin admin] PATH_INFO: `/v3/projects/f622a4cb39a14cc789f66eef0a5bf0d1/users/a4cade4474ff444d9734a0b688dd2a83/roles/bc3c3c9d654940cbaa5177a6be23d1bc` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:00:41.970458 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f218bdf3-b4ab-4ae2-88af-b4e1386be85a admin admin] RBAC: Authorizing `identity:create_grant(project_id=f622a4cb39a14cc789f66eef0a5bf0d1, user_id=a4cade4474ff444d9734a0b688dd2a83, role_id=bc3c3c9d654940cbaa5177a6be23d1bc)` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:00:41.974692 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f218bdf3-b4ab-4ae2-88af-b4e1386be85a admin admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:00:41.993041 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 940/1884] 10.222.0.30 () {60 vars in 1482 bytes} [Mon May 23 13:00:41 2022] PUT /identity/v3/projects/f622a4cb39a14cc789f66eef0a5bf0d1/users/a4cade4474ff444d9734a0b688dd2a83/roles/bc3c3c9d654940cbaa5177a6be23d1bc => generated 0 bytes in 129 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) May 23 13:00:42.008633 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7afeaec-155f-46a5-82e2-135dd8bfae11 None None] REQUEST_METHOD: `POST` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:00:42.008844 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7afeaec-155f-46a5-82e2-135dd8bfae11 None None] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:00:42.009129 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7afeaec-155f-46a5-82e2-135dd8bfae11 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:00:42.035772 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a5ccfb3-2d4f-4658-b051-7cce8a2e3eb9 None None] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:00:42.041158 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.auth.core [None req-f7afeaec-155f-46a5-82e2-135dd8bfae11 None None] MFA Rules not processed for user `a4cade4474ff444d9734a0b688dd2a83`. Rule list: `[]` (Enabled: `True`). {{(pid=92056) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 13:00:42.043728 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.fernet_utils [None req-f7afeaec-155f-46a5-82e2-135dd8bfae11 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92056) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:00:42.101622 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6098399-ee39-4636-81d6-fc6d47a4288b service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2eb83ca38e79449095f3a2f947bbb06a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9c389ae65aaa4542bb5a5c4fd0bacf5b', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:00:42.102320 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6098399-ee39-4636-81d6-fc6d47a4288b service neutron] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:00:42.102691 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6098399-ee39-4636-81d6-fc6d47a4288b service neutron] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:00:42.102963 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6098399-ee39-4636-81d6-fc6d47a4288b service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:00:42.134278 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 941/1885] 10.222.0.30 () {58 vars in 957 bytes} [Mon May 23 13:00:42 2022] POST /identity/v3/auth/tokens => generated 3319 bytes in 123 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) May 23 13:00:42.143737 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7afeaec-155f-46a5-82e2-135dd8bfae11 None None] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:00:42.145139 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.fernet_utils [None req-f7afeaec-155f-46a5-82e2-135dd8bfae11 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92056) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:00:42.186952 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6098399-ee39-4636-81d6-fc6d47a4288b service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:00:42.188494 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6098399-ee39-4636-81d6-fc6d47a4288b service neutron] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:00:42.200218 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 945/1886] 10.222.0.30 () {60 vars in 1365 bytes} [Mon May 23 13:00:41 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 241 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) May 23 13:00:42.221147 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7afeaec-155f-46a5-82e2-135dd8bfae11 None None] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:00:42.224868 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f9f79f7-e28e-44d3-a50f-a51fef8582a1 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1666097f8ed545c6b0e4809c93e4775a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9c389ae65aaa4542bb5a5c4fd0bacf5b', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:00:42.225527 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f9f79f7-e28e-44d3-a50f-a51fef8582a1 service cinder] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:00:42.225810 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f9f79f7-e28e-44d3-a50f-a51fef8582a1 service cinder] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:00:42.226120 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f9f79f7-e28e-44d3-a50f-a51fef8582a1 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:00:42.238136 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f9f79f7-e28e-44d3-a50f-a51fef8582a1 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:00:42.238136 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f9f79f7-e28e-44d3-a50f-a51fef8582a1 service cinder] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:00:42.248644 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 942/1887] 10.222.0.30 () {60 vars in 1365 bytes} [Mon May 23 13:00:42 2022] GET /identity/v3/auth/tokens => generated 3319 bytes in 105 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) May 23 13:00:42.642153 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6098399-ee39-4636-81d6-fc6d47a4288b service neutron] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:00:42.642153 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.fernet_utils [None req-b6098399-ee39-4636-81d6-fc6d47a4288b service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92057) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:00:42.655347 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6098399-ee39-4636-81d6-fc6d47a4288b service neutron] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:00:42.661248 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ad5813a-7366-4abb-bc19-4d8fec99c113 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:00:42.662289 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ad5813a-7366-4abb-bc19-4d8fec99c113 admin admin] REQUEST_METHOD: `DELETE` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:00:42.662817 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ad5813a-7366-4abb-bc19-4d8fec99c113 admin admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:00:42.663041 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ad5813a-7366-4abb-bc19-4d8fec99c113 admin admin] PATH_INFO: `/v3/projects/fbf2a131f924445c8c8651592f8db51e` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:00:42.664397 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ad5813a-7366-4abb-bc19-4d8fec99c113 admin admin] RBAC: Authorizing `identity:delete_project(project_id=fbf2a131f924445c8c8651592f8db51e)` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:00:42.666365 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ad5813a-7366-4abb-bc19-4d8fec99c113 admin admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:00:42.668596 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=100 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 13:00:42.792687 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [None req-6ad5813a-7366-4abb-bc19-4d8fec99c113 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'fbf2a131f924445c8c8651592f8db51e', 'request_id': 'req-6ad5813a-7366-4abb-bc19-4d8fec99c113'} {{(pid=92057) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:00:42.801448 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [None req-6ad5813a-7366-4abb-bc19-4d8fec99c113 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'fbf2a131f924445c8c8651592f8db51e', 'request_id': 'req-6ad5813a-7366-4abb-bc19-4d8fec99c113'} {{(pid=92057) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:00:42.913048 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [None req-6ad5813a-7366-4abb-bc19-4d8fec99c113 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'fbf2a131f924445c8c8651592f8db51e', 'request_id': 'req-6ad5813a-7366-4abb-bc19-4d8fec99c113'} {{(pid=92057) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:00:42.925263 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [None req-6ad5813a-7366-4abb-bc19-4d8fec99c113 admin admin] The token cache is being invalidate because project fbf2a131f924445c8c8651592f8db51e was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=92057) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} May 23 13:00:42.925558 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [None req-6ad5813a-7366-4abb-bc19-4d8fec99c113 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=92057) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:00:42.926239 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [None req-6ad5813a-7366-4abb-bc19-4d8fec99c113 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=92057) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:00:42.927793 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 946/1888] 10.222.0.30 () {58 vars in 1232 bytes} [Mon May 23 13:00:42 2022] DELETE /identity/v3/projects/fbf2a131f924445c8c8651592f8db51e => generated 0 bytes in 290 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) May 23 13:00:42.931834 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f9f79f7-e28e-44d3-a50f-a51fef8582a1 service cinder] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:00:42.943185 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f9f79f7-e28e-44d3-a50f-a51fef8582a1 service cinder] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:00:42.956231 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 13:00:43.008283 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ebf90206-e9e8-4cfe-9b25-4f9c14da0ea6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:00:43.009185 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebf90206-e9e8-4cfe-9b25-4f9c14da0ea6 admin admin] REQUEST_METHOD: `DELETE` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:00:43.017966 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebf90206-e9e8-4cfe-9b25-4f9c14da0ea6 admin admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:00:43.017966 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebf90206-e9e8-4cfe-9b25-4f9c14da0ea6 admin admin] PATH_INFO: `/v3/users/3ba8699f28764bd5af2d07b226d0e3a5` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:00:43.017966 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ebf90206-e9e8-4cfe-9b25-4f9c14da0ea6 admin admin] RBAC: Authorizing `identity:delete_user(user_id=3ba8699f28764bd5af2d07b226d0e3a5)` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:00:43.017966 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ebf90206-e9e8-4cfe-9b25-4f9c14da0ea6 admin admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:00:43.207688 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-ebf90206-e9e8-4cfe-9b25-4f9c14da0ea6 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '3ba8699f28764bd5af2d07b226d0e3a5'} {{(pid=92056) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:00:43.208485 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-ebf90206-e9e8-4cfe-9b25-4f9c14da0ea6 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '3ba8699f28764bd5af2d07b226d0e3a5'} {{(pid=92056) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:00:43.297240 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-ebf90206-e9e8-4cfe-9b25-4f9c14da0ea6 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '3ba8699f28764bd5af2d07b226d0e3a5'} {{(pid=92056) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:00:43.312479 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-ebf90206-e9e8-4cfe-9b25-4f9c14da0ea6 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '3ba8699f28764bd5af2d07b226d0e3a5'} {{(pid=92056) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:00:43.319483 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-ebf90206-e9e8-4cfe-9b25-4f9c14da0ea6 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '3ba8699f28764bd5af2d07b226d0e3a5'} {{(pid=92056) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:00:43.330823 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 943/1889] 10.222.0.30 () {58 vars in 1223 bytes} [Mon May 23 13:00:42 2022] DELETE /identity/v3/users/3ba8699f28764bd5af2d07b226d0e3a5 => generated 0 bytes in 395 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) May 23 13:00:43.456017 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ad5813a-7366-4abb-bc19-4d8fec99c113 admin admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:00:43.457419 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.fernet_utils [None req-6ad5813a-7366-4abb-bc19-4d8fec99c113 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92057) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:00:43.515317 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ad5813a-7366-4abb-bc19-4d8fec99c113 admin admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:00:43.568653 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4312749-4142-49d2-9628-5bf3fe736b13 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1666097f8ed545c6b0e4809c93e4775a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9c389ae65aaa4542bb5a5c4fd0bacf5b', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:00:43.569338 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4312749-4142-49d2-9628-5bf3fe736b13 service cinder] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:00:43.569623 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4312749-4142-49d2-9628-5bf3fe736b13 service cinder] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:00:43.569924 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4312749-4142-49d2-9628-5bf3fe736b13 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:00:43.571788 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.fernet_utils [None req-d4312749-4142-49d2-9628-5bf3fe736b13 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92057) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:00:43.631936 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4312749-4142-49d2-9628-5bf3fe736b13 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:00:43.633393 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4312749-4142-49d2-9628-5bf3fe736b13 service cinder] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:00:43.638427 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 13:00:43.643460 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 947/1890] 10.222.0.30 () {60 vars in 1365 bytes} [Mon May 23 13:00:43 2022] GET /identity/v3/auth/tokens => generated 3254 bytes in 188 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) May 23 13:00:43.735030 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ebf90206-e9e8-4cfe-9b25-4f9c14da0ea6 admin admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:00:43.736433 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.fernet_utils [None req-ebf90206-e9e8-4cfe-9b25-4f9c14da0ea6 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92056) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:00:43.806582 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ebf90206-e9e8-4cfe-9b25-4f9c14da0ea6 admin admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:00:43.811719 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86547f6c-5b1d-4cb0-9b8d-78b9f16a37dd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:00:43.812568 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86547f6c-5b1d-4cb0-9b8d-78b9f16a37dd admin admin] REQUEST_METHOD: `DELETE` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:00:43.812851 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86547f6c-5b1d-4cb0-9b8d-78b9f16a37dd admin admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:00:43.813126 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86547f6c-5b1d-4cb0-9b8d-78b9f16a37dd admin admin] PATH_INFO: `/v3/projects/4981e1e0af024a17b76a1136c7973976` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:00:43.815079 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-86547f6c-5b1d-4cb0-9b8d-78b9f16a37dd admin admin] RBAC: Authorizing `identity:delete_project(project_id=4981e1e0af024a17b76a1136c7973976)` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:00:43.817050 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-86547f6c-5b1d-4cb0-9b8d-78b9f16a37dd admin admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:00:43.925026 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=86,keystone:DELETE=13,keystone:INSERT=1 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 13:00:43.931985 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-86547f6c-5b1d-4cb0-9b8d-78b9f16a37dd admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '4981e1e0af024a17b76a1136c7973976', 'request_id': 'req-86547f6c-5b1d-4cb0-9b8d-78b9f16a37dd'} {{(pid=92056) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:00:44.068728 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-86547f6c-5b1d-4cb0-9b8d-78b9f16a37dd admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '4981e1e0af024a17b76a1136c7973976', 'request_id': 'req-86547f6c-5b1d-4cb0-9b8d-78b9f16a37dd'} {{(pid=92056) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:00:44.078023 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-86547f6c-5b1d-4cb0-9b8d-78b9f16a37dd admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '4981e1e0af024a17b76a1136c7973976', 'request_id': 'req-86547f6c-5b1d-4cb0-9b8d-78b9f16a37dd'} {{(pid=92056) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:00:44.097278 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-86547f6c-5b1d-4cb0-9b8d-78b9f16a37dd admin admin] The token cache is being invalidate because project 4981e1e0af024a17b76a1136c7973976 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=92056) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} May 23 13:00:44.100875 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-86547f6c-5b1d-4cb0-9b8d-78b9f16a37dd admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=92056) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:00:44.100875 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-86547f6c-5b1d-4cb0-9b8d-78b9f16a37dd admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=92056) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:00:44.100875 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 944/1891] 10.222.0.30 () {58 vars in 1232 bytes} [Mon May 23 13:00:43 2022] DELETE /identity/v3/projects/4981e1e0af024a17b76a1136c7973976 => generated 0 bytes in 365 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) May 23 13:00:44.105130 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb3b2329-d3e8-4100-b1e5-960445db7be9 None None] REQUEST_METHOD: `POST` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:00:44.105597 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb3b2329-d3e8-4100-b1e5-960445db7be9 None None] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:00:44.105871 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb3b2329-d3e8-4100-b1e5-960445db7be9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:00:44.123367 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.auth.core [None req-cb3b2329-d3e8-4100-b1e5-960445db7be9 None None] MFA Rules not processed for user `0f7119f8fa5c4445977671140ded91a4`. Rule list: `[]` (Enabled: `True`). {{(pid=92057) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 13:00:44.124601 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.fernet_utils [None req-cb3b2329-d3e8-4100-b1e5-960445db7be9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92057) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:00:44.183922 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 948/1892] 10.222.0.30 () {58 vars in 957 bytes} [Mon May 23 13:00:44 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 80 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) May 23 13:00:44.191653 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-231a53d6-009d-49aa-b701-53cb569c4427 None None] REQUEST_METHOD: `POST` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:00:44.191942 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-231a53d6-009d-49aa-b701-53cb569c4427 None None] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:00:44.192219 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-231a53d6-009d-49aa-b701-53cb569c4427 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:00:44.218842 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.auth.core [None req-231a53d6-009d-49aa-b701-53cb569c4427 None None] MFA Rules not processed for user `0f7119f8fa5c4445977671140ded91a4`. Rule list: `[]` (Enabled: `True`). {{(pid=92056) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 13:00:44.220390 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.fernet_utils [None req-231a53d6-009d-49aa-b701-53cb569c4427 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92056) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:00:44.230798 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 945/1893] 10.222.0.30 () {58 vars in 957 bytes} [Mon May 23 13:00:44 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 38 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) May 23 13:00:44.233652 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb3b2329-d3e8-4100-b1e5-960445db7be9 None None] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:00:44.245061 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb3b2329-d3e8-4100-b1e5-960445db7be9 None None] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:00:44.250458 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7405976f-05f4-43f6-8d81-2f0385d82fec admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:00:44.251124 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7405976f-05f4-43f6-8d81-2f0385d82fec admin admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:00:44.251841 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7405976f-05f4-43f6-8d81-2f0385d82fec admin admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:00:44.252268 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7405976f-05f4-43f6-8d81-2f0385d82fec admin admin] PATH_INFO: `/v3/domains` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:00:44.253597 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7405976f-05f4-43f6-8d81-2f0385d82fec admin admin] RBAC: Adding query filter params (name=Default) {{(pid=92057) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} May 23 13:00:44.254264 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7405976f-05f4-43f6-8d81-2f0385d82fec admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:00:44.257283 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7405976f-05f4-43f6-8d81-2f0385d82fec admin admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:00:44.288413 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 949/1894] 10.222.0.30 () {58 vars in 1152 bytes} [Mon May 23 13:00:44 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 55 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) May 23 13:00:44.294115 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-231a53d6-009d-49aa-b701-53cb569c4427 None None] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:00:44.320151 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-231a53d6-009d-49aa-b701-53cb569c4427 None None] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:00:44.327254 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bdd71e41-31c8-4a42-bad9-6ca902a48ca4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:00:44.328462 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdd71e41-31c8-4a42-bad9-6ca902a48ca4 admin admin] REQUEST_METHOD: `POST` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:00:44.330204 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdd71e41-31c8-4a42-bad9-6ca902a48ca4 admin admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:00:44.330862 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdd71e41-31c8-4a42-bad9-6ca902a48ca4 admin admin] PATH_INFO: `/v3/projects` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:00:44.332204 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bdd71e41-31c8-4a42-bad9-6ca902a48ca4 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:00:44.334439 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bdd71e41-31c8-4a42-bad9-6ca902a48ca4 admin admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:00:44.480878 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 946/1895] 10.222.0.30 () {60 vars in 1152 bytes} [Mon May 23 13:00:44 2022] POST /identity/v3/projects => generated 377 bytes in 185 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) May 23 13:00:44.482179 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7405976f-05f4-43f6-8d81-2f0385d82fec admin admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:00:44.492491 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7405976f-05f4-43f6-8d81-2f0385d82fec admin admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:00:44.561315 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ad509f4-8f52-4a9d-8cbe-65205b167d4f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:00:44.562163 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ad509f4-8f52-4a9d-8cbe-65205b167d4f admin admin] REQUEST_METHOD: `POST` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:00:44.564532 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ad509f4-8f52-4a9d-8cbe-65205b167d4f admin admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:00:44.564532 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ad509f4-8f52-4a9d-8cbe-65205b167d4f admin admin] PATH_INFO: `/v3/users` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:00:44.564532 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ad509f4-8f52-4a9d-8cbe-65205b167d4f admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:00:44.565958 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ad509f4-8f52-4a9d-8cbe-65205b167d4f admin admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:00:44.696942 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 950/1896] 10.222.0.30 () {60 vars in 1143 bytes} [Mon May 23 13:00:44 2022] POST /identity/v3/users => generated 289 bytes in 215 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) May 23 13:00:44.701175 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bdd71e41-31c8-4a42-bad9-6ca902a48ca4 admin admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:00:44.711497 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bdd71e41-31c8-4a42-bad9-6ca902a48ca4 admin admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:00:44.716487 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d3d7ee2-a283-4889-8eae-0a7a9d26607a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:00:44.717144 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d3d7ee2-a283-4889-8eae-0a7a9d26607a admin admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:00:44.717427 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d3d7ee2-a283-4889-8eae-0a7a9d26607a admin admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:00:44.717721 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d3d7ee2-a283-4889-8eae-0a7a9d26607a admin admin] PATH_INFO: `/v3/roles` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:00:44.718474 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d3d7ee2-a283-4889-8eae-0a7a9d26607a admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:00:44.720129 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d3d7ee2-a283-4889-8eae-0a7a9d26607a admin admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:00:44.731460 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 947/1897] 10.222.0.30 () {58 vars in 1121 bytes} [Mon May 23 13:00:44 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) May 23 13:00:44.735501 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ad509f4-8f52-4a9d-8cbe-65205b167d4f admin admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:00:44.747350 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ad509f4-8f52-4a9d-8cbe-65205b167d4f admin admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:00:44.751996 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-204936d0-00f7-4b37-8bf0-062a82e61a60 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:00:44.752672 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-204936d0-00f7-4b37-8bf0-062a82e61a60 admin admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:00:44.752954 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-204936d0-00f7-4b37-8bf0-062a82e61a60 admin admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:00:44.753228 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-204936d0-00f7-4b37-8bf0-062a82e61a60 admin admin] PATH_INFO: `/v3/roles` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:00:44.753972 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-204936d0-00f7-4b37-8bf0-062a82e61a60 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:00:44.755661 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-204936d0-00f7-4b37-8bf0-062a82e61a60 admin admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:00:44.767127 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 951/1898] 10.222.0.30 () {58 vars in 1121 bytes} [Mon May 23 13:00:44 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) May 23 13:00:44.771338 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d3d7ee2-a283-4889-8eae-0a7a9d26607a admin admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:00:44.782200 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d3d7ee2-a283-4889-8eae-0a7a9d26607a admin admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:00:44.786806 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4da853ca-4dcb-4021-b05f-dcb50eff715f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:00:44.787787 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4da853ca-4dcb-4021-b05f-dcb50eff715f admin admin] REQUEST_METHOD: `PUT` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:00:44.788336 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4da853ca-4dcb-4021-b05f-dcb50eff715f admin admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:00:44.788336 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4da853ca-4dcb-4021-b05f-dcb50eff715f admin admin] PATH_INFO: `/v3/projects/1b4f7473795d443a914ef0d5ce171ae8/users/5c4942bc48564b84bac366ba76920a70/roles/80fb22507c9a4ff09d44d8f12f6fff1a` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:00:44.803428 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4da853ca-4dcb-4021-b05f-dcb50eff715f admin admin] RBAC: Authorizing `identity:create_grant(project_id=1b4f7473795d443a914ef0d5ce171ae8, user_id=5c4942bc48564b84bac366ba76920a70, role_id=80fb22507c9a4ff09d44d8f12f6fff1a)` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:00:44.806207 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4da853ca-4dcb-4021-b05f-dcb50eff715f admin admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:00:44.875953 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 948/1899] 10.222.0.30 () {60 vars in 1482 bytes} [Mon May 23 13:00:44 2022] PUT /identity/v3/projects/1b4f7473795d443a914ef0d5ce171ae8/users/5c4942bc48564b84bac366ba76920a70/roles/80fb22507c9a4ff09d44d8f12f6fff1a => generated 0 bytes in 105 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) May 23 13:00:44.888377 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b41fb55-cedc-427a-9df3-b90d797b7fe4 None None] REQUEST_METHOD: `POST` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:00:44.888670 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b41fb55-cedc-427a-9df3-b90d797b7fe4 None None] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:00:44.888950 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b41fb55-cedc-427a-9df3-b90d797b7fe4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:00:44.916393 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.auth.core [None req-8b41fb55-cedc-427a-9df3-b90d797b7fe4 None None] MFA Rules not processed for user `5c4942bc48564b84bac366ba76920a70`. Rule list: `[]` (Enabled: `True`). {{(pid=92057) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 13:00:44.917647 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.fernet_utils [None req-8b41fb55-cedc-427a-9df3-b90d797b7fe4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92057) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:00:44.973470 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 13:00:44.982129 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 952/1900] 10.222.0.30 () {58 vars in 957 bytes} [Mon May 23 13:00:44 2022] POST /identity/v3/auth/tokens => generated 3244 bytes in 95 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) May 23 13:00:44.986878 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4da853ca-4dcb-4021-b05f-dcb50eff715f admin admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:00:45.048705 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4da853ca-4dcb-4021-b05f-dcb50eff715f admin admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:00:45.053305 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-569456c6-2fb7-4af4-8e00-658bf470fda1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:00:45.053987 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-569456c6-2fb7-4af4-8e00-658bf470fda1 admin admin] REQUEST_METHOD: `POST` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:00:45.054274 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-569456c6-2fb7-4af4-8e00-658bf470fda1 admin admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:00:45.054552 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-569456c6-2fb7-4af4-8e00-658bf470fda1 admin admin] PATH_INFO: `/v3/projects` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:00:45.055240 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-569456c6-2fb7-4af4-8e00-658bf470fda1 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:00:45.057762 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-569456c6-2fb7-4af4-8e00-658bf470fda1 admin admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:00:45.176212 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 949/1901] 10.222.0.30 () {60 vars in 1152 bytes} [Mon May 23 13:00:44 2022] POST /identity/v3/projects => generated 379 bytes in 189 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) May 23 13:00:45.180618 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b41fb55-cedc-427a-9df3-b90d797b7fe4 None None] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:00:45.192091 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b41fb55-cedc-427a-9df3-b90d797b7fe4 None None] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:00:45.263971 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0d0166f-7608-42e0-ba42-6389af433b11 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:00:45.264885 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0d0166f-7608-42e0-ba42-6389af433b11 admin admin] REQUEST_METHOD: `POST` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:00:45.265170 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0d0166f-7608-42e0-ba42-6389af433b11 admin admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:00:45.265448 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0d0166f-7608-42e0-ba42-6389af433b11 admin admin] PATH_INFO: `/v3/users` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:00:45.266187 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0d0166f-7608-42e0-ba42-6389af433b11 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:00:45.268115 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0d0166f-7608-42e0-ba42-6389af433b11 admin admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:00:45.364033 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 953/1902] 10.222.0.30 () {60 vars in 1143 bytes} [Mon May 23 13:00:45 2022] POST /identity/v3/users => generated 290 bytes in 181 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) May 23 13:00:45.364928 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-569456c6-2fb7-4af4-8e00-658bf470fda1 admin admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:00:45.375646 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-569456c6-2fb7-4af4-8e00-658bf470fda1 admin admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:00:45.381632 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d0907f9-7c88-4638-a5da-d5922c3ae842 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:00:45.382277 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d0907f9-7c88-4638-a5da-d5922c3ae842 admin admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:00:45.382568 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d0907f9-7c88-4638-a5da-d5922c3ae842 admin admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:00:45.382845 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d0907f9-7c88-4638-a5da-d5922c3ae842 admin admin] PATH_INFO: `/v3/roles` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:00:45.383972 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d0907f9-7c88-4638-a5da-d5922c3ae842 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:00:45.385728 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d0907f9-7c88-4638-a5da-d5922c3ae842 admin admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:00:45.399609 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 950/1903] 10.222.0.30 () {58 vars in 1121 bytes} [Mon May 23 13:00:45 2022] GET /identity/v3/roles => generated 1420 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) May 23 13:00:45.401583 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0d0166f-7608-42e0-ba42-6389af433b11 admin admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:00:45.403273 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d0907f9-7c88-4638-a5da-d5922c3ae842 admin admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:00:45.403649 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.fernet_utils [None req-d0d0166f-7608-42e0-ba42-6389af433b11 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92057) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:00:45.415573 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0d0166f-7608-42e0-ba42-6389af433b11 admin admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:00:45.418173 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d0907f9-7c88-4638-a5da-d5922c3ae842 admin admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:00:45.421988 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8fa5971-bb28-4b9a-bfa2-71ea2c11fe01 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:00:45.422918 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4d8d294-7ef4-4e9f-944b-e39c93dc53f8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:00:45.423039 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8fa5971-bb28-4b9a-bfa2-71ea2c11fe01 admin admin] REQUEST_METHOD: `DELETE` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:00:45.423221 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8fa5971-bb28-4b9a-bfa2-71ea2c11fe01 admin admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:00:45.423474 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8fa5971-bb28-4b9a-bfa2-71ea2c11fe01 admin admin] PATH_INFO: `/v3/users/4777a2b9585c446fad7630ad914716a0` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:00:45.423586 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4d8d294-7ef4-4e9f-944b-e39c93dc53f8 admin admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:00:45.423853 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4d8d294-7ef4-4e9f-944b-e39c93dc53f8 admin admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:00:45.424127 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4d8d294-7ef4-4e9f-944b-e39c93dc53f8 admin admin] PATH_INFO: `/v3/roles` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:00:45.424725 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d8fa5971-bb28-4b9a-bfa2-71ea2c11fe01 admin admin] RBAC: Authorizing `identity:delete_user(user_id=4777a2b9585c446fad7630ad914716a0)` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:00:45.424888 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f4d8d294-7ef4-4e9f-944b-e39c93dc53f8 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:00:45.426467 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f4d8d294-7ef4-4e9f-944b-e39c93dc53f8 admin admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:00:45.426713 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d8fa5971-bb28-4b9a-bfa2-71ea2c11fe01 admin admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:00:45.439023 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 951/1904] 10.222.0.30 () {58 vars in 1121 bytes} [Mon May 23 13:00:45 2022] GET /identity/v3/roles => generated 1420 bytes in 36 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) May 23 13:00:45.442669 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4d8d294-7ef4-4e9f-944b-e39c93dc53f8 admin admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:00:45.452301 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4d8d294-7ef4-4e9f-944b-e39c93dc53f8 admin admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:00:45.456875 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02044d30-8b9d-48ac-ac0a-f425314cf172 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:00:45.457749 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02044d30-8b9d-48ac-ac0a-f425314cf172 admin admin] REQUEST_METHOD: `PUT` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:00:45.458057 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02044d30-8b9d-48ac-ac0a-f425314cf172 admin admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:00:45.458335 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02044d30-8b9d-48ac-ac0a-f425314cf172 admin admin] PATH_INFO: `/v3/projects/c2089bce847847b4808e4c44e56bf2cf/users/99dd30f1f1124a9086f747794897bc0f/roles/80fb22507c9a4ff09d44d8f12f6fff1a` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:00:45.473919 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-02044d30-8b9d-48ac-ac0a-f425314cf172 admin admin] RBAC: Authorizing `identity:create_grant(project_id=c2089bce847847b4808e4c44e56bf2cf, user_id=99dd30f1f1124a9086f747794897bc0f, role_id=80fb22507c9a4ff09d44d8f12f6fff1a)` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:00:45.476579 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-02044d30-8b9d-48ac-ac0a-f425314cf172 admin admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:00:45.563095 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 952/1905] 10.222.0.30 () {60 vars in 1482 bytes} [Mon May 23 13:00:45 2022] PUT /identity/v3/projects/c2089bce847847b4808e4c44e56bf2cf/users/99dd30f1f1124a9086f747794897bc0f/roles/80fb22507c9a4ff09d44d8f12f6fff1a => generated 0 bytes in 121 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) May 23 13:00:45.573307 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e98de12-a686-4cba-923b-ef7674eee525 None None] REQUEST_METHOD: `POST` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:00:45.573598 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e98de12-a686-4cba-923b-ef7674eee525 None None] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:00:45.573910 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e98de12-a686-4cba-923b-ef7674eee525 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:00:45.579452 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=91,keystone:DELETE=4,keystone:INSERT=5 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 13:00:45.599604 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.auth.core [None req-2e98de12-a686-4cba-923b-ef7674eee525 None None] MFA Rules not processed for user `99dd30f1f1124a9086f747794897bc0f`. Rule list: `[]` (Enabled: `True`). {{(pid=92056) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 13:00:45.600820 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.fernet_utils [None req-2e98de12-a686-4cba-923b-ef7674eee525 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92056) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:00:45.636580 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [None req-d8fa5971-bb28-4b9a-bfa2-71ea2c11fe01 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '4777a2b9585c446fad7630ad914716a0'} {{(pid=92057) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:00:45.667342 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 953/1906] 10.222.0.30 () {58 vars in 957 bytes} [Mon May 23 13:00:45 2022] POST /identity/v3/auth/tokens => generated 3246 bytes in 95 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) May 23 13:00:45.672115 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e98de12-a686-4cba-923b-ef7674eee525 None None] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:00:45.719473 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [None req-d8fa5971-bb28-4b9a-bfa2-71ea2c11fe01 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '4777a2b9585c446fad7630ad914716a0'} {{(pid=92057) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:00:45.727528 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [None req-d8fa5971-bb28-4b9a-bfa2-71ea2c11fe01 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '4777a2b9585c446fad7630ad914716a0'} {{(pid=92057) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:00:45.729973 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [None req-d8fa5971-bb28-4b9a-bfa2-71ea2c11fe01 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '4777a2b9585c446fad7630ad914716a0'} {{(pid=92057) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:00:45.745308 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [None req-d8fa5971-bb28-4b9a-bfa2-71ea2c11fe01 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '4777a2b9585c446fad7630ad914716a0'} {{(pid=92057) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:00:45.749533 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e98de12-a686-4cba-923b-ef7674eee525 None None] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:00:45.754739 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 954/1907] 10.222.0.30 () {58 vars in 1223 bytes} [Mon May 23 13:00:45 2022] DELETE /identity/v3/users/4777a2b9585c446fad7630ad914716a0 => generated 0 bytes in 351 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) May 23 13:00:45.756579 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42227b2b-8a24-4dbc-9b15-325e82ef944b None None] REQUEST_METHOD: `POST` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:00:45.756866 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42227b2b-8a24-4dbc-9b15-325e82ef944b None None] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:00:45.757141 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42227b2b-8a24-4dbc-9b15-325e82ef944b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:00:45.757530 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62b74baa-4556-4e8f-8955-5b03dadaedff admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:00:45.759061 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62b74baa-4556-4e8f-8955-5b03dadaedff admin admin] REQUEST_METHOD: `POST` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:00:45.759352 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62b74baa-4556-4e8f-8955-5b03dadaedff admin admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:00:45.759631 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62b74baa-4556-4e8f-8955-5b03dadaedff admin admin] PATH_INFO: `/v3/projects` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:00:45.761109 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-62b74baa-4556-4e8f-8955-5b03dadaedff admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:00:45.764146 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-62b74baa-4556-4e8f-8955-5b03dadaedff admin admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:00:45.777145 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.auth.core [None req-42227b2b-8a24-4dbc-9b15-325e82ef944b None None] MFA Rules not processed for user `0f7119f8fa5c4445977671140ded91a4`. Rule list: `[]` (Enabled: `True`). {{(pid=92057) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 13:00:45.778789 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.fernet_utils [None req-42227b2b-8a24-4dbc-9b15-325e82ef944b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92057) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:00:45.833286 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=86,keystone:INSERT=4,keystone:DELETE=10 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 13:00:45.850700 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 955/1908] 10.222.0.30 () {58 vars in 957 bytes} [Mon May 23 13:00:45 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 95 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) May 23 13:00:45.851980 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 954/1909] 10.222.0.30 () {60 vars in 1152 bytes} [Mon May 23 13:00:45 2022] POST /identity/v3/projects => generated 379 bytes in 180 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) May 23 13:00:45.856441 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42227b2b-8a24-4dbc-9b15-325e82ef944b None None] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:00:45.857987 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.fernet_utils [None req-42227b2b-8a24-4dbc-9b15-325e82ef944b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92057) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:00:45.862575 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62b74baa-4556-4e8f-8955-5b03dadaedff admin admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:00:45.864356 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.fernet_utils [None req-62b74baa-4556-4e8f-8955-5b03dadaedff admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92056) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:00:45.872748 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42227b2b-8a24-4dbc-9b15-325e82ef944b None None] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:00:45.924081 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62b74baa-4556-4e8f-8955-5b03dadaedff admin admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:00:45.927786 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c32decf-6562-49b8-ba38-3026bb59d01a service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2eb83ca38e79449095f3a2f947bbb06a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9c389ae65aaa4542bb5a5c4fd0bacf5b', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:00:45.928430 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c32decf-6562-49b8-ba38-3026bb59d01a service neutron] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:00:45.928712 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c32decf-6562-49b8-ba38-3026bb59d01a service neutron] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:00:45.928981 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c32decf-6562-49b8-ba38-3026bb59d01a service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:00:45.933508 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d45727a1-d3d7-4cbb-bb52-8e531c939253 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:00:45.934223 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d45727a1-d3d7-4cbb-bb52-8e531c939253 admin admin] REQUEST_METHOD: `POST` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:00:45.934503 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d45727a1-d3d7-4cbb-bb52-8e531c939253 admin admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:00:45.934778 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d45727a1-d3d7-4cbb-bb52-8e531c939253 admin admin] PATH_INFO: `/v3/users` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:00:45.935470 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d45727a1-d3d7-4cbb-bb52-8e531c939253 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:00:45.937431 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d45727a1-d3d7-4cbb-bb52-8e531c939253 admin admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:00:45.939017 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c32decf-6562-49b8-ba38-3026bb59d01a service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:00:45.940434 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c32decf-6562-49b8-ba38-3026bb59d01a service neutron] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:00:45.950746 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 955/1910] 10.222.0.30 () {60 vars in 1365 bytes} [Mon May 23 13:00:45 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 89 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) May 23 13:00:46.016810 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 956/1911] 10.222.0.30 () {60 vars in 1143 bytes} [Mon May 23 13:00:45 2022] POST /identity/v3/users => generated 296 bytes in 158 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) May 23 13:00:46.017978 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c32decf-6562-49b8-ba38-3026bb59d01a service neutron] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:00:46.028420 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c32decf-6562-49b8-ba38-3026bb59d01a service neutron] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:00:46.033147 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dbb51c0a-6e66-43c0-8c35-b24edc6b36ae admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:00:46.033832 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbb51c0a-6e66-43c0-8c35-b24edc6b36ae admin admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:00:46.034153 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbb51c0a-6e66-43c0-8c35-b24edc6b36ae admin admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:00:46.034426 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbb51c0a-6e66-43c0-8c35-b24edc6b36ae admin admin] PATH_INFO: `/v3/roles` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:00:46.035164 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dbb51c0a-6e66-43c0-8c35-b24edc6b36ae admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:00:46.036743 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dbb51c0a-6e66-43c0-8c35-b24edc6b36ae admin admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:00:46.047927 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 956/1912] 10.222.0.30 () {58 vars in 1121 bytes} [Mon May 23 13:00:46 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) May 23 13:00:46.052089 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d45727a1-d3d7-4cbb-bb52-8e531c939253 admin admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:00:46.062008 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d45727a1-d3d7-4cbb-bb52-8e531c939253 admin admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:00:46.066802 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a4e8e1f-3805-413a-b99a-5c5577a24bf0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:00:46.067704 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a4e8e1f-3805-413a-b99a-5c5577a24bf0 admin admin] REQUEST_METHOD: `PUT` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:00:46.068013 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a4e8e1f-3805-413a-b99a-5c5577a24bf0 admin admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:00:46.068285 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a4e8e1f-3805-413a-b99a-5c5577a24bf0 admin admin] PATH_INFO: `/v3/projects/c9f491f5b03e43cca9d4469a0129f84e/users/8a3a22b47b7041358020098e10951ca8/roles/bc3c3c9d654940cbaa5177a6be23d1bc` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:00:46.083435 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a4e8e1f-3805-413a-b99a-5c5577a24bf0 admin admin] RBAC: Authorizing `identity:create_grant(project_id=c9f491f5b03e43cca9d4469a0129f84e, user_id=8a3a22b47b7041358020098e10951ca8, role_id=bc3c3c9d654940cbaa5177a6be23d1bc)` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:00:46.086224 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a4e8e1f-3805-413a-b99a-5c5577a24bf0 admin admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:00:46.147074 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 957/1913] 10.222.0.30 () {60 vars in 1482 bytes} [Mon May 23 13:00:46 2022] PUT /identity/v3/projects/c9f491f5b03e43cca9d4469a0129f84e/users/8a3a22b47b7041358020098e10951ca8/roles/bc3c3c9d654940cbaa5177a6be23d1bc => generated 0 bytes in 95 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) May 23 13:00:46.153832 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dbb51c0a-6e66-43c0-8c35-b24edc6b36ae admin admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:00:46.164179 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dbb51c0a-6e66-43c0-8c35-b24edc6b36ae admin admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:00:46.169818 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 13:00:46.225522 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0ce54e1-96c0-40ed-b6b5-756dc2f97714 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:00:46.226230 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0ce54e1-96c0-40ed-b6b5-756dc2f97714 admin admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:00:46.226518 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0ce54e1-96c0-40ed-b6b5-756dc2f97714 admin admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:00:46.226795 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0ce54e1-96c0-40ed-b6b5-756dc2f97714 admin admin] PATH_INFO: `/v3/roles` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:00:46.227534 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0ce54e1-96c0-40ed-b6b5-756dc2f97714 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:00:46.230698 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0ce54e1-96c0-40ed-b6b5-756dc2f97714 admin admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:00:46.240048 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 957/1914] 10.222.0.30 () {58 vars in 1121 bytes} [Mon May 23 13:00:46 2022] GET /identity/v3/roles => generated 1420 bytes in 87 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) May 23 13:00:46.244214 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a4e8e1f-3805-413a-b99a-5c5577a24bf0 admin admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:00:46.255173 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a4e8e1f-3805-413a-b99a-5c5577a24bf0 admin admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:00:46.260713 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c6458de-34d4-447e-8ebf-dbf50bfb8ab1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:00:46.261716 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c6458de-34d4-447e-8ebf-dbf50bfb8ab1 admin admin] REQUEST_METHOD: `PUT` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:00:46.261927 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c6458de-34d4-447e-8ebf-dbf50bfb8ab1 admin admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:00:46.262470 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c6458de-34d4-447e-8ebf-dbf50bfb8ab1 admin admin] PATH_INFO: `/v3/projects/c9f491f5b03e43cca9d4469a0129f84e/users/8a3a22b47b7041358020098e10951ca8/roles/bc3c3c9d654940cbaa5177a6be23d1bc` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:00:46.264591 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c6458de-34d4-447e-8ebf-dbf50bfb8ab1 admin admin] RBAC: Authorizing `identity:create_grant(project_id=c9f491f5b03e43cca9d4469a0129f84e, user_id=8a3a22b47b7041358020098e10951ca8, role_id=bc3c3c9d654940cbaa5177a6be23d1bc)` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:00:46.267381 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c6458de-34d4-447e-8ebf-dbf50bfb8ab1 admin admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:00:46.278750 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 958/1915] 10.222.0.30 () {60 vars in 1482 bytes} [Mon May 23 13:00:46 2022] PUT /identity/v3/projects/c9f491f5b03e43cca9d4469a0129f84e/users/8a3a22b47b7041358020098e10951ca8/roles/bc3c3c9d654940cbaa5177a6be23d1bc => generated 0 bytes in 35 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) May 23 13:00:46.289003 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23b884e7-bf22-4d4a-89f4-d4557197cd42 None None] REQUEST_METHOD: `POST` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:00:46.289305 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23b884e7-bf22-4d4a-89f4-d4557197cd42 None None] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:00:46.289592 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23b884e7-bf22-4d4a-89f4-d4557197cd42 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:00:46.326394 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.auth.core [None req-23b884e7-bf22-4d4a-89f4-d4557197cd42 None None] MFA Rules not processed for user `8a3a22b47b7041358020098e10951ca8`. Rule list: `[]` (Enabled: `True`). {{(pid=92056) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 13:00:46.326394 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.fernet_utils [None req-23b884e7-bf22-4d4a-89f4-d4557197cd42 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92056) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:00:46.359761 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c6458de-34d4-447e-8ebf-dbf50bfb8ab1 admin admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:00:46.362005 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.fernet_utils [None req-6c6458de-34d4-447e-8ebf-dbf50bfb8ab1 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92057) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:00:46.421120 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 958/1916] 10.222.0.30 () {58 vars in 957 bytes} [Mon May 23 13:00:46 2022] POST /identity/v3/auth/tokens => generated 3313 bytes in 128 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) May 23 13:00:46.428630 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23b884e7-bf22-4d4a-89f4-d4557197cd42 None None] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:00:46.436807 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.fernet_utils [None req-23b884e7-bf22-4d4a-89f4-d4557197cd42 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92056) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:00:46.456321 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c6458de-34d4-447e-8ebf-dbf50bfb8ab1 admin admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:00:46.464201 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7bc32914-ef46-4ee5-beb4-1e6008ca62e3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:00:46.464992 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bc32914-ef46-4ee5-beb4-1e6008ca62e3 admin admin] REQUEST_METHOD: `DELETE` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:00:46.465270 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bc32914-ef46-4ee5-beb4-1e6008ca62e3 admin admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:00:46.465539 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bc32914-ef46-4ee5-beb4-1e6008ca62e3 admin admin] PATH_INFO: `/v3/projects/65d681ef71ba4ac4af80d806a3076cbd` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:00:46.466801 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7bc32914-ef46-4ee5-beb4-1e6008ca62e3 admin admin] RBAC: Authorizing `identity:delete_project(project_id=65d681ef71ba4ac4af80d806a3076cbd)` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:00:46.473187 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7bc32914-ef46-4ee5-beb4-1e6008ca62e3 admin admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:00:46.516714 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23b884e7-bf22-4d4a-89f4-d4557197cd42 None None] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:00:46.520567 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6ae197c-fc62-4891-b0c6-5468eb69a3e5 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1666097f8ed545c6b0e4809c93e4775a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9c389ae65aaa4542bb5a5c4fd0bacf5b', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:00:46.521210 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6ae197c-fc62-4891-b0c6-5468eb69a3e5 service cinder] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:00:46.521495 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6ae197c-fc62-4891-b0c6-5468eb69a3e5 service cinder] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:00:46.521776 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6ae197c-fc62-4891-b0c6-5468eb69a3e5 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:00:46.531936 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6ae197c-fc62-4891-b0c6-5468eb69a3e5 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:00:46.533363 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6ae197c-fc62-4891-b0c6-5468eb69a3e5 service cinder] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:00:46.543789 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 959/1917] 10.222.0.30 () {60 vars in 1365 bytes} [Mon May 23 13:00:46 2022] GET /identity/v3/auth/tokens => generated 3313 bytes in 116 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) May 23 13:00:46.584855 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [None req-7bc32914-ef46-4ee5-beb4-1e6008ca62e3 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '65d681ef71ba4ac4af80d806a3076cbd', 'request_id': 'req-7bc32914-ef46-4ee5-beb4-1e6008ca62e3'} {{(pid=92057) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:00:46.586814 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=90,keystone:INSERT=5,keystone:DELETE=5 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 13:00:46.591502 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [None req-7bc32914-ef46-4ee5-beb4-1e6008ca62e3 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '65d681ef71ba4ac4af80d806a3076cbd', 'request_id': 'req-7bc32914-ef46-4ee5-beb4-1e6008ca62e3'} {{(pid=92057) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:00:46.670711 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6ae197c-fc62-4891-b0c6-5468eb69a3e5 service cinder] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:00:46.672121 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.fernet_utils [None req-a6ae197c-fc62-4891-b0c6-5468eb69a3e5 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92056) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:00:46.684374 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6ae197c-fc62-4891-b0c6-5468eb69a3e5 service cinder] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:00:46.702341 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [None req-7bc32914-ef46-4ee5-beb4-1e6008ca62e3 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '65d681ef71ba4ac4af80d806a3076cbd', 'request_id': 'req-7bc32914-ef46-4ee5-beb4-1e6008ca62e3'} {{(pid=92057) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:00:46.718576 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [None req-7bc32914-ef46-4ee5-beb4-1e6008ca62e3 admin admin] The token cache is being invalidate because project 65d681ef71ba4ac4af80d806a3076cbd was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=92057) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} May 23 13:00:46.718906 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [None req-7bc32914-ef46-4ee5-beb4-1e6008ca62e3 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=92057) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:00:46.720542 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [None req-7bc32914-ef46-4ee5-beb4-1e6008ca62e3 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=92057) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:00:46.725421 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 959/1918] 10.222.0.30 () {58 vars in 1232 bytes} [Mon May 23 13:00:46 2022] DELETE /identity/v3/projects/65d681ef71ba4ac4af80d806a3076cbd => generated 0 bytes in 366 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) May 23 13:00:46.729044 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7bc32914-ef46-4ee5-beb4-1e6008ca62e3 admin admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:00:46.730988 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.fernet_utils [None req-7bc32914-ef46-4ee5-beb4-1e6008ca62e3 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92057) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:00:46.765370 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=100 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 13:00:46.798723 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a14d28b-9dc2-4a18-b48d-3976cb533311 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1666097f8ed545c6b0e4809c93e4775a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9c389ae65aaa4542bb5a5c4fd0bacf5b', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:00:46.800927 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a14d28b-9dc2-4a18-b48d-3976cb533311 service cinder] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:00:46.801407 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a14d28b-9dc2-4a18-b48d-3976cb533311 service cinder] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:00:46.801871 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a14d28b-9dc2-4a18-b48d-3976cb533311 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:00:46.804281 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.fernet_utils [None req-1a14d28b-9dc2-4a18-b48d-3976cb533311 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92056) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:00:46.867440 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7bc32914-ef46-4ee5-beb4-1e6008ca62e3 admin admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:00:46.873649 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eaa69122-d8ea-403c-b2c8-217c42376413 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:00:46.875151 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eaa69122-d8ea-403c-b2c8-217c42376413 admin admin] REQUEST_METHOD: `DELETE` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:00:46.875534 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eaa69122-d8ea-403c-b2c8-217c42376413 admin admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:00:46.875887 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eaa69122-d8ea-403c-b2c8-217c42376413 admin admin] PATH_INFO: `/v3/users/3321e1372371457c9f19cdde83bb1ccc` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:00:46.877345 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eaa69122-d8ea-403c-b2c8-217c42376413 admin admin] RBAC: Authorizing `identity:delete_user(user_id=3321e1372371457c9f19cdde83bb1ccc)` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:00:46.879645 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eaa69122-d8ea-403c-b2c8-217c42376413 admin admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:00:46.903302 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a14d28b-9dc2-4a18-b48d-3976cb533311 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:00:46.904805 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a14d28b-9dc2-4a18-b48d-3976cb533311 service cinder] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:00:46.915844 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 960/1919] 10.222.0.30 () {60 vars in 1365 bytes} [Mon May 23 13:00:46 2022] GET /identity/v3/auth/tokens => generated 3244 bytes in 245 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) May 23 13:00:47.162662 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [None req-eaa69122-d8ea-403c-b2c8-217c42376413 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '3321e1372371457c9f19cdde83bb1ccc'} {{(pid=92057) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:00:47.248003 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [None req-eaa69122-d8ea-403c-b2c8-217c42376413 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '3321e1372371457c9f19cdde83bb1ccc'} {{(pid=92057) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:00:47.255994 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [None req-eaa69122-d8ea-403c-b2c8-217c42376413 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '3321e1372371457c9f19cdde83bb1ccc'} {{(pid=92057) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:00:47.256734 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [None req-eaa69122-d8ea-403c-b2c8-217c42376413 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '3321e1372371457c9f19cdde83bb1ccc'} {{(pid=92057) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:00:47.274257 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [None req-eaa69122-d8ea-403c-b2c8-217c42376413 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '3321e1372371457c9f19cdde83bb1ccc'} {{(pid=92057) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:00:47.283707 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 960/1920] 10.222.0.30 () {58 vars in 1223 bytes} [Mon May 23 13:00:46 2022] DELETE /identity/v3/users/3321e1372371457c9f19cdde83bb1ccc => generated 0 bytes in 555 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) May 23 13:00:47.835049 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a14d28b-9dc2-4a18-b48d-3976cb533311 service cinder] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:00:47.847154 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a14d28b-9dc2-4a18-b48d-3976cb533311 service cinder] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:00:47.933996 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f9b9101-c22e-4fa0-95ab-f767106dfc42 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:00:47.934913 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f9b9101-c22e-4fa0-95ab-f767106dfc42 admin admin] REQUEST_METHOD: `DELETE` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:00:47.935198 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f9b9101-c22e-4fa0-95ab-f767106dfc42 admin admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:00:47.935487 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f9b9101-c22e-4fa0-95ab-f767106dfc42 admin admin] PATH_INFO: `/v3/projects/b0e5a990b2584c4db0a06a4308eb0864` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:00:47.938344 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f9b9101-c22e-4fa0-95ab-f767106dfc42 admin admin] RBAC: Authorizing `identity:delete_project(project_id=b0e5a990b2584c4db0a06a4308eb0864)` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:00:47.940284 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f9b9101-c22e-4fa0-95ab-f767106dfc42 admin admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:00:48.111844 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-6f9b9101-c22e-4fa0-95ab-f767106dfc42 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'b0e5a990b2584c4db0a06a4308eb0864', 'request_id': 'req-6f9b9101-c22e-4fa0-95ab-f767106dfc42'} {{(pid=92056) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:00:48.183148 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-6f9b9101-c22e-4fa0-95ab-f767106dfc42 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'b0e5a990b2584c4db0a06a4308eb0864', 'request_id': 'req-6f9b9101-c22e-4fa0-95ab-f767106dfc42'} {{(pid=92056) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:00:48.190352 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-6f9b9101-c22e-4fa0-95ab-f767106dfc42 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'b0e5a990b2584c4db0a06a4308eb0864', 'request_id': 'req-6f9b9101-c22e-4fa0-95ab-f767106dfc42'} {{(pid=92056) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:00:48.202082 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-6f9b9101-c22e-4fa0-95ab-f767106dfc42 admin admin] The token cache is being invalidate because project b0e5a990b2584c4db0a06a4308eb0864 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=92056) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} May 23 13:00:48.202367 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-6f9b9101-c22e-4fa0-95ab-f767106dfc42 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=92056) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:00:48.203015 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-6f9b9101-c22e-4fa0-95ab-f767106dfc42 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=92056) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:00:48.204650 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 961/1921] 10.222.0.30 () {58 vars in 1232 bytes} [Mon May 23 13:00:47 2022] DELETE /identity/v3/projects/b0e5a990b2584c4db0a06a4308eb0864 => generated 0 bytes in 372 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) May 23 13:00:48.209865 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b6dcb3e-d5ed-4885-b25b-a2b32f7532b9 None None] REQUEST_METHOD: `POST` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:00:48.210181 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b6dcb3e-d5ed-4885-b25b-a2b32f7532b9 None None] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:00:48.210461 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b6dcb3e-d5ed-4885-b25b-a2b32f7532b9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:00:48.229726 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.auth.core [None req-8b6dcb3e-d5ed-4885-b25b-a2b32f7532b9 None None] MFA Rules not processed for user `0f7119f8fa5c4445977671140ded91a4`. Rule list: `[]` (Enabled: `True`). {{(pid=92057) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 13:00:48.231005 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.fernet_utils [None req-8b6dcb3e-d5ed-4885-b25b-a2b32f7532b9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92057) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:00:48.285519 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=86,keystone:DELETE=12,keystone:INSERT=2 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 13:00:48.293800 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 961/1922] 10.222.0.30 () {58 vars in 957 bytes} [Mon May 23 13:00:48 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) May 23 13:00:48.302519 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5d830e3-93d7-4b2c-ad6d-d9e003339ef6 None None] REQUEST_METHOD: `POST` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:00:48.302810 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5d830e3-93d7-4b2c-ad6d-d9e003339ef6 None None] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:00:48.303091 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5d830e3-93d7-4b2c-ad6d-d9e003339ef6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:00:48.322404 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.auth.core [None req-c5d830e3-93d7-4b2c-ad6d-d9e003339ef6 None None] MFA Rules not processed for user `0f7119f8fa5c4445977671140ded91a4`. Rule list: `[]` (Enabled: `True`). {{(pid=92056) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 13:00:48.324476 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.fernet_utils [None req-c5d830e3-93d7-4b2c-ad6d-d9e003339ef6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92056) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:00:48.332322 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 962/1923] 10.222.0.30 () {58 vars in 957 bytes} [Mon May 23 13:00:48 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) May 23 13:00:48.336667 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b6dcb3e-d5ed-4885-b25b-a2b32f7532b9 None None] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:00:48.346454 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b6dcb3e-d5ed-4885-b25b-a2b32f7532b9 None None] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:00:48.351169 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58b3e012-4642-40f6-b26f-c6d4d6754c71 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:00:48.352098 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58b3e012-4642-40f6-b26f-c6d4d6754c71 admin admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:00:48.353593 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58b3e012-4642-40f6-b26f-c6d4d6754c71 admin admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:00:48.353593 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58b3e012-4642-40f6-b26f-c6d4d6754c71 admin admin] PATH_INFO: `/v3/domains` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:00:48.353593 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58b3e012-4642-40f6-b26f-c6d4d6754c71 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=92057) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} May 23 13:00:48.354454 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58b3e012-4642-40f6-b26f-c6d4d6754c71 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:00:48.355891 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58b3e012-4642-40f6-b26f-c6d4d6754c71 admin admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:00:48.368133 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 962/1924] 10.222.0.30 () {58 vars in 1152 bytes} [Mon May 23 13:00:48 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) May 23 13:00:48.373389 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5d830e3-93d7-4b2c-ad6d-d9e003339ef6 None None] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:00:48.383598 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5d830e3-93d7-4b2c-ad6d-d9e003339ef6 None None] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:00:48.399397 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-519afec0-7b5b-4e46-8dc1-7005c177fdb6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:00:48.400309 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-519afec0-7b5b-4e46-8dc1-7005c177fdb6 admin admin] REQUEST_METHOD: `POST` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:00:48.401571 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-519afec0-7b5b-4e46-8dc1-7005c177fdb6 admin admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:00:48.402506 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-519afec0-7b5b-4e46-8dc1-7005c177fdb6 admin admin] PATH_INFO: `/v3/projects` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:00:48.404501 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-519afec0-7b5b-4e46-8dc1-7005c177fdb6 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:00:48.408195 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-519afec0-7b5b-4e46-8dc1-7005c177fdb6 admin admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:00:48.433147 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 13:00:48.518338 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 963/1925] 10.222.0.30 () {60 vars in 1152 bytes} [Mon May 23 13:00:48 2022] POST /identity/v3/projects => generated 367 bytes in 145 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) May 23 13:00:48.523990 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58b3e012-4642-40f6-b26f-c6d4d6754c71 admin admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:00:48.537259 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58b3e012-4642-40f6-b26f-c6d4d6754c71 admin admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:00:48.616619 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e22a83e6-9bcf-4868-a7c2-16417228f26a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:00:48.617292 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e22a83e6-9bcf-4868-a7c2-16417228f26a admin admin] REQUEST_METHOD: `POST` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:00:48.617573 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e22a83e6-9bcf-4868-a7c2-16417228f26a admin admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:00:48.617853 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e22a83e6-9bcf-4868-a7c2-16417228f26a admin admin] PATH_INFO: `/v3/users` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:00:48.618565 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e22a83e6-9bcf-4868-a7c2-16417228f26a admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:00:48.621441 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e22a83e6-9bcf-4868-a7c2-16417228f26a admin admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:00:48.713921 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 963/1926] 10.222.0.30 () {60 vars in 1143 bytes} [Mon May 23 13:00:48 2022] POST /identity/v3/users => generated 284 bytes in 188 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) May 23 13:00:48.718821 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-519afec0-7b5b-4e46-8dc1-7005c177fdb6 admin admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:00:48.730487 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-519afec0-7b5b-4e46-8dc1-7005c177fdb6 admin admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:00:48.740582 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d224f31-862d-4f2e-8a52-d87e6be303f4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:00:48.744551 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d224f31-862d-4f2e-8a52-d87e6be303f4 admin admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:00:48.744551 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d224f31-862d-4f2e-8a52-d87e6be303f4 admin admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:00:48.744551 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d224f31-862d-4f2e-8a52-d87e6be303f4 admin admin] PATH_INFO: `/v3/roles` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:00:48.744551 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d224f31-862d-4f2e-8a52-d87e6be303f4 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:00:48.744551 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d224f31-862d-4f2e-8a52-d87e6be303f4 admin admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:00:48.761179 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 964/1927] 10.222.0.30 () {58 vars in 1121 bytes} [Mon May 23 13:00:48 2022] GET /identity/v3/roles => generated 1420 bytes in 45 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) May 23 13:00:48.765456 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e22a83e6-9bcf-4868-a7c2-16417228f26a admin admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:00:48.777437 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e22a83e6-9bcf-4868-a7c2-16417228f26a admin admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:00:48.782274 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad08bff3-e653-425d-b291-e80a95c371cb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:00:48.782931 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad08bff3-e653-425d-b291-e80a95c371cb admin admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:00:48.783215 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad08bff3-e653-425d-b291-e80a95c371cb admin admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:00:48.783483 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad08bff3-e653-425d-b291-e80a95c371cb admin admin] PATH_INFO: `/v3/roles` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:00:48.784206 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad08bff3-e653-425d-b291-e80a95c371cb admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:00:48.785848 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad08bff3-e653-425d-b291-e80a95c371cb admin admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:00:48.797537 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 964/1928] 10.222.0.30 () {58 vars in 1121 bytes} [Mon May 23 13:00:48 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) May 23 13:00:48.801757 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d224f31-862d-4f2e-8a52-d87e6be303f4 admin admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:00:48.812118 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d224f31-862d-4f2e-8a52-d87e6be303f4 admin admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:00:48.822243 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21a8d03f-c824-4c64-b7bb-a32d83d9e54a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:00:48.823518 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21a8d03f-c824-4c64-b7bb-a32d83d9e54a admin admin] REQUEST_METHOD: `PUT` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:00:48.824017 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21a8d03f-c824-4c64-b7bb-a32d83d9e54a admin admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:00:48.824135 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21a8d03f-c824-4c64-b7bb-a32d83d9e54a admin admin] PATH_INFO: `/v3/projects/e805a9b04cbb4815a6ff030da2f110fb/users/a1cc3a7861c44e3688155c7066fdfb6b/roles/80fb22507c9a4ff09d44d8f12f6fff1a` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:00:48.845048 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-21a8d03f-c824-4c64-b7bb-a32d83d9e54a admin admin] RBAC: Authorizing `identity:create_grant(project_id=e805a9b04cbb4815a6ff030da2f110fb, user_id=a1cc3a7861c44e3688155c7066fdfb6b, role_id=80fb22507c9a4ff09d44d8f12f6fff1a)` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:00:48.845048 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-21a8d03f-c824-4c64-b7bb-a32d83d9e54a admin admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:00:48.956611 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 965/1929] 10.222.0.30 () {60 vars in 1482 bytes} [Mon May 23 13:00:48 2022] PUT /identity/v3/projects/e805a9b04cbb4815a6ff030da2f110fb/users/a1cc3a7861c44e3688155c7066fdfb6b/roles/80fb22507c9a4ff09d44d8f12f6fff1a => generated 0 bytes in 155 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) May 23 13:00:48.967395 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a89777a9-108b-410c-91ca-2911210dc4a7 None None] REQUEST_METHOD: `POST` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:00:48.967690 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a89777a9-108b-410c-91ca-2911210dc4a7 None None] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:00:48.967975 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a89777a9-108b-410c-91ca-2911210dc4a7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:00:48.996464 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.auth.core [None req-a89777a9-108b-410c-91ca-2911210dc4a7 None None] MFA Rules not processed for user `a1cc3a7861c44e3688155c7066fdfb6b`. Rule list: `[]` (Enabled: `True`). {{(pid=92057) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 13:00:48.997708 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.fernet_utils [None req-a89777a9-108b-410c-91ca-2911210dc4a7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92057) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:00:49.061943 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 965/1930] 10.222.0.30 () {58 vars in 957 bytes} [Mon May 23 13:00:48 2022] POST /identity/v3/auth/tokens => generated 3234 bytes in 95 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) May 23 13:00:49.066389 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21a8d03f-c824-4c64-b7bb-a32d83d9e54a admin admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:00:49.127811 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21a8d03f-c824-4c64-b7bb-a32d83d9e54a admin admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:00:49.132474 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1c2f976-9e39-482c-ad44-de1ed1f439d5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:00:49.133120 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1c2f976-9e39-482c-ad44-de1ed1f439d5 admin admin] REQUEST_METHOD: `POST` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:00:49.133401 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1c2f976-9e39-482c-ad44-de1ed1f439d5 admin admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:00:49.133674 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1c2f976-9e39-482c-ad44-de1ed1f439d5 admin admin] PATH_INFO: `/v3/projects` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:00:49.134408 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1c2f976-9e39-482c-ad44-de1ed1f439d5 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:00:49.136905 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1c2f976-9e39-482c-ad44-de1ed1f439d5 admin admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:00:49.225183 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 966/1931] 10.222.0.30 () {60 vars in 1152 bytes} [Mon May 23 13:00:49 2022] POST /identity/v3/projects => generated 367 bytes in 159 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) May 23 13:00:49.229905 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a89777a9-108b-410c-91ca-2911210dc4a7 None None] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:00:49.240668 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a89777a9-108b-410c-91ca-2911210dc4a7 None None] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:00:49.289399 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 13:00:49.296880 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2789cf27-33fd-4414-a404-f5da8a328226 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:00:49.297533 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2789cf27-33fd-4414-a404-f5da8a328226 admin admin] REQUEST_METHOD: `POST` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:00:49.297814 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2789cf27-33fd-4414-a404-f5da8a328226 admin admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:00:49.298121 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2789cf27-33fd-4414-a404-f5da8a328226 admin admin] PATH_INFO: `/v3/users` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:00:49.298827 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2789cf27-33fd-4414-a404-f5da8a328226 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:00:49.300733 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2789cf27-33fd-4414-a404-f5da8a328226 admin admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:00:49.373290 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 966/1932] 10.222.0.30 () {60 vars in 1143 bytes} [Mon May 23 13:00:49 2022] POST /identity/v3/users => generated 290 bytes in 144 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) May 23 13:00:49.377383 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1c2f976-9e39-482c-ad44-de1ed1f439d5 admin admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:00:49.388195 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1c2f976-9e39-482c-ad44-de1ed1f439d5 admin admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:00:49.392854 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-949b4899-f5d4-4e3b-a81f-a80a530a1bd9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:00:49.393510 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-949b4899-f5d4-4e3b-a81f-a80a530a1bd9 admin admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:00:49.393796 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-949b4899-f5d4-4e3b-a81f-a80a530a1bd9 admin admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:00:49.394108 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-949b4899-f5d4-4e3b-a81f-a80a530a1bd9 admin admin] PATH_INFO: `/v3/roles` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:00:49.394837 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-949b4899-f5d4-4e3b-a81f-a80a530a1bd9 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:00:49.396446 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-949b4899-f5d4-4e3b-a81f-a80a530a1bd9 admin admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:00:49.407116 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 967/1933] 10.222.0.30 () {58 vars in 1121 bytes} [Mon May 23 13:00:49 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) May 23 13:00:49.411195 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2789cf27-33fd-4414-a404-f5da8a328226 admin admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:00:49.421581 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2789cf27-33fd-4414-a404-f5da8a328226 admin admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:00:49.426269 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8678d581-c565-4af2-80b5-55170cc6c3e4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:00:49.427160 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8678d581-c565-4af2-80b5-55170cc6c3e4 admin admin] REQUEST_METHOD: `PUT` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:00:49.427438 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8678d581-c565-4af2-80b5-55170cc6c3e4 admin admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:00:49.427706 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8678d581-c565-4af2-80b5-55170cc6c3e4 admin admin] PATH_INFO: `/v3/projects/cffd0d83c01b45738651a4a7dc2aacb0/users/e592dac401c948ce99cd8797af3b7dd5/roles/bc3c3c9d654940cbaa5177a6be23d1bc` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:00:49.442901 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8678d581-c565-4af2-80b5-55170cc6c3e4 admin admin] RBAC: Authorizing `identity:create_grant(project_id=cffd0d83c01b45738651a4a7dc2aacb0, user_id=e592dac401c948ce99cd8797af3b7dd5, role_id=bc3c3c9d654940cbaa5177a6be23d1bc)` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:00:49.445554 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8678d581-c565-4af2-80b5-55170cc6c3e4 admin admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:00:49.510184 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 967/1934] 10.222.0.30 () {60 vars in 1482 bytes} [Mon May 23 13:00:49 2022] PUT /identity/v3/projects/cffd0d83c01b45738651a4a7dc2aacb0/users/e592dac401c948ce99cd8797af3b7dd5/roles/bc3c3c9d654940cbaa5177a6be23d1bc => generated 0 bytes in 99 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) May 23 13:00:49.514438 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-949b4899-f5d4-4e3b-a81f-a80a530a1bd9 admin admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:00:49.525221 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-949b4899-f5d4-4e3b-a81f-a80a530a1bd9 admin admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:00:49.581478 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e96858e-2c82-4750-bf93-7150e96b87fd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:00:49.582190 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e96858e-2c82-4750-bf93-7150e96b87fd admin admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:00:49.582480 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e96858e-2c82-4750-bf93-7150e96b87fd admin admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:00:49.582763 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e96858e-2c82-4750-bf93-7150e96b87fd admin admin] PATH_INFO: `/v3/roles` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:00:49.583490 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e96858e-2c82-4750-bf93-7150e96b87fd admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:00:49.585097 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e96858e-2c82-4750-bf93-7150e96b87fd admin admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:00:49.595326 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 968/1935] 10.222.0.30 () {58 vars in 1121 bytes} [Mon May 23 13:00:49 2022] GET /identity/v3/roles => generated 1420 bytes in 81 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) May 23 13:00:49.599437 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8678d581-c565-4af2-80b5-55170cc6c3e4 admin admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:00:49.610293 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8678d581-c565-4af2-80b5-55170cc6c3e4 admin admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:00:49.614977 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9136e03-2bd6-43e3-828f-10fd31c472b3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:00:49.615859 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9136e03-2bd6-43e3-828f-10fd31c472b3 admin admin] REQUEST_METHOD: `PUT` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:00:49.616147 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9136e03-2bd6-43e3-828f-10fd31c472b3 admin admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:00:49.616426 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9136e03-2bd6-43e3-828f-10fd31c472b3 admin admin] PATH_INFO: `/v3/projects/cffd0d83c01b45738651a4a7dc2aacb0/users/e592dac401c948ce99cd8797af3b7dd5/roles/bc3c3c9d654940cbaa5177a6be23d1bc` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:00:49.618516 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9136e03-2bd6-43e3-828f-10fd31c472b3 admin admin] RBAC: Authorizing `identity:create_grant(project_id=cffd0d83c01b45738651a4a7dc2aacb0, user_id=e592dac401c948ce99cd8797af3b7dd5, role_id=bc3c3c9d654940cbaa5177a6be23d1bc)` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:00:49.621306 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9136e03-2bd6-43e3-828f-10fd31c472b3 admin admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:00:49.639148 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 968/1936] 10.222.0.30 () {60 vars in 1482 bytes} [Mon May 23 13:00:49 2022] PUT /identity/v3/projects/cffd0d83c01b45738651a4a7dc2aacb0/users/e592dac401c948ce99cd8797af3b7dd5/roles/bc3c3c9d654940cbaa5177a6be23d1bc => generated 0 bytes in 34 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) May 23 13:00:49.644293 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d75eda90-59f8-4df7-94bf-eb3163164bce None None] REQUEST_METHOD: `POST` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:00:49.644583 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d75eda90-59f8-4df7-94bf-eb3163164bce None None] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:00:49.644859 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d75eda90-59f8-4df7-94bf-eb3163164bce None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:00:49.675677 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.auth.core [None req-d75eda90-59f8-4df7-94bf-eb3163164bce None None] MFA Rules not processed for user `e592dac401c948ce99cd8797af3b7dd5`. Rule list: `[]` (Enabled: `True`). {{(pid=92056) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 13:00:49.676877 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.fernet_utils [None req-d75eda90-59f8-4df7-94bf-eb3163164bce None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92056) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:00:49.695367 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 13:00:49.757048 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 969/1937] 10.222.0.30 () {58 vars in 957 bytes} [Mon May 23 13:00:49 2022] POST /identity/v3/auth/tokens => generated 3301 bytes in 114 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) May 23 13:00:49.769243 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9136e03-2bd6-43e3-828f-10fd31c472b3 admin admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:00:49.770958 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.fernet_utils [None req-f9136e03-2bd6-43e3-828f-10fd31c472b3 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92057) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:00:49.843103 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9136e03-2bd6-43e3-828f-10fd31c472b3 admin admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:00:49.846837 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73552497-3490-43c3-8470-420f942bd92a service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1666097f8ed545c6b0e4809c93e4775a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9c389ae65aaa4542bb5a5c4fd0bacf5b', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:00:49.847477 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73552497-3490-43c3-8470-420f942bd92a service cinder] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:00:49.847756 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73552497-3490-43c3-8470-420f942bd92a service cinder] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:00:49.848026 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73552497-3490-43c3-8470-420f942bd92a service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:00:49.860711 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-73552497-3490-43c3-8470-420f942bd92a service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:00:49.862451 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-73552497-3490-43c3-8470-420f942bd92a service cinder] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:00:49.872647 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 969/1938] 10.222.0.30 () {60 vars in 1365 bytes} [Mon May 23 13:00:49 2022] GET /identity/v3/auth/tokens => generated 3301 bytes in 104 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) May 23 13:00:50.110734 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d75eda90-59f8-4df7-94bf-eb3163164bce None None] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:00:50.112270 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.fernet_utils [None req-d75eda90-59f8-4df7-94bf-eb3163164bce None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92056) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:00:50.126140 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d75eda90-59f8-4df7-94bf-eb3163164bce None None] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:00:50.129716 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e91e7463-540a-4158-8393-e8a85664c9ad service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1666097f8ed545c6b0e4809c93e4775a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9c389ae65aaa4542bb5a5c4fd0bacf5b', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:00:50.130334 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e91e7463-540a-4158-8393-e8a85664c9ad service cinder] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:00:50.133863 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e91e7463-540a-4158-8393-e8a85664c9ad service cinder] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:00:50.133863 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e91e7463-540a-4158-8393-e8a85664c9ad service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:00:50.196460 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e91e7463-540a-4158-8393-e8a85664c9ad service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:00:50.197959 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e91e7463-540a-4158-8393-e8a85664c9ad service cinder] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:00:50.206829 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 970/1939] 10.222.0.30 () {60 vars in 1365 bytes} [Mon May 23 13:00:50 2022] GET /identity/v3/auth/tokens => generated 3234 bytes in 97 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) May 23 13:00:57.751786 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73552497-3490-43c3-8470-420f942bd92a service cinder] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:00:57.757895 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.fernet_utils [None req-73552497-3490-43c3-8470-420f942bd92a service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92057) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:00:57.821864 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73552497-3490-43c3-8470-420f942bd92a service cinder] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:00:57.827950 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b61e43e-e2df-48f2-a618-88083cfbee59 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:00:57.828658 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b61e43e-e2df-48f2-a618-88083cfbee59 admin admin] REQUEST_METHOD: `DELETE` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:00:57.829031 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b61e43e-e2df-48f2-a618-88083cfbee59 admin admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:00:57.829328 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b61e43e-e2df-48f2-a618-88083cfbee59 admin admin] PATH_INFO: `/v3/users/5c4942bc48564b84bac366ba76920a70` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:00:57.830523 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b61e43e-e2df-48f2-a618-88083cfbee59 admin admin] RBAC: Authorizing `identity:delete_user(user_id=5c4942bc48564b84bac366ba76920a70)` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:00:57.832634 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b61e43e-e2df-48f2-a618-88083cfbee59 admin admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:00:58.053056 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [None req-9b61e43e-e2df-48f2-a618-88083cfbee59 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '5c4942bc48564b84bac366ba76920a70'} {{(pid=92057) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:00:58.056647 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=86,keystone:INSERT=6,keystone:DELETE=8 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 13:00:58.139144 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [None req-9b61e43e-e2df-48f2-a618-88083cfbee59 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '5c4942bc48564b84bac366ba76920a70'} {{(pid=92057) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:00:58.154754 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [None req-9b61e43e-e2df-48f2-a618-88083cfbee59 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '5c4942bc48564b84bac366ba76920a70'} {{(pid=92057) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:00:58.155567 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [None req-9b61e43e-e2df-48f2-a618-88083cfbee59 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '5c4942bc48564b84bac366ba76920a70'} {{(pid=92057) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:00:58.170574 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [None req-9b61e43e-e2df-48f2-a618-88083cfbee59 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '5c4942bc48564b84bac366ba76920a70'} {{(pid=92057) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:00:58.179528 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 970/1940] 10.222.0.30 () {58 vars in 1223 bytes} [Mon May 23 13:00:57 2022] DELETE /identity/v3/users/5c4942bc48564b84bac366ba76920a70 => generated 0 bytes in 428 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) May 23 13:00:58.184608 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3d15af8-c58d-4532-9435-5f145c9a324a None None] REQUEST_METHOD: `POST` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:00:58.185080 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3d15af8-c58d-4532-9435-5f145c9a324a None None] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:00:58.187601 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3d15af8-c58d-4532-9435-5f145c9a324a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:00:58.204289 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.auth.core [None req-d3d15af8-c58d-4532-9435-5f145c9a324a None None] MFA Rules not processed for user `0f7119f8fa5c4445977671140ded91a4`. Rule list: `[]` (Enabled: `True`). {{(pid=92056) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 13:00:58.205522 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.fernet_utils [None req-d3d15af8-c58d-4532-9435-5f145c9a324a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92056) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:00:58.275572 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 971/1941] 10.222.0.30 () {58 vars in 957 bytes} [Mon May 23 13:00:58 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 92 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) May 23 13:00:58.285065 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b61e43e-e2df-48f2-a618-88083cfbee59 admin admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:00:58.286477 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.fernet_utils [None req-9b61e43e-e2df-48f2-a618-88083cfbee59 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92057) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:00:58.345975 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b61e43e-e2df-48f2-a618-88083cfbee59 admin admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:00:58.349771 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2483388c-3f06-479c-a591-d680feee5b12 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2eb83ca38e79449095f3a2f947bbb06a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9c389ae65aaa4542bb5a5c4fd0bacf5b', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:00:58.351448 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2483388c-3f06-479c-a591-d680feee5b12 service neutron] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:00:58.351736 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2483388c-3f06-479c-a591-d680feee5b12 service neutron] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:00:58.351823 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2483388c-3f06-479c-a591-d680feee5b12 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:00:58.362276 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2483388c-3f06-479c-a591-d680feee5b12 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:00:58.363667 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2483388c-3f06-479c-a591-d680feee5b12 service neutron] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:00:58.375179 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 971/1942] 10.222.0.30 () {60 vars in 1365 bytes} [Mon May 23 13:00:58 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 90 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) May 23 13:00:58.763565 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3d15af8-c58d-4532-9435-5f145c9a324a None None] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:00:58.765136 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.fernet_utils [None req-d3d15af8-c58d-4532-9435-5f145c9a324a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92056) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:00:58.828176 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=100 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 13:00:58.832833 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3d15af8-c58d-4532-9435-5f145c9a324a None None] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:00:58.836530 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a97e0a8-5181-4924-b805-65785b33217c service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1666097f8ed545c6b0e4809c93e4775a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9c389ae65aaa4542bb5a5c4fd0bacf5b', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:00:58.837228 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a97e0a8-5181-4924-b805-65785b33217c service cinder] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:00:58.837505 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a97e0a8-5181-4924-b805-65785b33217c service cinder] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:00:58.837776 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a97e0a8-5181-4924-b805-65785b33217c service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:00:58.839642 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.fernet_utils [None req-0a97e0a8-5181-4924-b805-65785b33217c service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92056) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:00:58.908064 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a97e0a8-5181-4924-b805-65785b33217c service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:00:58.908064 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a97e0a8-5181-4924-b805-65785b33217c service cinder] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:00:58.909111 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2483388c-3f06-479c-a591-d680feee5b12 service neutron] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:00:58.910932 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.fernet_utils [None req-2483388c-3f06-479c-a591-d680feee5b12 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92057) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:00:58.921679 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 972/1943] 10.222.0.30 () {60 vars in 1365 bytes} [Mon May 23 13:00:58 2022] GET /identity/v3/auth/tokens => generated 3252 bytes in 159 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) May 23 13:00:58.929326 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2483388c-3f06-479c-a591-d680feee5b12 service neutron] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:00:58.935078 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ecba3ce6-ec10-4f47-9033-537f25aa724c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:00:58.935903 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecba3ce6-ec10-4f47-9033-537f25aa724c admin admin] REQUEST_METHOD: `DELETE` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:00:58.936346 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecba3ce6-ec10-4f47-9033-537f25aa724c admin admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:00:58.936774 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecba3ce6-ec10-4f47-9033-537f25aa724c admin admin] PATH_INFO: `/v3/projects/1b4f7473795d443a914ef0d5ce171ae8` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:00:58.938410 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ecba3ce6-ec10-4f47-9033-537f25aa724c admin admin] RBAC: Authorizing `identity:delete_project(project_id=1b4f7473795d443a914ef0d5ce171ae8)` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:00:58.940809 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ecba3ce6-ec10-4f47-9033-537f25aa724c admin admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:00:59.209464 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [None req-ecba3ce6-ec10-4f47-9033-537f25aa724c admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '1b4f7473795d443a914ef0d5ce171ae8', 'request_id': 'req-ecba3ce6-ec10-4f47-9033-537f25aa724c'} {{(pid=92057) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:00:59.219004 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [None req-ecba3ce6-ec10-4f47-9033-537f25aa724c admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '1b4f7473795d443a914ef0d5ce171ae8', 'request_id': 'req-ecba3ce6-ec10-4f47-9033-537f25aa724c'} {{(pid=92057) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:00:59.625602 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [None req-ecba3ce6-ec10-4f47-9033-537f25aa724c admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '1b4f7473795d443a914ef0d5ce171ae8', 'request_id': 'req-ecba3ce6-ec10-4f47-9033-537f25aa724c'} {{(pid=92057) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:00:59.651171 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [None req-ecba3ce6-ec10-4f47-9033-537f25aa724c admin admin] The token cache is being invalidate because project 1b4f7473795d443a914ef0d5ce171ae8 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=92057) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} May 23 13:00:59.651586 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [None req-ecba3ce6-ec10-4f47-9033-537f25aa724c admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=92057) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:00:59.652602 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [None req-ecba3ce6-ec10-4f47-9033-537f25aa724c admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=92057) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:00:59.655025 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 972/1944] 10.222.0.30 () {58 vars in 1232 bytes} [Mon May 23 13:00:58 2022] DELETE /identity/v3/projects/1b4f7473795d443a914ef0d5ce171ae8 => generated 0 bytes in 746 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) May 23 13:00:59.661319 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a97e0a8-5181-4924-b805-65785b33217c service cinder] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:00:59.680445 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a97e0a8-5181-4924-b805-65785b33217c service cinder] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:00:59.750152 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77fdb27b-0c27-48dd-a27f-abc243f323ff admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:00:59.750152 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77fdb27b-0c27-48dd-a27f-abc243f323ff admin admin] REQUEST_METHOD: `DELETE` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:00:59.750152 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77fdb27b-0c27-48dd-a27f-abc243f323ff admin admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:00:59.750152 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77fdb27b-0c27-48dd-a27f-abc243f323ff admin admin] PATH_INFO: `/v3/users/99dd30f1f1124a9086f747794897bc0f` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:00:59.751683 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77fdb27b-0c27-48dd-a27f-abc243f323ff admin admin] RBAC: Authorizing `identity:delete_user(user_id=99dd30f1f1124a9086f747794897bc0f)` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:00:59.754601 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77fdb27b-0c27-48dd-a27f-abc243f323ff admin admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:00:59.905381 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-77fdb27b-0c27-48dd-a27f-abc243f323ff admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '99dd30f1f1124a9086f747794897bc0f'} {{(pid=92056) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:00:59.905996 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-77fdb27b-0c27-48dd-a27f-abc243f323ff admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '99dd30f1f1124a9086f747794897bc0f'} {{(pid=92056) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:01:00.028811 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-77fdb27b-0c27-48dd-a27f-abc243f323ff admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '99dd30f1f1124a9086f747794897bc0f'} {{(pid=92056) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:01:00.064722 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-77fdb27b-0c27-48dd-a27f-abc243f323ff admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '99dd30f1f1124a9086f747794897bc0f'} {{(pid=92056) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:01:00.074789 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-77fdb27b-0c27-48dd-a27f-abc243f323ff admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '99dd30f1f1124a9086f747794897bc0f'} {{(pid=92056) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:01:00.092225 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 973/1945] 10.222.0.30 () {58 vars in 1223 bytes} [Mon May 23 13:00:59 2022] DELETE /identity/v3/users/99dd30f1f1124a9086f747794897bc0f => generated 0 bytes in 432 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) May 23 13:01:00.406149 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ecba3ce6-ec10-4f47-9033-537f25aa724c admin admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:01:00.407648 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.fernet_utils [None req-ecba3ce6-ec10-4f47-9033-537f25aa724c admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92057) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:01:00.423390 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ecba3ce6-ec10-4f47-9033-537f25aa724c admin admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:01:00.489919 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=90,keystone:DELETE=9,keystone:INSERT=1 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 13:01:00.497380 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75a2ab1b-41eb-4e6c-9baf-fc0dac41fca5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:01:00.498446 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75a2ab1b-41eb-4e6c-9baf-fc0dac41fca5 admin admin] REQUEST_METHOD: `DELETE` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:01:00.498732 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75a2ab1b-41eb-4e6c-9baf-fc0dac41fca5 admin admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:01:00.499008 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75a2ab1b-41eb-4e6c-9baf-fc0dac41fca5 admin admin] PATH_INFO: `/v3/users/a1cc3a7861c44e3688155c7066fdfb6b` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:01:00.500313 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75a2ab1b-41eb-4e6c-9baf-fc0dac41fca5 admin admin] RBAC: Authorizing `identity:delete_user(user_id=a1cc3a7861c44e3688155c7066fdfb6b)` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:01:00.507818 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75a2ab1b-41eb-4e6c-9baf-fc0dac41fca5 admin admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:01:00.551497 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77fdb27b-0c27-48dd-a27f-abc243f323ff admin admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:01:00.553162 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.fernet_utils [None req-77fdb27b-0c27-48dd-a27f-abc243f323ff admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92056) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:01:00.608750 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77fdb27b-0c27-48dd-a27f-abc243f323ff admin admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:01:00.618722 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-856c2e94-945e-40e5-9c9a-8698f64ccf89 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:01:00.620444 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-856c2e94-945e-40e5-9c9a-8698f64ccf89 admin admin] REQUEST_METHOD: `DELETE` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:01:00.620911 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-856c2e94-945e-40e5-9c9a-8698f64ccf89 admin admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:01:00.621443 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-856c2e94-945e-40e5-9c9a-8698f64ccf89 admin admin] PATH_INFO: `/v3/projects/c2089bce847847b4808e4c44e56bf2cf` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:01:00.625337 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-856c2e94-945e-40e5-9c9a-8698f64ccf89 admin admin] RBAC: Authorizing `identity:delete_project(project_id=c2089bce847847b4808e4c44e56bf2cf)` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:01:00.627778 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-856c2e94-945e-40e5-9c9a-8698f64ccf89 admin admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:01:00.650454 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 13:01:00.787941 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [None req-75a2ab1b-41eb-4e6c-9baf-fc0dac41fca5 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'a1cc3a7861c44e3688155c7066fdfb6b'} {{(pid=92057) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:01:00.788048 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-856c2e94-945e-40e5-9c9a-8698f64ccf89 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'c2089bce847847b4808e4c44e56bf2cf', 'request_id': 'req-856c2e94-945e-40e5-9c9a-8698f64ccf89'} {{(pid=92056) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:01:00.864891 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [None req-75a2ab1b-41eb-4e6c-9baf-fc0dac41fca5 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'a1cc3a7861c44e3688155c7066fdfb6b'} {{(pid=92057) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:01:00.877774 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [None req-75a2ab1b-41eb-4e6c-9baf-fc0dac41fca5 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'a1cc3a7861c44e3688155c7066fdfb6b'} {{(pid=92057) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:01:00.879733 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [None req-75a2ab1b-41eb-4e6c-9baf-fc0dac41fca5 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'a1cc3a7861c44e3688155c7066fdfb6b'} {{(pid=92057) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:01:00.893610 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [None req-75a2ab1b-41eb-4e6c-9baf-fc0dac41fca5 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'a1cc3a7861c44e3688155c7066fdfb6b'} {{(pid=92057) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:01:00.900261 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 973/1946] 10.222.0.30 () {58 vars in 1223 bytes} [Mon May 23 13:01:00 2022] DELETE /identity/v3/users/a1cc3a7861c44e3688155c7066fdfb6b => generated 0 bytes in 495 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) May 23 13:01:00.905347 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16346219-69f8-48b3-b2be-6fb6751e80d6 None None] REQUEST_METHOD: `POST` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:01:00.918350 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16346219-69f8-48b3-b2be-6fb6751e80d6 None None] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:01:00.918350 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16346219-69f8-48b3-b2be-6fb6751e80d6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:01:00.926527 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.auth.core [None req-16346219-69f8-48b3-b2be-6fb6751e80d6 None None] MFA Rules not processed for user `0f7119f8fa5c4445977671140ded91a4`. Rule list: `[]` (Enabled: `True`). {{(pid=92057) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 13:01:00.927753 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.fernet_utils [None req-16346219-69f8-48b3-b2be-6fb6751e80d6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92057) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:01:00.961784 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-856c2e94-945e-40e5-9c9a-8698f64ccf89 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'c2089bce847847b4808e4c44e56bf2cf', 'request_id': 'req-856c2e94-945e-40e5-9c9a-8698f64ccf89'} {{(pid=92056) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:01:00.970509 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-856c2e94-945e-40e5-9c9a-8698f64ccf89 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'c2089bce847847b4808e4c44e56bf2cf', 'request_id': 'req-856c2e94-945e-40e5-9c9a-8698f64ccf89'} {{(pid=92056) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:01:00.983333 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-856c2e94-945e-40e5-9c9a-8698f64ccf89 admin admin] The token cache is being invalidate because project c2089bce847847b4808e4c44e56bf2cf was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=92056) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} May 23 13:01:00.983935 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-856c2e94-945e-40e5-9c9a-8698f64ccf89 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=92056) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:01:00.984682 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-856c2e94-945e-40e5-9c9a-8698f64ccf89 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=92056) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:01:00.986856 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 974/1947] 10.222.0.30 () {58 vars in 1232 bytes} [Mon May 23 13:01:00 2022] DELETE /identity/v3/projects/c2089bce847847b4808e4c44e56bf2cf => generated 0 bytes in 436 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) May 23 13:01:00.992728 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-856c2e94-945e-40e5-9c9a-8698f64ccf89 admin admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:01:00.994548 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 974/1948] 10.222.0.30 () {58 vars in 957 bytes} [Mon May 23 13:01:00 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 90 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) May 23 13:01:00.999834 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.fernet_utils [None req-856c2e94-945e-40e5-9c9a-8698f64ccf89 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92056) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:01:01.011950 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16346219-69f8-48b3-b2be-6fb6751e80d6 None None] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:01:01.014899 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.fernet_utils [None req-16346219-69f8-48b3-b2be-6fb6751e80d6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92057) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:01:01.077000 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-856c2e94-945e-40e5-9c9a-8698f64ccf89 admin admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:01:01.082106 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3bddd32a-53dc-4231-a549-a4bfa94f48aa admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:01:01.083178 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bddd32a-53dc-4231-a549-a4bfa94f48aa admin admin] REQUEST_METHOD: `DELETE` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:01:01.083462 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bddd32a-53dc-4231-a549-a4bfa94f48aa admin admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:01:01.083740 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bddd32a-53dc-4231-a549-a4bfa94f48aa admin admin] PATH_INFO: `/v3/users/8a3a22b47b7041358020098e10951ca8` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:01:01.085642 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3bddd32a-53dc-4231-a549-a4bfa94f48aa admin admin] RBAC: Authorizing `identity:delete_user(user_id=8a3a22b47b7041358020098e10951ca8)` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:01:01.087321 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16346219-69f8-48b3-b2be-6fb6751e80d6 None None] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:01:01.091540 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3bddd32a-53dc-4231-a549-a4bfa94f48aa admin admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:01:01.093254 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c28fdcb0-449c-41bb-9e6f-c9e12e3a2913 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2eb83ca38e79449095f3a2f947bbb06a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9c389ae65aaa4542bb5a5c4fd0bacf5b', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:01:01.094122 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c28fdcb0-449c-41bb-9e6f-c9e12e3a2913 service neutron] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:01:01.094802 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c28fdcb0-449c-41bb-9e6f-c9e12e3a2913 service neutron] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:01:01.095540 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c28fdcb0-449c-41bb-9e6f-c9e12e3a2913 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:01:01.121601 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c28fdcb0-449c-41bb-9e6f-c9e12e3a2913 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:01:01.123349 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c28fdcb0-449c-41bb-9e6f-c9e12e3a2913 service neutron] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:01:01.134527 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 975/1949] 10.222.0.30 () {60 vars in 1365 bytes} [Mon May 23 13:01:01 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 124 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) May 23 13:01:01.369700 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-3bddd32a-53dc-4231-a549-a4bfa94f48aa admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '8a3a22b47b7041358020098e10951ca8'} {{(pid=92056) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:01:01.371042 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-3bddd32a-53dc-4231-a549-a4bfa94f48aa admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '8a3a22b47b7041358020098e10951ca8'} {{(pid=92056) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:01:01.453337 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-3bddd32a-53dc-4231-a549-a4bfa94f48aa admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '8a3a22b47b7041358020098e10951ca8'} {{(pid=92056) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:01:01.469702 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-3bddd32a-53dc-4231-a549-a4bfa94f48aa admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '8a3a22b47b7041358020098e10951ca8'} {{(pid=92056) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:01:01.478264 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-3bddd32a-53dc-4231-a549-a4bfa94f48aa admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '8a3a22b47b7041358020098e10951ca8'} {{(pid=92056) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:01:01.485143 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 975/1950] 10.222.0.30 () {58 vars in 1223 bytes} [Mon May 23 13:01:00 2022] DELETE /identity/v3/users/8a3a22b47b7041358020098e10951ca8 => generated 0 bytes in 494 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) May 23 13:01:01.603353 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c28fdcb0-449c-41bb-9e6f-c9e12e3a2913 service neutron] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:01:01.608827 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.fernet_utils [None req-c28fdcb0-449c-41bb-9e6f-c9e12e3a2913 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92057) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:01:01.620100 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c28fdcb0-449c-41bb-9e6f-c9e12e3a2913 service neutron] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:01:01.638918 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 13:01:01.716324 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-073d7bb6-6ede-42b3-88f6-7fd95502d86b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:01:01.718043 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-073d7bb6-6ede-42b3-88f6-7fd95502d86b admin admin] REQUEST_METHOD: `DELETE` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:01:01.718043 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-073d7bb6-6ede-42b3-88f6-7fd95502d86b admin admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:01:01.718043 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-073d7bb6-6ede-42b3-88f6-7fd95502d86b admin admin] PATH_INFO: `/v3/projects/e805a9b04cbb4815a6ff030da2f110fb` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:01:01.719628 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-073d7bb6-6ede-42b3-88f6-7fd95502d86b admin admin] RBAC: Authorizing `identity:delete_project(project_id=e805a9b04cbb4815a6ff030da2f110fb)` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:01:01.721391 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-073d7bb6-6ede-42b3-88f6-7fd95502d86b admin admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:01:01.859017 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [None req-073d7bb6-6ede-42b3-88f6-7fd95502d86b admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'e805a9b04cbb4815a6ff030da2f110fb', 'request_id': 'req-073d7bb6-6ede-42b3-88f6-7fd95502d86b'} {{(pid=92057) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:01:01.870096 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [None req-073d7bb6-6ede-42b3-88f6-7fd95502d86b admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'e805a9b04cbb4815a6ff030da2f110fb', 'request_id': 'req-073d7bb6-6ede-42b3-88f6-7fd95502d86b'} {{(pid=92057) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:01:01.965373 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [None req-073d7bb6-6ede-42b3-88f6-7fd95502d86b admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'e805a9b04cbb4815a6ff030da2f110fb', 'request_id': 'req-073d7bb6-6ede-42b3-88f6-7fd95502d86b'} {{(pid=92057) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:01:01.984158 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [None req-073d7bb6-6ede-42b3-88f6-7fd95502d86b admin admin] The token cache is being invalidate because project e805a9b04cbb4815a6ff030da2f110fb was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=92057) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} May 23 13:01:01.984481 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [None req-073d7bb6-6ede-42b3-88f6-7fd95502d86b admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=92057) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:01:01.985243 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [None req-073d7bb6-6ede-42b3-88f6-7fd95502d86b admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=92057) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:01:01.989967 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 976/1951] 10.222.0.30 () {58 vars in 1232 bytes} [Mon May 23 13:01:01 2022] DELETE /identity/v3/projects/e805a9b04cbb4815a6ff030da2f110fb => generated 0 bytes in 384 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) May 23 13:01:01.991362 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3bddd32a-53dc-4231-a549-a4bfa94f48aa admin admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:01:01.994002 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.fernet_utils [None req-3bddd32a-53dc-4231-a549-a4bfa94f48aa admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92056) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:01:02.009274 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3bddd32a-53dc-4231-a549-a4bfa94f48aa admin admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:01:02.032728 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=81,keystone:DELETE=17,keystone:INSERT=2 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 13:01:02.102272 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-584cb091-15d6-4768-a624-057831caedd4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:01:02.103574 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-584cb091-15d6-4768-a624-057831caedd4 admin admin] REQUEST_METHOD: `DELETE` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:01:02.103905 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-584cb091-15d6-4768-a624-057831caedd4 admin admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:01:02.105340 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-584cb091-15d6-4768-a624-057831caedd4 admin admin] PATH_INFO: `/v3/users/e592dac401c948ce99cd8797af3b7dd5` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:01:02.107403 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-584cb091-15d6-4768-a624-057831caedd4 admin admin] RBAC: Authorizing `identity:delete_user(user_id=e592dac401c948ce99cd8797af3b7dd5)` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:01:02.109351 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-584cb091-15d6-4768-a624-057831caedd4 admin admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:01:02.172944 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-073d7bb6-6ede-42b3-88f6-7fd95502d86b admin admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:01:02.174420 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.fernet_utils [None req-073d7bb6-6ede-42b3-88f6-7fd95502d86b admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92057) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:01:02.197215 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-073d7bb6-6ede-42b3-88f6-7fd95502d86b admin admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:01:02.205174 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e451c08-3981-4159-9519-7e783372c5c1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:01:02.205174 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e451c08-3981-4159-9519-7e783372c5c1 admin admin] REQUEST_METHOD: `DELETE` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:01:02.205174 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e451c08-3981-4159-9519-7e783372c5c1 admin admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:01:02.205174 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e451c08-3981-4159-9519-7e783372c5c1 admin admin] PATH_INFO: `/v3/projects/c9f491f5b03e43cca9d4469a0129f84e` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:01:02.206485 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e451c08-3981-4159-9519-7e783372c5c1 admin admin] RBAC: Authorizing `identity:delete_project(project_id=c9f491f5b03e43cca9d4469a0129f84e)` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:01:02.209315 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e451c08-3981-4159-9519-7e783372c5c1 admin admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:01:02.354024 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-584cb091-15d6-4768-a624-057831caedd4 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'e592dac401c948ce99cd8797af3b7dd5'} {{(pid=92056) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:01:02.356117 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-584cb091-15d6-4768-a624-057831caedd4 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'e592dac401c948ce99cd8797af3b7dd5'} {{(pid=92056) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:01:02.356275 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [None req-4e451c08-3981-4159-9519-7e783372c5c1 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'c9f491f5b03e43cca9d4469a0129f84e', 'request_id': 'req-4e451c08-3981-4159-9519-7e783372c5c1'} {{(pid=92057) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:01:02.362627 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [None req-4e451c08-3981-4159-9519-7e783372c5c1 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'c9f491f5b03e43cca9d4469a0129f84e', 'request_id': 'req-4e451c08-3981-4159-9519-7e783372c5c1'} {{(pid=92057) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:01:02.462999 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-584cb091-15d6-4768-a624-057831caedd4 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'e592dac401c948ce99cd8797af3b7dd5'} {{(pid=92056) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:01:02.482612 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-584cb091-15d6-4768-a624-057831caedd4 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'e592dac401c948ce99cd8797af3b7dd5'} {{(pid=92056) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:01:02.493314 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-584cb091-15d6-4768-a624-057831caedd4 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'e592dac401c948ce99cd8797af3b7dd5'} {{(pid=92056) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:01:02.520896 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 976/1952] 10.222.0.30 () {58 vars in 1223 bytes} [Mon May 23 13:01:01 2022] DELETE /identity/v3/users/e592dac401c948ce99cd8797af3b7dd5 => generated 0 bytes in 529 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) May 23 13:01:02.560450 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [None req-4e451c08-3981-4159-9519-7e783372c5c1 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'c9f491f5b03e43cca9d4469a0129f84e', 'request_id': 'req-4e451c08-3981-4159-9519-7e783372c5c1'} {{(pid=92057) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:01:02.579220 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [None req-4e451c08-3981-4159-9519-7e783372c5c1 admin admin] The token cache is being invalidate because project c9f491f5b03e43cca9d4469a0129f84e was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=92057) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} May 23 13:01:02.580352 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [None req-4e451c08-3981-4159-9519-7e783372c5c1 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=92057) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:01:02.581930 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [None req-4e451c08-3981-4159-9519-7e783372c5c1 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=92057) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:01:02.587612 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 977/1953] 10.222.0.30 () {58 vars in 1232 bytes} [Mon May 23 13:01:02 2022] DELETE /identity/v3/projects/c9f491f5b03e43cca9d4469a0129f84e => generated 0 bytes in 415 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) May 23 13:01:02.594991 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbb7acaa-d56c-4687-8e8f-54980ef63e2e None None] REQUEST_METHOD: `POST` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:01:02.598004 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbb7acaa-d56c-4687-8e8f-54980ef63e2e None None] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:01:02.598004 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbb7acaa-d56c-4687-8e8f-54980ef63e2e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:01:02.620887 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.auth.core [None req-fbb7acaa-d56c-4687-8e8f-54980ef63e2e None None] MFA Rules not processed for user `0f7119f8fa5c4445977671140ded91a4`. Rule list: `[]` (Enabled: `True`). {{(pid=92056) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 13:01:02.622704 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.fernet_utils [None req-fbb7acaa-d56c-4687-8e8f-54980ef63e2e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92056) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:01:02.709781 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 977/1954] 10.222.0.30 () {58 vars in 957 bytes} [Mon May 23 13:01:02 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 116 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) May 23 13:01:02.718679 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55a22a0c-a84b-4eeb-b1eb-1d512730fa83 None None] REQUEST_METHOD: `POST` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:01:02.719238 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55a22a0c-a84b-4eeb-b1eb-1d512730fa83 None None] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:01:02.719238 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55a22a0c-a84b-4eeb-b1eb-1d512730fa83 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:01:02.744720 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.auth.core [None req-55a22a0c-a84b-4eeb-b1eb-1d512730fa83 None None] MFA Rules not processed for user `0f7119f8fa5c4445977671140ded91a4`. Rule list: `[]` (Enabled: `True`). {{(pid=92057) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 13:01:02.744720 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.fernet_utils [None req-55a22a0c-a84b-4eeb-b1eb-1d512730fa83 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92057) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:01:02.757237 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 978/1955] 10.222.0.30 () {58 vars in 957 bytes} [Mon May 23 13:01:02 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 37 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) May 23 13:01:02.758452 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fbb7acaa-d56c-4687-8e8f-54980ef63e2e None None] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:01:02.771564 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fbb7acaa-d56c-4687-8e8f-54980ef63e2e None None] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:01:02.781644 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d233da42-a9cd-4c6e-85bf-8f764267ce76 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:01:02.782440 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d233da42-a9cd-4c6e-85bf-8f764267ce76 admin admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:01:02.783396 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d233da42-a9cd-4c6e-85bf-8f764267ce76 admin admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:01:02.783678 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d233da42-a9cd-4c6e-85bf-8f764267ce76 admin admin] PATH_INFO: `/v3/domains` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:01:02.784467 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d233da42-a9cd-4c6e-85bf-8f764267ce76 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=92056) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} May 23 13:01:02.784868 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d233da42-a9cd-4c6e-85bf-8f764267ce76 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:01:02.786562 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d233da42-a9cd-4c6e-85bf-8f764267ce76 admin admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:01:02.803807 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 978/1956] 10.222.0.30 () {58 vars in 1152 bytes} [Mon May 23 13:01:02 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 46 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) May 23 13:01:02.809267 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55a22a0c-a84b-4eeb-b1eb-1d512730fa83 None None] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:01:02.819793 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55a22a0c-a84b-4eeb-b1eb-1d512730fa83 None None] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:01:02.827300 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9f56609-1617-4201-8d96-df69668bd783 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:01:02.828337 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9f56609-1617-4201-8d96-df69668bd783 admin admin] REQUEST_METHOD: `POST` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:01:02.828788 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9f56609-1617-4201-8d96-df69668bd783 admin admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:01:02.832540 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9f56609-1617-4201-8d96-df69668bd783 admin admin] PATH_INFO: `/v3/projects` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:01:02.834081 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9f56609-1617-4201-8d96-df69668bd783 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:01:02.838058 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9f56609-1617-4201-8d96-df69668bd783 admin admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:01:02.954679 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 979/1957] 10.222.0.30 () {60 vars in 1152 bytes} [Mon May 23 13:01:02 2022] POST /identity/v3/projects => generated 385 bytes in 146 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) May 23 13:01:02.958532 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d233da42-a9cd-4c6e-85bf-8f764267ce76 admin admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:01:02.972946 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d233da42-a9cd-4c6e-85bf-8f764267ce76 admin admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:01:03.047359 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-574015f9-2171-4ebc-acb9-adaa8d5325dd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:01:03.047359 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-574015f9-2171-4ebc-acb9-adaa8d5325dd admin admin] REQUEST_METHOD: `POST` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:01:03.047359 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-574015f9-2171-4ebc-acb9-adaa8d5325dd admin admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:01:03.047359 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-574015f9-2171-4ebc-acb9-adaa8d5325dd admin admin] PATH_INFO: `/v3/users` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:01:03.048270 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-574015f9-2171-4ebc-acb9-adaa8d5325dd admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:01:03.048523 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-574015f9-2171-4ebc-acb9-adaa8d5325dd admin admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:01:03.148617 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 979/1958] 10.222.0.30 () {60 vars in 1143 bytes} [Mon May 23 13:01:02 2022] POST /identity/v3/users => generated 293 bytes in 191 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) May 23 13:01:03.153575 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9f56609-1617-4201-8d96-df69668bd783 admin admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:01:03.159110 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=83,keystone:DELETE=14,keystone:INSERT=3 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 13:01:03.175088 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9f56609-1617-4201-8d96-df69668bd783 admin admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:01:03.179691 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-574015f9-2171-4ebc-acb9-adaa8d5325dd admin admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:01:03.181684 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.fernet_utils [None req-574015f9-2171-4ebc-acb9-adaa8d5325dd admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92056) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:01:03.182214 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a12924a9-ba12-43df-9a94-c198aa28a6ec admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:01:03.183179 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a12924a9-ba12-43df-9a94-c198aa28a6ec admin admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:01:03.183432 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a12924a9-ba12-43df-9a94-c198aa28a6ec admin admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:01:03.183718 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a12924a9-ba12-43df-9a94-c198aa28a6ec admin admin] PATH_INFO: `/v3/roles` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:01:03.184459 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a12924a9-ba12-43df-9a94-c198aa28a6ec admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:01:03.186115 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a12924a9-ba12-43df-9a94-c198aa28a6ec admin admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:01:03.196674 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-574015f9-2171-4ebc-acb9-adaa8d5325dd admin admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:01:03.201347 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 980/1959] 10.222.0.30 () {58 vars in 1121 bytes} [Mon May 23 13:01:03 2022] GET /identity/v3/roles => generated 1420 bytes in 48 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) May 23 13:01:03.206875 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4bf8511e-9501-4c46-b5fb-9a5b99cd52cf admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:01:03.207761 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bf8511e-9501-4c46-b5fb-9a5b99cd52cf admin admin] REQUEST_METHOD: `DELETE` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:01:03.208044 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bf8511e-9501-4c46-b5fb-9a5b99cd52cf admin admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:01:03.208338 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bf8511e-9501-4c46-b5fb-9a5b99cd52cf admin admin] PATH_INFO: `/v3/projects/cffd0d83c01b45738651a4a7dc2aacb0` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:01:03.209080 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a12924a9-ba12-43df-9a94-c198aa28a6ec admin admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:01:03.209539 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4bf8511e-9501-4c46-b5fb-9a5b99cd52cf admin admin] RBAC: Authorizing `identity:delete_project(project_id=cffd0d83c01b45738651a4a7dc2aacb0)` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:01:03.213183 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4bf8511e-9501-4c46-b5fb-9a5b99cd52cf admin admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:01:03.226118 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a12924a9-ba12-43df-9a94-c198aa28a6ec admin admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:01:03.235093 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-837a4dcc-0042-4b26-8bbf-5ce164ce7940 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:01:03.235816 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-837a4dcc-0042-4b26-8bbf-5ce164ce7940 admin admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:01:03.236159 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-837a4dcc-0042-4b26-8bbf-5ce164ce7940 admin admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:01:03.236442 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-837a4dcc-0042-4b26-8bbf-5ce164ce7940 admin admin] PATH_INFO: `/v3/roles` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:01:03.237232 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-837a4dcc-0042-4b26-8bbf-5ce164ce7940 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:01:03.239104 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-837a4dcc-0042-4b26-8bbf-5ce164ce7940 admin admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:01:03.252547 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 981/1960] 10.222.0.30 () {58 vars in 1121 bytes} [Mon May 23 13:01:03 2022] GET /identity/v3/roles => generated 1420 bytes in 45 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) May 23 13:01:03.256884 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-837a4dcc-0042-4b26-8bbf-5ce164ce7940 admin admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:01:03.267888 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-837a4dcc-0042-4b26-8bbf-5ce164ce7940 admin admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:01:03.272713 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8d4917e-7e78-473c-9c38-18eca99c6814 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:01:03.273661 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8d4917e-7e78-473c-9c38-18eca99c6814 admin admin] REQUEST_METHOD: `PUT` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:01:03.273992 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8d4917e-7e78-473c-9c38-18eca99c6814 admin admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:01:03.274266 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8d4917e-7e78-473c-9c38-18eca99c6814 admin admin] PATH_INFO: `/v3/projects/29bf5c208f864889acd12cfd43bc53a4/users/a84cf597c8c04cbe86f673cd9796146e/roles/80fb22507c9a4ff09d44d8f12f6fff1a` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:01:03.290479 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8d4917e-7e78-473c-9c38-18eca99c6814 admin admin] RBAC: Authorizing `identity:create_grant(project_id=29bf5c208f864889acd12cfd43bc53a4, user_id=a84cf597c8c04cbe86f673cd9796146e, role_id=80fb22507c9a4ff09d44d8f12f6fff1a)` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:01:03.294404 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8d4917e-7e78-473c-9c38-18eca99c6814 admin admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:01:03.360568 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 982/1961] 10.222.0.30 () {60 vars in 1482 bytes} [Mon May 23 13:01:03 2022] PUT /identity/v3/projects/29bf5c208f864889acd12cfd43bc53a4/users/a84cf597c8c04cbe86f673cd9796146e/roles/80fb22507c9a4ff09d44d8f12f6fff1a => generated 0 bytes in 104 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) May 23 13:01:03.371690 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-4bf8511e-9501-4c46-b5fb-9a5b99cd52cf admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'cffd0d83c01b45738651a4a7dc2aacb0', 'request_id': 'req-4bf8511e-9501-4c46-b5fb-9a5b99cd52cf'} {{(pid=92056) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:01:03.372227 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7d351b2-5740-4fee-8e32-98a8f5db730f None None] REQUEST_METHOD: `POST` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:01:03.372524 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7d351b2-5740-4fee-8e32-98a8f5db730f None None] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:01:03.372803 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7d351b2-5740-4fee-8e32-98a8f5db730f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:01:03.382351 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 13:01:03.413856 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.auth.core [None req-e7d351b2-5740-4fee-8e32-98a8f5db730f None None] MFA Rules not processed for user `a84cf597c8c04cbe86f673cd9796146e`. Rule list: `[]` (Enabled: `True`). {{(pid=92057) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 13:01:03.413856 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.fernet_utils [None req-e7d351b2-5740-4fee-8e32-98a8f5db730f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92057) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:01:03.447194 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-4bf8511e-9501-4c46-b5fb-9a5b99cd52cf admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'cffd0d83c01b45738651a4a7dc2aacb0', 'request_id': 'req-4bf8511e-9501-4c46-b5fb-9a5b99cd52cf'} {{(pid=92056) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:01:03.456502 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-4bf8511e-9501-4c46-b5fb-9a5b99cd52cf admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'cffd0d83c01b45738651a4a7dc2aacb0', 'request_id': 'req-4bf8511e-9501-4c46-b5fb-9a5b99cd52cf'} {{(pid=92056) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:01:03.472068 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-4bf8511e-9501-4c46-b5fb-9a5b99cd52cf admin admin] The token cache is being invalidate because project cffd0d83c01b45738651a4a7dc2aacb0 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=92056) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} May 23 13:01:03.472305 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-4bf8511e-9501-4c46-b5fb-9a5b99cd52cf admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=92056) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:01:03.473178 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-4bf8511e-9501-4c46-b5fb-9a5b99cd52cf admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=92056) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:01:03.476411 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 983/1963] 10.222.0.30 () {58 vars in 957 bytes} [Mon May 23 13:01:03 2022] POST /identity/v3/auth/tokens => generated 3252 bytes in 105 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) May 23 13:01:03.479581 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 980/1963] 10.222.0.30 () {58 vars in 1232 bytes} [Mon May 23 13:01:03 2022] DELETE /identity/v3/projects/cffd0d83c01b45738651a4a7dc2aacb0 => generated 0 bytes in 296 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) May 23 13:01:03.479883 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-061b11bf-d20e-41c7-8b1b-2d3871f75c66 None None] REQUEST_METHOD: `POST` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:01:03.480244 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-061b11bf-d20e-41c7-8b1b-2d3871f75c66 None None] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:01:03.484709 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-061b11bf-d20e-41c7-8b1b-2d3871f75c66 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:01:03.496104 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4bf8511e-9501-4c46-b5fb-9a5b99cd52cf admin admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:01:03.497628 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.fernet_utils [None req-4bf8511e-9501-4c46-b5fb-9a5b99cd52cf admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92056) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:01:03.504479 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.auth.core [None req-061b11bf-d20e-41c7-8b1b-2d3871f75c66 None None] MFA Rules not processed for user `0f7119f8fa5c4445977671140ded91a4`. Rule list: `[]` (Enabled: `True`). {{(pid=92057) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 13:01:03.506137 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.fernet_utils [None req-061b11bf-d20e-41c7-8b1b-2d3871f75c66 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92057) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:01:03.560043 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4bf8511e-9501-4c46-b5fb-9a5b99cd52cf admin admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:01:03.563775 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74b55b6c-fac5-4a6e-b053-2d327df4be76 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1666097f8ed545c6b0e4809c93e4775a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9c389ae65aaa4542bb5a5c4fd0bacf5b', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:01:03.564419 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74b55b6c-fac5-4a6e-b053-2d327df4be76 service cinder] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:01:03.564706 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74b55b6c-fac5-4a6e-b053-2d327df4be76 service cinder] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:01:03.565100 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74b55b6c-fac5-4a6e-b053-2d327df4be76 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:01:03.567786 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.fernet_utils [None req-74b55b6c-fac5-4a6e-b053-2d327df4be76 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92056) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:01:03.575187 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 984/1964] 10.222.0.30 () {58 vars in 957 bytes} [Mon May 23 13:01:03 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 97 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) May 23 13:01:03.579419 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74b55b6c-fac5-4a6e-b053-2d327df4be76 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:01:03.580861 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74b55b6c-fac5-4a6e-b053-2d327df4be76 service cinder] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:01:03.583087 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6e0c6d9-ea2e-4359-aa75-8ae745b03760 None None] REQUEST_METHOD: `POST` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:01:03.583498 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6e0c6d9-ea2e-4359-aa75-8ae745b03760 None None] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:01:03.583784 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6e0c6d9-ea2e-4359-aa75-8ae745b03760 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:01:03.591357 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 981/1965] 10.222.0.30 () {60 vars in 1365 bytes} [Mon May 23 13:01:03 2022] GET /identity/v3/auth/tokens => generated 3252 bytes in 96 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) May 23 13:01:03.603335 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.auth.core [None req-c6e0c6d9-ea2e-4359-aa75-8ae745b03760 None None] MFA Rules not processed for user `0f7119f8fa5c4445977671140ded91a4`. Rule list: `[]` (Enabled: `True`). {{(pid=92057) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 13:01:03.605450 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.fernet_utils [None req-c6e0c6d9-ea2e-4359-aa75-8ae745b03760 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92057) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:01:03.612724 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74b55b6c-fac5-4a6e-b053-2d327df4be76 service cinder] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:01:03.614130 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.fernet_utils [None req-74b55b6c-fac5-4a6e-b053-2d327df4be76 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92056) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:01:03.614565 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 985/1966] 10.222.0.30 () {58 vars in 957 bytes} [Mon May 23 13:01:03 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 33 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) May 23 13:01:03.618564 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6e0c6d9-ea2e-4359-aa75-8ae745b03760 None None] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:01:03.631160 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6e0c6d9-ea2e-4359-aa75-8ae745b03760 None None] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:01:03.632508 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74b55b6c-fac5-4a6e-b053-2d327df4be76 service cinder] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:01:03.639665 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e24eabbb-7520-4498-8f7a-6a9f493d9df6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:01:03.640689 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e24eabbb-7520-4498-8f7a-6a9f493d9df6 admin admin] REQUEST_METHOD: `DELETE` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:01:03.640938 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e24eabbb-7520-4498-8f7a-6a9f493d9df6 admin admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:01:03.641378 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e24eabbb-7520-4498-8f7a-6a9f493d9df6 admin admin] PATH_INFO: `/v3/users/a84cf597c8c04cbe86f673cd9796146e` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:01:03.642865 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e24eabbb-7520-4498-8f7a-6a9f493d9df6 admin admin] RBAC: Authorizing `identity:delete_user(user_id=a84cf597c8c04cbe86f673cd9796146e)` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:01:03.644222 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc97bbd9-9599-492c-b698-dc5eea37681a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:01:03.644979 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc97bbd9-9599-492c-b698-dc5eea37681a admin admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:01:03.645098 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e24eabbb-7520-4498-8f7a-6a9f493d9df6 admin admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:01:03.645267 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc97bbd9-9599-492c-b698-dc5eea37681a admin admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:01:03.645541 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc97bbd9-9599-492c-b698-dc5eea37681a admin admin] PATH_INFO: `/v3/domains` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:01:03.646325 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc97bbd9-9599-492c-b698-dc5eea37681a admin admin] RBAC: Adding query filter params (name=Default) {{(pid=92057) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} May 23 13:01:03.646637 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc97bbd9-9599-492c-b698-dc5eea37681a admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:01:03.657052 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc97bbd9-9599-492c-b698-dc5eea37681a admin admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:01:03.671680 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 986/1967] 10.222.0.30 () {58 vars in 1152 bytes} [Mon May 23 13:01:03 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 54 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) May 23 13:01:03.676532 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc97bbd9-9599-492c-b698-dc5eea37681a admin admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:01:03.690125 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc97bbd9-9599-492c-b698-dc5eea37681a admin admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:01:03.696353 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af253c7a-c6ef-41e3-ba04-03c8ea772168 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:01:03.697042 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af253c7a-c6ef-41e3-ba04-03c8ea772168 admin admin] REQUEST_METHOD: `POST` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:01:03.697748 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af253c7a-c6ef-41e3-ba04-03c8ea772168 admin admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:01:03.697748 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af253c7a-c6ef-41e3-ba04-03c8ea772168 admin admin] PATH_INFO: `/v3/projects` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:01:03.699234 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af253c7a-c6ef-41e3-ba04-03c8ea772168 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:01:03.701224 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af253c7a-c6ef-41e3-ba04-03c8ea772168 admin admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:01:03.824141 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 987/1968] 10.222.0.30 () {60 vars in 1152 bytes} [Mon May 23 13:01:03 2022] POST /identity/v3/projects => generated 381 bytes in 148 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) May 23 13:01:03.828223 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af253c7a-c6ef-41e3-ba04-03c8ea772168 admin admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:01:03.861449 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-e24eabbb-7520-4498-8f7a-6a9f493d9df6 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'a84cf597c8c04cbe86f673cd9796146e'} {{(pid=92056) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:01:03.862150 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-e24eabbb-7520-4498-8f7a-6a9f493d9df6 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'a84cf597c8c04cbe86f673cd9796146e'} {{(pid=92056) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:01:03.901589 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af253c7a-c6ef-41e3-ba04-03c8ea772168 admin admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:01:03.906260 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7bd663e-d646-4624-afc0-fc3dd9a30dbb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:01:03.906981 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7bd663e-d646-4624-afc0-fc3dd9a30dbb admin admin] REQUEST_METHOD: `POST` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:01:03.907260 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7bd663e-d646-4624-afc0-fc3dd9a30dbb admin admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:01:03.907539 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7bd663e-d646-4624-afc0-fc3dd9a30dbb admin admin] PATH_INFO: `/v3/users` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:01:03.908223 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7bd663e-d646-4624-afc0-fc3dd9a30dbb admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:01:03.910222 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7bd663e-d646-4624-afc0-fc3dd9a30dbb admin admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:01:03.944655 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-e24eabbb-7520-4498-8f7a-6a9f493d9df6 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'a84cf597c8c04cbe86f673cd9796146e'} {{(pid=92056) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:01:03.958239 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-e24eabbb-7520-4498-8f7a-6a9f493d9df6 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'a84cf597c8c04cbe86f673cd9796146e'} {{(pid=92056) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:01:03.966428 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-e24eabbb-7520-4498-8f7a-6a9f493d9df6 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'a84cf597c8c04cbe86f673cd9796146e'} {{(pid=92056) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:01:03.973274 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 982/1969] 10.222.0.30 () {58 vars in 1223 bytes} [Mon May 23 13:01:03 2022] DELETE /identity/v3/users/a84cf597c8c04cbe86f673cd9796146e => generated 0 bytes in 362 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) May 23 13:01:03.977618 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc160f41-7efe-4b1f-a353-c6bfa4bdeab7 None None] REQUEST_METHOD: `POST` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:01:03.977946 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc160f41-7efe-4b1f-a353-c6bfa4bdeab7 None None] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:01:03.978235 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc160f41-7efe-4b1f-a353-c6bfa4bdeab7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:01:03.996625 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.auth.core [None req-dc160f41-7efe-4b1f-a353-c6bfa4bdeab7 None None] MFA Rules not processed for user `0f7119f8fa5c4445977671140ded91a4`. Rule list: `[]` (Enabled: `True`). {{(pid=92056) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 13:01:03.997818 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.fernet_utils [None req-dc160f41-7efe-4b1f-a353-c6bfa4bdeab7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92056) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:01:04.014396 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 988/1970] 10.222.0.30 () {60 vars in 1143 bytes} [Mon May 23 13:01:03 2022] POST /identity/v3/users => generated 291 bytes in 187 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) May 23 13:01:04.021490 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7bd663e-d646-4624-afc0-fc3dd9a30dbb admin admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:01:04.026991 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.fernet_utils [None req-a7bd663e-d646-4624-afc0-fc3dd9a30dbb admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92057) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:01:04.051929 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7bd663e-d646-4624-afc0-fc3dd9a30dbb admin admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:01:04.068895 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 983/1971] 10.222.0.30 () {58 vars in 957 bytes} [Mon May 23 13:01:03 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 91 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) May 23 13:01:04.085901 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc160f41-7efe-4b1f-a353-c6bfa4bdeab7 None None] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:01:04.087682 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.fernet_utils [None req-dc160f41-7efe-4b1f-a353-c6bfa4bdeab7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92056) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:01:04.142770 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77f1ca2a-a139-4697-bbe1-27002407e502 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:01:04.143450 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77f1ca2a-a139-4697-bbe1-27002407e502 admin admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:01:04.143738 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77f1ca2a-a139-4697-bbe1-27002407e502 admin admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:01:04.144018 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77f1ca2a-a139-4697-bbe1-27002407e502 admin admin] PATH_INFO: `/v3/roles` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:01:04.144913 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77f1ca2a-a139-4697-bbe1-27002407e502 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:01:04.146589 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77f1ca2a-a139-4697-bbe1-27002407e502 admin admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:01:04.158303 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 989/1972] 10.222.0.30 () {58 vars in 1121 bytes} [Mon May 23 13:01:04 2022] GET /identity/v3/roles => generated 1420 bytes in 137 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) May 23 13:01:04.167937 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77f1ca2a-a139-4697-bbe1-27002407e502 admin admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:01:04.173373 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77f1ca2a-a139-4697-bbe1-27002407e502 admin admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:01:04.177375 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc160f41-7efe-4b1f-a353-c6bfa4bdeab7 None None] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:01:04.178172 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3ef0d07-101b-430e-8114-a98f5a611407 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:01:04.178860 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3ef0d07-101b-430e-8114-a98f5a611407 admin admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:01:04.179143 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3ef0d07-101b-430e-8114-a98f5a611407 admin admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:01:04.179424 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3ef0d07-101b-430e-8114-a98f5a611407 admin admin] PATH_INFO: `/v3/roles` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:01:04.180148 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c3ef0d07-101b-430e-8114-a98f5a611407 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:01:04.181767 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c3ef0d07-101b-430e-8114-a98f5a611407 admin admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:01:04.182000 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d709d4b2-a6b3-46ac-b59b-10e80f0731ec service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2eb83ca38e79449095f3a2f947bbb06a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9c389ae65aaa4542bb5a5c4fd0bacf5b', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:01:04.182703 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d709d4b2-a6b3-46ac-b59b-10e80f0731ec service neutron] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:01:04.183007 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d709d4b2-a6b3-46ac-b59b-10e80f0731ec service neutron] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:01:04.183254 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d709d4b2-a6b3-46ac-b59b-10e80f0731ec service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:01:04.192583 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d709d4b2-a6b3-46ac-b59b-10e80f0731ec service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:01:04.193116 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 990/1973] 10.222.0.30 () {58 vars in 1121 bytes} [Mon May 23 13:01:04 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) May 23 13:01:04.194073 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d709d4b2-a6b3-46ac-b59b-10e80f0731ec service neutron] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:01:04.197114 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3ef0d07-101b-430e-8114-a98f5a611407 admin admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:01:04.204579 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 984/1974] 10.222.0.30 () {60 vars in 1365 bytes} [Mon May 23 13:01:04 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 119 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) May 23 13:01:04.208109 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3ef0d07-101b-430e-8114-a98f5a611407 admin admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:01:04.213344 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1980b0e9-485e-4c41-a2d7-06979dd6dea4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:01:04.214949 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1980b0e9-485e-4c41-a2d7-06979dd6dea4 admin admin] REQUEST_METHOD: `PUT` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:01:04.215141 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1980b0e9-485e-4c41-a2d7-06979dd6dea4 admin admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:01:04.215415 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1980b0e9-485e-4c41-a2d7-06979dd6dea4 admin admin] PATH_INFO: `/v3/projects/cb8d79a531c84f92a60c08ffccbeda0f/users/1394d8fb0f084cfd82bfacff79ad7b5f/roles/80fb22507c9a4ff09d44d8f12f6fff1a` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:01:04.233356 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1980b0e9-485e-4c41-a2d7-06979dd6dea4 admin admin] RBAC: Authorizing `identity:create_grant(project_id=cb8d79a531c84f92a60c08ffccbeda0f, user_id=1394d8fb0f084cfd82bfacff79ad7b5f, role_id=80fb22507c9a4ff09d44d8f12f6fff1a)` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:01:04.236413 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1980b0e9-485e-4c41-a2d7-06979dd6dea4 admin admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:01:04.340193 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 13:01:04.347867 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 991/1975] 10.222.0.30 () {60 vars in 1482 bytes} [Mon May 23 13:01:04 2022] PUT /identity/v3/projects/cb8d79a531c84f92a60c08ffccbeda0f/users/1394d8fb0f084cfd82bfacff79ad7b5f/roles/80fb22507c9a4ff09d44d8f12f6fff1a => generated 0 bytes in 151 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) May 23 13:01:04.359427 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ef4c7ac-2df9-46f5-91f3-ea156583c01a None None] REQUEST_METHOD: `POST` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:01:04.359724 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ef4c7ac-2df9-46f5-91f3-ea156583c01a None None] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:01:04.360002 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ef4c7ac-2df9-46f5-91f3-ea156583c01a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:01:04.387119 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.auth.core [None req-9ef4c7ac-2df9-46f5-91f3-ea156583c01a None None] MFA Rules not processed for user `1394d8fb0f084cfd82bfacff79ad7b5f`. Rule list: `[]` (Enabled: `True`). {{(pid=92056) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 13:01:04.388574 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.fernet_utils [None req-9ef4c7ac-2df9-46f5-91f3-ea156583c01a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92056) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:01:04.424680 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=83,keystone:INSERT=4,keystone:DELETE=13 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 13:01:04.475979 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 985/1976] 10.222.0.30 () {58 vars in 957 bytes} [Mon May 23 13:01:04 2022] POST /identity/v3/auth/tokens => generated 3248 bytes in 118 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) May 23 13:01:04.480512 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1980b0e9-485e-4c41-a2d7-06979dd6dea4 admin admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:01:04.554629 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1980b0e9-485e-4c41-a2d7-06979dd6dea4 admin admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:01:04.560462 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c574c00f-2613-4979-8de6-e1ae8e6ad94c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:01:04.561132 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c574c00f-2613-4979-8de6-e1ae8e6ad94c admin admin] REQUEST_METHOD: `POST` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:01:04.561416 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c574c00f-2613-4979-8de6-e1ae8e6ad94c admin admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:01:04.561695 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c574c00f-2613-4979-8de6-e1ae8e6ad94c admin admin] PATH_INFO: `/v3/projects` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:01:04.562420 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c574c00f-2613-4979-8de6-e1ae8e6ad94c admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:01:04.564367 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c574c00f-2613-4979-8de6-e1ae8e6ad94c admin admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:01:04.720516 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ef4c7ac-2df9-46f5-91f3-ea156583c01a None None] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:01:04.720632 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 992/1977] 10.222.0.30 () {60 vars in 1152 bytes} [Mon May 23 13:01:04 2022] POST /identity/v3/projects => generated 383 bytes in 239 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) May 23 13:01:04.722959 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c574c00f-2613-4979-8de6-e1ae8e6ad94c admin admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:01:04.730373 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.fernet_utils [None req-9ef4c7ac-2df9-46f5-91f3-ea156583c01a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92056) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:01:04.792962 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ef4c7ac-2df9-46f5-91f3-ea156583c01a None None] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:01:04.798345 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a2ffcf2-0426-461d-8fe7-586c6ab33267 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:01:04.799290 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a2ffcf2-0426-461d-8fe7-586c6ab33267 admin admin] REQUEST_METHOD: `DELETE` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:01:04.799580 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a2ffcf2-0426-461d-8fe7-586c6ab33267 admin admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:01:04.799970 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a2ffcf2-0426-461d-8fe7-586c6ab33267 admin admin] PATH_INFO: `/v3/projects/29bf5c208f864889acd12cfd43bc53a4` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:01:04.801082 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a2ffcf2-0426-461d-8fe7-586c6ab33267 admin admin] RBAC: Authorizing `identity:delete_project(project_id=29bf5c208f864889acd12cfd43bc53a4)` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:01:04.803051 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a2ffcf2-0426-461d-8fe7-586c6ab33267 admin admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:01:04.803300 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c574c00f-2613-4979-8de6-e1ae8e6ad94c admin admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:01:04.818581 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-107583c3-dfe6-4aaf-89a9-c615f0c135e4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:01:04.819918 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-107583c3-dfe6-4aaf-89a9-c615f0c135e4 admin admin] REQUEST_METHOD: `POST` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:01:04.820282 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-107583c3-dfe6-4aaf-89a9-c615f0c135e4 admin admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:01:04.820673 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-107583c3-dfe6-4aaf-89a9-c615f0c135e4 admin admin] PATH_INFO: `/v3/users` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:01:04.821370 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-107583c3-dfe6-4aaf-89a9-c615f0c135e4 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:01:04.824397 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-107583c3-dfe6-4aaf-89a9-c615f0c135e4 admin admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:01:04.881814 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 13:01:04.945170 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 993/1978] 10.222.0.30 () {60 vars in 1143 bytes} [Mon May 23 13:01:04 2022] POST /identity/v3/users => generated 298 bytes in 222 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) May 23 13:01:04.946266 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-107583c3-dfe6-4aaf-89a9-c615f0c135e4 admin admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:01:04.951131 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.fernet_utils [None req-107583c3-dfe6-4aaf-89a9-c615f0c135e4 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92057) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:01:04.964612 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-107583c3-dfe6-4aaf-89a9-c615f0c135e4 admin admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:01:04.978745 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-3a2ffcf2-0426-461d-8fe7-586c6ab33267 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '29bf5c208f864889acd12cfd43bc53a4', 'request_id': 'req-3a2ffcf2-0426-461d-8fe7-586c6ab33267'} {{(pid=92056) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:01:05.024918 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2bb4a7bf-d27e-4f9e-8bf1-61b387e48ee7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:01:05.025779 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bb4a7bf-d27e-4f9e-8bf1-61b387e48ee7 admin admin] REQUEST_METHOD: `DELETE` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:01:05.026202 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bb4a7bf-d27e-4f9e-8bf1-61b387e48ee7 admin admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:01:05.026503 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bb4a7bf-d27e-4f9e-8bf1-61b387e48ee7 admin admin] PATH_INFO: `/v3/users/c66bce3e7b31423ca3cdc8bf15713e60` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:01:05.027711 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2bb4a7bf-d27e-4f9e-8bf1-61b387e48ee7 admin admin] RBAC: Authorizing `identity:delete_user(user_id=c66bce3e7b31423ca3cdc8bf15713e60)` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:01:05.029738 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2bb4a7bf-d27e-4f9e-8bf1-61b387e48ee7 admin admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:01:05.053487 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-3a2ffcf2-0426-461d-8fe7-586c6ab33267 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '29bf5c208f864889acd12cfd43bc53a4', 'request_id': 'req-3a2ffcf2-0426-461d-8fe7-586c6ab33267'} {{(pid=92056) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:01:05.061073 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-3a2ffcf2-0426-461d-8fe7-586c6ab33267 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '29bf5c208f864889acd12cfd43bc53a4', 'request_id': 'req-3a2ffcf2-0426-461d-8fe7-586c6ab33267'} {{(pid=92056) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:01:05.071473 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-3a2ffcf2-0426-461d-8fe7-586c6ab33267 admin admin] The token cache is being invalidate because project 29bf5c208f864889acd12cfd43bc53a4 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=92056) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} May 23 13:01:05.072062 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-3a2ffcf2-0426-461d-8fe7-586c6ab33267 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=92056) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:01:05.072547 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-3a2ffcf2-0426-461d-8fe7-586c6ab33267 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=92056) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:01:05.078396 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 986/1979] 10.222.0.30 () {58 vars in 1232 bytes} [Mon May 23 13:01:04 2022] DELETE /identity/v3/projects/29bf5c208f864889acd12cfd43bc53a4 => generated 0 bytes in 364 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) May 23 13:01:05.078396 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a2ffcf2-0426-461d-8fe7-586c6ab33267 admin admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:01:05.078396 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.fernet_utils [None req-3a2ffcf2-0426-461d-8fe7-586c6ab33267 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92056) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:01:05.090280 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a2ffcf2-0426-461d-8fe7-586c6ab33267 admin admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:01:05.095203 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7275393d-7771-46a9-8d64-bedd0cba3cea admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:01:05.096284 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7275393d-7771-46a9-8d64-bedd0cba3cea admin admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:01:05.096799 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7275393d-7771-46a9-8d64-bedd0cba3cea admin admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:01:05.097312 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7275393d-7771-46a9-8d64-bedd0cba3cea admin admin] PATH_INFO: `/v3/roles` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:01:05.098716 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7275393d-7771-46a9-8d64-bedd0cba3cea admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:01:05.101452 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7275393d-7771-46a9-8d64-bedd0cba3cea admin admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:01:05.112972 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 987/1980] 10.222.0.30 () {58 vars in 1121 bytes} [Mon May 23 13:01:05 2022] GET /identity/v3/roles => generated 1420 bytes in 37 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) May 23 13:01:05.117219 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6752823-446d-4256-8691-d757c0f8df3b None None] REQUEST_METHOD: `POST` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:01:05.117219 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6752823-446d-4256-8691-d757c0f8df3b None None] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:01:05.117219 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6752823-446d-4256-8691-d757c0f8df3b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:01:05.133023 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.auth.core [None req-c6752823-446d-4256-8691-d757c0f8df3b None None] MFA Rules not processed for user `0f7119f8fa5c4445977671140ded91a4`. Rule list: `[]` (Enabled: `True`). {{(pid=92056) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 13:01:05.134253 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.fernet_utils [None req-c6752823-446d-4256-8691-d757c0f8df3b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92056) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:01:05.143253 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 988/1981] 10.222.0.30 () {58 vars in 957 bytes} [Mon May 23 13:01:05 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 30 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) May 23 13:01:05.144020 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6752823-446d-4256-8691-d757c0f8df3b None None] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:01:05.153165 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6752823-446d-4256-8691-d757c0f8df3b None None] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:01:05.157833 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b60a5644-384e-4102-bb8e-efd03112127c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:01:05.158714 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b60a5644-384e-4102-bb8e-efd03112127c admin admin] REQUEST_METHOD: `PUT` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:01:05.159002 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b60a5644-384e-4102-bb8e-efd03112127c admin admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:01:05.159283 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b60a5644-384e-4102-bb8e-efd03112127c admin admin] PATH_INFO: `/v3/projects/1ec206af9f5b42d1bf65059555fe306d/users/19294839cf0a4fac8c748f76c95e6f38/roles/bc3c3c9d654940cbaa5177a6be23d1bc` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:01:05.173704 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b60a5644-384e-4102-bb8e-efd03112127c admin admin] RBAC: Authorizing `identity:create_grant(project_id=1ec206af9f5b42d1bf65059555fe306d, user_id=19294839cf0a4fac8c748f76c95e6f38, role_id=bc3c3c9d654940cbaa5177a6be23d1bc)` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:01:05.176471 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b60a5644-384e-4102-bb8e-efd03112127c admin admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:01:05.219107 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=95,keystone:DELETE=4,keystone:INSERT=1 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 13:01:05.219273 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [None req-2bb4a7bf-d27e-4f9e-8bf1-61b387e48ee7 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'c66bce3e7b31423ca3cdc8bf15713e60'} {{(pid=92057) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:01:05.268332 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 989/1982] 10.222.0.30 () {60 vars in 1482 bytes} [Mon May 23 13:01:05 2022] PUT /identity/v3/projects/1ec206af9f5b42d1bf65059555fe306d/users/19294839cf0a4fac8c748f76c95e6f38/roles/bc3c3c9d654940cbaa5177a6be23d1bc => generated 0 bytes in 125 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) May 23 13:01:05.269603 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a770735-caf4-48f7-9cfc-f50605ca4514 None None] REQUEST_METHOD: `POST` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:01:05.269917 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a770735-caf4-48f7-9cfc-f50605ca4514 None None] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:01:05.270203 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a770735-caf4-48f7-9cfc-f50605ca4514 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:01:05.287474 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.auth.core [None req-1a770735-caf4-48f7-9cfc-f50605ca4514 None None] MFA Rules not processed for user `0f7119f8fa5c4445977671140ded91a4`. Rule list: `[]` (Enabled: `True`). {{(pid=92056) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 13:01:05.288666 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.fernet_utils [None req-1a770735-caf4-48f7-9cfc-f50605ca4514 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92056) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:01:05.290682 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [None req-2bb4a7bf-d27e-4f9e-8bf1-61b387e48ee7 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'c66bce3e7b31423ca3cdc8bf15713e60'} {{(pid=92057) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:01:05.298963 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [None req-2bb4a7bf-d27e-4f9e-8bf1-61b387e48ee7 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'c66bce3e7b31423ca3cdc8bf15713e60'} {{(pid=92057) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:01:05.306660 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [None req-2bb4a7bf-d27e-4f9e-8bf1-61b387e48ee7 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'c66bce3e7b31423ca3cdc8bf15713e60'} {{(pid=92057) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:01:05.312775 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [None req-2bb4a7bf-d27e-4f9e-8bf1-61b387e48ee7 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'c66bce3e7b31423ca3cdc8bf15713e60'} {{(pid=92057) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:01:05.318864 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 994/1983] 10.222.0.30 () {58 vars in 1223 bytes} [Mon May 23 13:01:04 2022] DELETE /identity/v3/users/c66bce3e7b31423ca3cdc8bf15713e60 => generated 0 bytes in 373 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) May 23 13:01:05.322628 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2bb4a7bf-d27e-4f9e-8bf1-61b387e48ee7 admin admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:01:05.324019 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.fernet_utils [None req-2bb4a7bf-d27e-4f9e-8bf1-61b387e48ee7 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92057) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:01:05.349284 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 990/1984] 10.222.0.30 () {58 vars in 957 bytes} [Mon May 23 13:01:05 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 80 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) May 23 13:01:05.350739 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7785e92d-29ea-4178-b686-043331d4a58c None None] REQUEST_METHOD: `POST` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:01:05.351034 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7785e92d-29ea-4178-b686-043331d4a58c None None] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:01:05.351313 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7785e92d-29ea-4178-b686-043331d4a58c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:01:05.369249 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.auth.core [None req-7785e92d-29ea-4178-b686-043331d4a58c None None] MFA Rules not processed for user `0f7119f8fa5c4445977671140ded91a4`. Rule list: `[]` (Enabled: `True`). {{(pid=92056) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 13:01:05.371067 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.fernet_utils [None req-7785e92d-29ea-4178-b686-043331d4a58c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92056) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:01:05.390610 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2bb4a7bf-d27e-4f9e-8bf1-61b387e48ee7 admin admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:01:05.395547 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f392611d-09cf-4598-bfb6-3a822df65412 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:01:05.397549 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=86,keystone:INSERT=5,keystone:DELETE=9 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 13:01:05.400291 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f392611d-09cf-4598-bfb6-3a822df65412 admin admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:01:05.400629 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f392611d-09cf-4598-bfb6-3a822df65412 admin admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:01:05.405480 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f392611d-09cf-4598-bfb6-3a822df65412 admin admin] PATH_INFO: `/v3/roles` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:01:05.405480 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f392611d-09cf-4598-bfb6-3a822df65412 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:01:05.405480 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f392611d-09cf-4598-bfb6-3a822df65412 admin admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:01:05.416546 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 995/1985] 10.222.0.30 () {58 vars in 1121 bytes} [Mon May 23 13:01:05 2022] GET /identity/v3/roles => generated 1420 bytes in 94 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) May 23 13:01:05.416546 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f392611d-09cf-4598-bfb6-3a822df65412 admin admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:01:05.424855 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f392611d-09cf-4598-bfb6-3a822df65412 admin admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:01:05.429568 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b8bf35f-569f-479b-a3f8-e1d314050e02 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:01:05.437436 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b8bf35f-569f-479b-a3f8-e1d314050e02 admin admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:01:05.437436 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b8bf35f-569f-479b-a3f8-e1d314050e02 admin admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:01:05.437436 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b8bf35f-569f-479b-a3f8-e1d314050e02 admin admin] PATH_INFO: `/v3/domains` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:01:05.437436 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b8bf35f-569f-479b-a3f8-e1d314050e02 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=92057) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} May 23 13:01:05.437436 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b8bf35f-569f-479b-a3f8-e1d314050e02 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:01:05.437837 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 991/1986] 10.222.0.30 () {58 vars in 957 bytes} [Mon May 23 13:01:05 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 84 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) May 23 13:01:05.438244 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b8bf35f-569f-479b-a3f8-e1d314050e02 admin admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:01:05.438344 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7785e92d-29ea-4178-b686-043331d4a58c None None] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:01:05.445167 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7785e92d-29ea-4178-b686-043331d4a58c None None] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:01:05.450001 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 996/1987] 10.222.0.30 () {58 vars in 1152 bytes} [Mon May 23 13:01:05 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) May 23 13:01:05.450001 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b8bf35f-569f-479b-a3f8-e1d314050e02 admin admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:01:05.450001 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.fernet_utils [None req-8b8bf35f-569f-479b-a3f8-e1d314050e02 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92057) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:01:05.450470 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74581bc0-6d9f-450b-bfca-6f9a587c1c94 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:01:05.451398 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74581bc0-6d9f-450b-bfca-6f9a587c1c94 admin admin] REQUEST_METHOD: `PUT` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:01:05.452002 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74581bc0-6d9f-450b-bfca-6f9a587c1c94 admin admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:01:05.452292 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74581bc0-6d9f-450b-bfca-6f9a587c1c94 admin admin] PATH_INFO: `/v3/projects/1ec206af9f5b42d1bf65059555fe306d/users/19294839cf0a4fac8c748f76c95e6f38/roles/bc3c3c9d654940cbaa5177a6be23d1bc` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:01:05.454557 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74581bc0-6d9f-450b-bfca-6f9a587c1c94 admin admin] RBAC: Authorizing `identity:create_grant(project_id=1ec206af9f5b42d1bf65059555fe306d, user_id=19294839cf0a4fac8c748f76c95e6f38, role_id=bc3c3c9d654940cbaa5177a6be23d1bc)` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:01:05.457569 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74581bc0-6d9f-450b-bfca-6f9a587c1c94 admin admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:01:05.473072 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 992/1988] 10.222.0.30 () {60 vars in 1482 bytes} [Mon May 23 13:01:05 2022] PUT /identity/v3/projects/1ec206af9f5b42d1bf65059555fe306d/users/19294839cf0a4fac8c748f76c95e6f38/roles/bc3c3c9d654940cbaa5177a6be23d1bc => generated 0 bytes in 38 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) May 23 13:01:05.474786 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74581bc0-6d9f-450b-bfca-6f9a587c1c94 admin admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:01:05.508589 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b8bf35f-569f-479b-a3f8-e1d314050e02 admin admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:01:05.544971 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74581bc0-6d9f-450b-bfca-6f9a587c1c94 admin admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:01:05.547595 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=91,keystone:DELETE=7,keystone:INSERT=2 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 13:01:05.551715 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cadb3378-e1ad-4591-be67-faca9a1b855d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:01:05.552376 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cadb3378-e1ad-4591-be67-faca9a1b855d admin admin] REQUEST_METHOD: `POST` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:01:05.552664 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cadb3378-e1ad-4591-be67-faca9a1b855d admin admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:01:05.552938 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cadb3378-e1ad-4591-be67-faca9a1b855d admin admin] PATH_INFO: `/v3/projects` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:01:05.554592 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cadb3378-e1ad-4591-be67-faca9a1b855d admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:01:05.555563 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cadb3378-e1ad-4591-be67-faca9a1b855d admin admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:01:05.562627 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eca095c9-15d5-41e6-a34a-4a04ce5241c3 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2eb83ca38e79449095f3a2f947bbb06a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9c389ae65aaa4542bb5a5c4fd0bacf5b', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:01:05.563274 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eca095c9-15d5-41e6-a34a-4a04ce5241c3 service neutron] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:01:05.563556 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eca095c9-15d5-41e6-a34a-4a04ce5241c3 service neutron] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:01:05.563827 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eca095c9-15d5-41e6-a34a-4a04ce5241c3 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:01:05.573121 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eca095c9-15d5-41e6-a34a-4a04ce5241c3 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:01:05.574587 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eca095c9-15d5-41e6-a34a-4a04ce5241c3 service neutron] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:01:05.584341 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 997/1989] 10.222.0.30 () {60 vars in 1365 bytes} [Mon May 23 13:01:05 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 137 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) May 23 13:01:05.585742 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-727a9fdc-dcdd-4d8c-9c9b-3e5b4acf4565 None None] REQUEST_METHOD: `POST` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:01:05.586062 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-727a9fdc-dcdd-4d8c-9c9b-3e5b4acf4565 None None] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:01:05.586339 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-727a9fdc-dcdd-4d8c-9c9b-3e5b4acf4565 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:01:05.614869 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.auth.core [None req-727a9fdc-dcdd-4d8c-9c9b-3e5b4acf4565 None None] MFA Rules not processed for user `19294839cf0a4fac8c748f76c95e6f38`. Rule list: `[]` (Enabled: `True`). {{(pid=92057) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 13:01:05.616101 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.fernet_utils [None req-727a9fdc-dcdd-4d8c-9c9b-3e5b4acf4565 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92057) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:01:05.634590 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=99,keystone:DELETE=1 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 13:01:05.656421 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 993/1990] 10.222.0.30 () {60 vars in 1152 bytes} [Mon May 23 13:01:05 2022] POST /identity/v3/projects => generated 373 bytes in 183 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) May 23 13:01:05.660515 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cadb3378-e1ad-4591-be67-faca9a1b855d admin admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:01:05.689924 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 998/1991] 10.222.0.30 () {58 vars in 957 bytes} [Mon May 23 13:01:05 2022] POST /identity/v3/auth/tokens => generated 3317 bytes in 105 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) May 23 13:01:05.703201 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-727a9fdc-dcdd-4d8c-9c9b-3e5b4acf4565 None None] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:01:05.704750 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.fernet_utils [None req-727a9fdc-dcdd-4d8c-9c9b-3e5b4acf4565 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92057) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:01:05.734494 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cadb3378-e1ad-4591-be67-faca9a1b855d admin admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:01:05.740424 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2107e3f-832b-4847-bac6-532a2e04246e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:01:05.741247 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2107e3f-832b-4847-bac6-532a2e04246e admin admin] REQUEST_METHOD: `POST` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:01:05.741564 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2107e3f-832b-4847-bac6-532a2e04246e admin admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:01:05.741816 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2107e3f-832b-4847-bac6-532a2e04246e admin admin] PATH_INFO: `/v3/users` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:01:05.742546 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f2107e3f-832b-4847-bac6-532a2e04246e admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:01:05.744473 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f2107e3f-832b-4847-bac6-532a2e04246e admin admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:01:05.769182 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-727a9fdc-dcdd-4d8c-9c9b-3e5b4acf4565 None None] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:01:05.829693 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-256b0327-b4cb-4c11-b5b6-8f32c090bc3d service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1666097f8ed545c6b0e4809c93e4775a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9c389ae65aaa4542bb5a5c4fd0bacf5b', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:01:05.830393 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-256b0327-b4cb-4c11-b5b6-8f32c090bc3d service cinder] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:01:05.830686 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-256b0327-b4cb-4c11-b5b6-8f32c090bc3d service cinder] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:01:05.830964 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-256b0327-b4cb-4c11-b5b6-8f32c090bc3d service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:01:05.875881 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 994/1992] 10.222.0.30 () {60 vars in 1143 bytes} [Mon May 23 13:01:05 2022] POST /identity/v3/users => generated 287 bytes in 216 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) May 23 13:01:05.880283 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2107e3f-832b-4847-bac6-532a2e04246e admin admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:01:05.895850 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2107e3f-832b-4847-bac6-532a2e04246e admin admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:01:05.915118 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb740977-af37-400f-ab51-a2b109af1bd8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:01:05.915118 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb740977-af37-400f-ab51-a2b109af1bd8 admin admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:01:05.915118 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb740977-af37-400f-ab51-a2b109af1bd8 admin admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:01:05.915118 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb740977-af37-400f-ab51-a2b109af1bd8 admin admin] PATH_INFO: `/v3/roles` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:01:05.915453 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb740977-af37-400f-ab51-a2b109af1bd8 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:01:05.915453 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb740977-af37-400f-ab51-a2b109af1bd8 admin admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:01:05.930096 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 995/1993] 10.222.0.30 () {58 vars in 1121 bytes} [Mon May 23 13:01:05 2022] GET /identity/v3/roles => generated 1420 bytes in 50 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) May 23 13:01:05.931033 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb740977-af37-400f-ab51-a2b109af1bd8 admin admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:01:05.942401 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb740977-af37-400f-ab51-a2b109af1bd8 admin admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:01:05.952498 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-256b0327-b4cb-4c11-b5b6-8f32c090bc3d service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:01:05.952651 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed410da4-f640-4204-acf6-8bc2994a2212 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:01:05.953183 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed410da4-f640-4204-acf6-8bc2994a2212 admin admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:01:05.953468 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed410da4-f640-4204-acf6-8bc2994a2212 admin admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:01:05.953750 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed410da4-f640-4204-acf6-8bc2994a2212 admin admin] PATH_INFO: `/v3/roles` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:01:05.954029 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-256b0327-b4cb-4c11-b5b6-8f32c090bc3d service cinder] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:01:05.954969 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed410da4-f640-4204-acf6-8bc2994a2212 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:01:05.956958 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed410da4-f640-4204-acf6-8bc2994a2212 admin admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:01:05.965865 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 999/1994] 10.222.0.30 () {60 vars in 1365 bytes} [Mon May 23 13:01:05 2022] GET /identity/v3/auth/tokens => generated 3317 bytes in 263 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) May 23 13:01:05.981924 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 996/1995] 10.222.0.30 () {58 vars in 1121 bytes} [Mon May 23 13:01:05 2022] GET /identity/v3/roles => generated 1420 bytes in 45 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) May 23 13:01:05.983555 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-256b0327-b4cb-4c11-b5b6-8f32c090bc3d service cinder] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:01:05.995657 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-256b0327-b4cb-4c11-b5b6-8f32c090bc3d service cinder] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:01:06.000929 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32c6473e-f981-4749-833b-b4670e7f83c3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:01:06.002230 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32c6473e-f981-4749-833b-b4670e7f83c3 admin admin] REQUEST_METHOD: `PUT` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:01:06.002796 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32c6473e-f981-4749-833b-b4670e7f83c3 admin admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:01:06.003622 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32c6473e-f981-4749-833b-b4670e7f83c3 admin admin] PATH_INFO: `/v3/projects/304365ce2f104a70b78b1f9750d22022/users/f57cf434ad6247528e3e629cdfa7c980/roles/80fb22507c9a4ff09d44d8f12f6fff1a` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:01:06.023322 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32c6473e-f981-4749-833b-b4670e7f83c3 admin admin] RBAC: Authorizing `identity:create_grant(project_id=304365ce2f104a70b78b1f9750d22022, user_id=f57cf434ad6247528e3e629cdfa7c980, role_id=80fb22507c9a4ff09d44d8f12f6fff1a)` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:01:06.026054 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32c6473e-f981-4749-833b-b4670e7f83c3 admin admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:01:06.072164 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed410da4-f640-4204-acf6-8bc2994a2212 admin admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:01:06.074556 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.fernet_utils [None req-ed410da4-f640-4204-acf6-8bc2994a2212 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92056) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:01:06.090995 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed410da4-f640-4204-acf6-8bc2994a2212 admin admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:01:06.096716 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91841c1e-247b-4d15-b541-daa86caaa459 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:01:06.097640 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91841c1e-247b-4d15-b541-daa86caaa459 admin admin] REQUEST_METHOD: `DELETE` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:01:06.098389 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 13:01:06.099189 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91841c1e-247b-4d15-b541-daa86caaa459 admin admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:01:06.099644 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91841c1e-247b-4d15-b541-daa86caaa459 admin admin] PATH_INFO: `/v3/projects/1f7aeeb0a04e43e7a2d5f10bade807d0` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:01:06.101930 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-91841c1e-247b-4d15-b541-daa86caaa459 admin admin] RBAC: Authorizing `identity:delete_project(project_id=1f7aeeb0a04e43e7a2d5f10bade807d0)` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:01:06.105891 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-91841c1e-247b-4d15-b541-daa86caaa459 admin admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:01:06.109926 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 1000/1996] 10.222.0.30 () {60 vars in 1482 bytes} [Mon May 23 13:01:05 2022] PUT /identity/v3/projects/304365ce2f104a70b78b1f9750d22022/users/f57cf434ad6247528e3e629cdfa7c980/roles/80fb22507c9a4ff09d44d8f12f6fff1a => generated 0 bytes in 125 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) May 23 13:01:06.116131 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 13:01:06.122120 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1da490c3-ce09-4d0e-a4de-67efb3f72bad None None] REQUEST_METHOD: `POST` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:01:06.123159 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1da490c3-ce09-4d0e-a4de-67efb3f72bad None None] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:01:06.123793 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1da490c3-ce09-4d0e-a4de-67efb3f72bad None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:01:06.157257 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.auth.core [None req-1da490c3-ce09-4d0e-a4de-67efb3f72bad None None] MFA Rules not processed for user `f57cf434ad6247528e3e629cdfa7c980`. Rule list: `[]` (Enabled: `True`). {{(pid=92057) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 13:01:06.162870 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.fernet_utils [None req-1da490c3-ce09-4d0e-a4de-67efb3f72bad None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92057) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:01:06.238719 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 1001/1997] 10.222.0.30 () {58 vars in 957 bytes} [Mon May 23 13:01:06 2022] POST /identity/v3/auth/tokens => generated 3240 bytes in 119 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) May 23 13:01:06.239796 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1da490c3-ce09-4d0e-a4de-67efb3f72bad None None] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:01:06.251551 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-91841c1e-247b-4d15-b541-daa86caaa459 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '1f7aeeb0a04e43e7a2d5f10bade807d0', 'request_id': 'req-91841c1e-247b-4d15-b541-daa86caaa459'} {{(pid=92056) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:01:06.299405 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1da490c3-ce09-4d0e-a4de-67efb3f72bad None None] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:01:06.333173 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-91841c1e-247b-4d15-b541-daa86caaa459 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '1f7aeeb0a04e43e7a2d5f10bade807d0', 'request_id': 'req-91841c1e-247b-4d15-b541-daa86caaa459'} {{(pid=92056) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:01:06.340058 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-91841c1e-247b-4d15-b541-daa86caaa459 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '1f7aeeb0a04e43e7a2d5f10bade807d0', 'request_id': 'req-91841c1e-247b-4d15-b541-daa86caaa459'} {{(pid=92056) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:01:06.349679 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5cf934a6-413d-4609-98c6-40c33a021ba2 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1666097f8ed545c6b0e4809c93e4775a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9c389ae65aaa4542bb5a5c4fd0bacf5b', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:01:06.350367 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cf934a6-413d-4609-98c6-40c33a021ba2 service cinder] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:01:06.350649 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cf934a6-413d-4609-98c6-40c33a021ba2 service cinder] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:01:06.350753 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-91841c1e-247b-4d15-b541-daa86caaa459 admin admin] The token cache is being invalidate because project 1f7aeeb0a04e43e7a2d5f10bade807d0 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=92056) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} May 23 13:01:06.350952 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cf934a6-413d-4609-98c6-40c33a021ba2 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:01:06.351076 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-91841c1e-247b-4d15-b541-daa86caaa459 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=92056) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:01:06.351653 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-91841c1e-247b-4d15-b541-daa86caaa459 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=92056) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:01:06.352816 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.fernet_utils [None req-5cf934a6-413d-4609-98c6-40c33a021ba2 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92057) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:01:06.353450 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 997/1998] 10.222.0.30 () {58 vars in 1232 bytes} [Mon May 23 13:01:06 2022] DELETE /identity/v3/projects/1f7aeeb0a04e43e7a2d5f10bade807d0 => generated 0 bytes in 282 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) May 23 13:01:06.357681 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91841c1e-247b-4d15-b541-daa86caaa459 admin admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:01:06.359650 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.fernet_utils [None req-91841c1e-247b-4d15-b541-daa86caaa459 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92056) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:01:06.413678 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5cf934a6-413d-4609-98c6-40c33a021ba2 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:01:06.415178 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5cf934a6-413d-4609-98c6-40c33a021ba2 service cinder] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:01:06.424767 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 1002/1999] 10.222.0.30 () {60 vars in 1365 bytes} [Mon May 23 13:01:06 2022] GET /identity/v3/auth/tokens => generated 3248 bytes in 185 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) May 23 13:01:06.426234 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91841c1e-247b-4d15-b541-daa86caaa459 admin admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:01:06.430878 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c86c5ac6-5510-4068-874e-4b7e9c021d29 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:01:06.431754 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c86c5ac6-5510-4068-874e-4b7e9c021d29 admin admin] REQUEST_METHOD: `DELETE` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:01:06.432036 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c86c5ac6-5510-4068-874e-4b7e9c021d29 admin admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:01:06.432312 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c86c5ac6-5510-4068-874e-4b7e9c021d29 admin admin] PATH_INFO: `/v3/users/2ba91aaaa18a48de964d9079b1df1182` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:01:06.434745 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c86c5ac6-5510-4068-874e-4b7e9c021d29 admin admin] RBAC: Authorizing `identity:delete_user(user_id=2ba91aaaa18a48de964d9079b1df1182)` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:01:06.436749 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c86c5ac6-5510-4068-874e-4b7e9c021d29 admin admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:01:06.445760 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=100 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 13:01:06.453042 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5cf934a6-413d-4609-98c6-40c33a021ba2 service cinder] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:01:06.458493 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.fernet_utils [None req-5cf934a6-413d-4609-98c6-40c33a021ba2 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92057) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:01:06.471011 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5cf934a6-413d-4609-98c6-40c33a021ba2 service cinder] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:01:06.475068 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ce24f05-d010-4431-ab1c-8b70a21202f2 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1666097f8ed545c6b0e4809c93e4775a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9c389ae65aaa4542bb5a5c4fd0bacf5b', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:01:06.475714 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ce24f05-d010-4431-ab1c-8b70a21202f2 service cinder] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:01:06.475999 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ce24f05-d010-4431-ab1c-8b70a21202f2 service cinder] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:01:06.476279 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ce24f05-d010-4431-ab1c-8b70a21202f2 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:01:06.478243 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.fernet_utils [None req-2ce24f05-d010-4431-ab1c-8b70a21202f2 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92057) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:01:06.542506 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ce24f05-d010-4431-ab1c-8b70a21202f2 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:01:06.544257 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ce24f05-d010-4431-ab1c-8b70a21202f2 service cinder] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:01:06.553456 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 1003/2000] 10.222.0.30 () {60 vars in 1365 bytes} [Mon May 23 13:01:06 2022] GET /identity/v3/auth/tokens => generated 3240 bytes in 102 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) May 23 13:01:06.574850 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ce24f05-d010-4431-ab1c-8b70a21202f2 service cinder] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:01:06.576302 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.fernet_utils [None req-2ce24f05-d010-4431-ab1c-8b70a21202f2 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92057) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:01:06.591076 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ce24f05-d010-4431-ab1c-8b70a21202f2 service cinder] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:01:06.596089 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b2965cc-d04c-4344-ba63-fa3b7c556b95 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:01:06.596963 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b2965cc-d04c-4344-ba63-fa3b7c556b95 admin admin] REQUEST_METHOD: `DELETE` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:01:06.597243 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b2965cc-d04c-4344-ba63-fa3b7c556b95 admin admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:01:06.598159 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b2965cc-d04c-4344-ba63-fa3b7c556b95 admin admin] PATH_INFO: `/v3/users/f57cf434ad6247528e3e629cdfa7c980` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:01:06.598931 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b2965cc-d04c-4344-ba63-fa3b7c556b95 admin admin] RBAC: Authorizing `identity:delete_user(user_id=f57cf434ad6247528e3e629cdfa7c980)` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:01:06.600858 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b2965cc-d04c-4344-ba63-fa3b7c556b95 admin admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:01:06.693849 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-c86c5ac6-5510-4068-874e-4b7e9c021d29 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '2ba91aaaa18a48de964d9079b1df1182'} {{(pid=92056) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:01:06.694598 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-c86c5ac6-5510-4068-874e-4b7e9c021d29 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '2ba91aaaa18a48de964d9079b1df1182'} {{(pid=92056) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:01:06.777069 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-c86c5ac6-5510-4068-874e-4b7e9c021d29 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '2ba91aaaa18a48de964d9079b1df1182'} {{(pid=92056) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:01:06.795044 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-c86c5ac6-5510-4068-874e-4b7e9c021d29 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '2ba91aaaa18a48de964d9079b1df1182'} {{(pid=92056) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:01:06.803290 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-c86c5ac6-5510-4068-874e-4b7e9c021d29 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '2ba91aaaa18a48de964d9079b1df1182'} {{(pid=92056) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:01:06.812055 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 998/2001] 10.222.0.30 () {58 vars in 1223 bytes} [Mon May 23 13:01:06 2022] DELETE /identity/v3/users/2ba91aaaa18a48de964d9079b1df1182 => generated 0 bytes in 454 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) May 23 13:01:06.831792 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [None req-7b2965cc-d04c-4344-ba63-fa3b7c556b95 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'f57cf434ad6247528e3e629cdfa7c980'} {{(pid=92057) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:01:06.950395 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [None req-7b2965cc-d04c-4344-ba63-fa3b7c556b95 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'f57cf434ad6247528e3e629cdfa7c980'} {{(pid=92057) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:01:06.958044 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [None req-7b2965cc-d04c-4344-ba63-fa3b7c556b95 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'f57cf434ad6247528e3e629cdfa7c980'} {{(pid=92057) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:01:06.958695 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [None req-7b2965cc-d04c-4344-ba63-fa3b7c556b95 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'f57cf434ad6247528e3e629cdfa7c980'} {{(pid=92057) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:01:06.971740 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [None req-7b2965cc-d04c-4344-ba63-fa3b7c556b95 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'f57cf434ad6247528e3e629cdfa7c980'} {{(pid=92057) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:01:06.984556 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 1004/2002] 10.222.0.30 () {58 vars in 1223 bytes} [Mon May 23 13:01:06 2022] DELETE /identity/v3/users/f57cf434ad6247528e3e629cdfa7c980 => generated 0 bytes in 405 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) May 23 13:01:06.989249 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcf0b7b6-682b-4a19-8d79-c621e4b41c89 None None] REQUEST_METHOD: `POST` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:01:06.989249 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcf0b7b6-682b-4a19-8d79-c621e4b41c89 None None] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:01:06.989249 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcf0b7b6-682b-4a19-8d79-c621e4b41c89 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:01:07.008948 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.auth.core [None req-fcf0b7b6-682b-4a19-8d79-c621e4b41c89 None None] MFA Rules not processed for user `0f7119f8fa5c4445977671140ded91a4`. Rule list: `[]` (Enabled: `True`). {{(pid=92056) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 13:01:07.011188 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.fernet_utils [None req-fcf0b7b6-682b-4a19-8d79-c621e4b41c89 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92056) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:01:07.023040 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=81,keystone:DELETE=17,keystone:INSERT=2 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 13:01:07.089954 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 999/2003] 10.222.0.30 () {58 vars in 957 bytes} [Mon May 23 13:01:06 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 105 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) May 23 13:01:07.100382 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b2965cc-d04c-4344-ba63-fa3b7c556b95 admin admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:01:07.102018 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.fernet_utils [None req-7b2965cc-d04c-4344-ba63-fa3b7c556b95 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92057) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:01:07.171175 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b2965cc-d04c-4344-ba63-fa3b7c556b95 admin admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:01:07.175182 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80cd5120-1de1-450b-ae6c-ba55458809da service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2eb83ca38e79449095f3a2f947bbb06a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9c389ae65aaa4542bb5a5c4fd0bacf5b', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:01:07.175837 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80cd5120-1de1-450b-ae6c-ba55458809da service neutron] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:01:07.176157 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80cd5120-1de1-450b-ae6c-ba55458809da service neutron] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:01:07.176432 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80cd5120-1de1-450b-ae6c-ba55458809da service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:01:07.190301 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80cd5120-1de1-450b-ae6c-ba55458809da service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:01:07.191808 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80cd5120-1de1-450b-ae6c-ba55458809da service neutron] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:01:07.202079 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fcf0b7b6-682b-4a19-8d79-c621e4b41c89 None None] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:01:07.203548 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.fernet_utils [None req-fcf0b7b6-682b-4a19-8d79-c621e4b41c89 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92056) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:01:07.205558 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 1005/2004] 10.222.0.30 () {60 vars in 1365 bytes} [Mon May 23 13:01:07 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 106 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) May 23 13:01:07.219146 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fcf0b7b6-682b-4a19-8d79-c621e4b41c89 None None] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:01:07.222751 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c3d2fd3-3855-4f96-80e8-7eaad29594f6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:01:07.223751 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c3d2fd3-3855-4f96-80e8-7eaad29594f6 admin admin] REQUEST_METHOD: `DELETE` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:01:07.224101 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c3d2fd3-3855-4f96-80e8-7eaad29594f6 admin admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:01:07.224440 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c3d2fd3-3855-4f96-80e8-7eaad29594f6 admin admin] PATH_INFO: `/v3/users/1394d8fb0f084cfd82bfacff79ad7b5f` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:01:07.225723 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c3d2fd3-3855-4f96-80e8-7eaad29594f6 admin admin] RBAC: Authorizing `identity:delete_user(user_id=1394d8fb0f084cfd82bfacff79ad7b5f)` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:01:07.227740 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c3d2fd3-3855-4f96-80e8-7eaad29594f6 admin admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:01:07.274968 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80cd5120-1de1-450b-ae6c-ba55458809da service neutron] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:01:07.279301 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.fernet_utils [None req-80cd5120-1de1-450b-ae6c-ba55458809da service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92057) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:01:07.305709 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80cd5120-1de1-450b-ae6c-ba55458809da service neutron] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:01:07.312038 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75077355-d0f9-4a22-9728-d5d3b2080fa7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:01:07.312679 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75077355-d0f9-4a22-9728-d5d3b2080fa7 admin admin] REQUEST_METHOD: `DELETE` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:01:07.312970 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75077355-d0f9-4a22-9728-d5d3b2080fa7 admin admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:01:07.313252 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75077355-d0f9-4a22-9728-d5d3b2080fa7 admin admin] PATH_INFO: `/v3/projects/cd1c87d63b9d408daec92fa22bdf787f` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:01:07.314506 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75077355-d0f9-4a22-9728-d5d3b2080fa7 admin admin] RBAC: Authorizing `identity:delete_project(project_id=cd1c87d63b9d408daec92fa22bdf787f)` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:01:07.316434 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75077355-d0f9-4a22-9728-d5d3b2080fa7 admin admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:01:07.324681 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 13:01:07.492145 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [None req-75077355-d0f9-4a22-9728-d5d3b2080fa7 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'cd1c87d63b9d408daec92fa22bdf787f', 'request_id': 'req-75077355-d0f9-4a22-9728-d5d3b2080fa7'} {{(pid=92057) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:01:07.499570 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-6c3d2fd3-3855-4f96-80e8-7eaad29594f6 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '1394d8fb0f084cfd82bfacff79ad7b5f'} {{(pid=92056) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:01:07.500591 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-6c3d2fd3-3855-4f96-80e8-7eaad29594f6 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '1394d8fb0f084cfd82bfacff79ad7b5f'} {{(pid=92056) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:01:07.502680 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [None req-75077355-d0f9-4a22-9728-d5d3b2080fa7 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'cd1c87d63b9d408daec92fa22bdf787f', 'request_id': 'req-75077355-d0f9-4a22-9728-d5d3b2080fa7'} {{(pid=92057) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:01:07.582128 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-6c3d2fd3-3855-4f96-80e8-7eaad29594f6 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '1394d8fb0f084cfd82bfacff79ad7b5f'} {{(pid=92056) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:01:07.596223 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-6c3d2fd3-3855-4f96-80e8-7eaad29594f6 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '1394d8fb0f084cfd82bfacff79ad7b5f'} {{(pid=92056) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:01:07.602884 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-6c3d2fd3-3855-4f96-80e8-7eaad29594f6 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '1394d8fb0f084cfd82bfacff79ad7b5f'} {{(pid=92056) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:01:07.610746 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 1000/2005] 10.222.0.30 () {58 vars in 1223 bytes} [Mon May 23 13:01:07 2022] DELETE /identity/v3/users/1394d8fb0f084cfd82bfacff79ad7b5f => generated 0 bytes in 409 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) May 23 13:01:07.615711 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13febf2c-26cc-4f48-8b40-e5df491520a9 None None] REQUEST_METHOD: `POST` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:01:07.616004 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13febf2c-26cc-4f48-8b40-e5df491520a9 None None] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:01:07.616290 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13febf2c-26cc-4f48-8b40-e5df491520a9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:01:07.630470 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [None req-75077355-d0f9-4a22-9728-d5d3b2080fa7 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'cd1c87d63b9d408daec92fa22bdf787f', 'request_id': 'req-75077355-d0f9-4a22-9728-d5d3b2080fa7'} {{(pid=92057) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:01:07.636309 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.auth.core [None req-13febf2c-26cc-4f48-8b40-e5df491520a9 None None] MFA Rules not processed for user `0f7119f8fa5c4445977671140ded91a4`. Rule list: `[]` (Enabled: `True`). {{(pid=92056) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 13:01:07.638210 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.fernet_utils [None req-13febf2c-26cc-4f48-8b40-e5df491520a9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92056) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:01:07.645913 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [None req-75077355-d0f9-4a22-9728-d5d3b2080fa7 admin admin] The token cache is being invalidate because project cd1c87d63b9d408daec92fa22bdf787f was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=92057) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} May 23 13:01:07.645913 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [None req-75077355-d0f9-4a22-9728-d5d3b2080fa7 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=92057) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:01:07.646839 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [None req-75077355-d0f9-4a22-9728-d5d3b2080fa7 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=92057) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:01:07.648565 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 1006/2006] 10.222.0.30 () {58 vars in 1232 bytes} [Mon May 23 13:01:07 2022] DELETE /identity/v3/projects/cd1c87d63b9d408daec92fa22bdf787f => generated 0 bytes in 374 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) May 23 13:01:07.653114 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75077355-d0f9-4a22-9728-d5d3b2080fa7 admin admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:01:07.654708 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.fernet_utils [None req-75077355-d0f9-4a22-9728-d5d3b2080fa7 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92057) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:01:07.710953 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 1001/2007] 10.222.0.30 () {58 vars in 957 bytes} [Mon May 23 13:01:07 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 96 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) May 23 13:01:07.720669 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75077355-d0f9-4a22-9728-d5d3b2080fa7 admin admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:01:07.721698 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13febf2c-26cc-4f48-8b40-e5df491520a9 None None] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:01:07.723420 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.fernet_utils [None req-13febf2c-26cc-4f48-8b40-e5df491520a9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92056) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:01:07.725321 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-688c9c50-feb4-40a0-ba0e-5516e41fe00a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:01:07.726308 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-688c9c50-feb4-40a0-ba0e-5516e41fe00a admin admin] REQUEST_METHOD: `DELETE` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:01:07.726528 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-688c9c50-feb4-40a0-ba0e-5516e41fe00a admin admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:01:07.726799 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-688c9c50-feb4-40a0-ba0e-5516e41fe00a admin admin] PATH_INFO: `/v3/users/a4cade4474ff444d9734a0b688dd2a83` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:01:07.727975 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-688c9c50-feb4-40a0-ba0e-5516e41fe00a admin admin] RBAC: Authorizing `identity:delete_user(user_id=a4cade4474ff444d9734a0b688dd2a83)` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:01:07.730080 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-688c9c50-feb4-40a0-ba0e-5516e41fe00a admin admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:01:07.756542 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 13:01:07.791047 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13febf2c-26cc-4f48-8b40-e5df491520a9 None None] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:01:07.795081 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6dee7949-72b1-44da-b7d0-ef7c007dc209 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2eb83ca38e79449095f3a2f947bbb06a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9c389ae65aaa4542bb5a5c4fd0bacf5b', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:01:07.795734 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6dee7949-72b1-44da-b7d0-ef7c007dc209 service neutron] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:01:07.796017 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6dee7949-72b1-44da-b7d0-ef7c007dc209 service neutron] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:01:07.796295 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6dee7949-72b1-44da-b7d0-ef7c007dc209 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:01:07.806352 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6dee7949-72b1-44da-b7d0-ef7c007dc209 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:01:07.808346 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6dee7949-72b1-44da-b7d0-ef7c007dc209 service neutron] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:01:07.817913 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 1002/2008] 10.222.0.30 () {60 vars in 1365 bytes} [Mon May 23 13:01:07 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 97 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) May 23 13:01:07.818707 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6dee7949-72b1-44da-b7d0-ef7c007dc209 service neutron] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:01:07.819981 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.fernet_utils [None req-6dee7949-72b1-44da-b7d0-ef7c007dc209 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92056) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:01:07.833123 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6dee7949-72b1-44da-b7d0-ef7c007dc209 service neutron] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:01:07.837919 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a43cafbd-43de-4c31-852d-1dec104f5802 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:01:07.838951 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a43cafbd-43de-4c31-852d-1dec104f5802 admin admin] REQUEST_METHOD: `DELETE` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:01:07.839389 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a43cafbd-43de-4c31-852d-1dec104f5802 admin admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:01:07.840792 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a43cafbd-43de-4c31-852d-1dec104f5802 admin admin] PATH_INFO: `/v3/projects/304365ce2f104a70b78b1f9750d22022` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:01:07.840912 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a43cafbd-43de-4c31-852d-1dec104f5802 admin admin] RBAC: Authorizing `identity:delete_project(project_id=304365ce2f104a70b78b1f9750d22022)` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:01:07.843109 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a43cafbd-43de-4c31-852d-1dec104f5802 admin admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:01:08.022172 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [None req-688c9c50-feb4-40a0-ba0e-5516e41fe00a admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'a4cade4474ff444d9734a0b688dd2a83'} {{(pid=92057) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:01:08.025617 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-a43cafbd-43de-4c31-852d-1dec104f5802 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '304365ce2f104a70b78b1f9750d22022', 'request_id': 'req-a43cafbd-43de-4c31-852d-1dec104f5802'} {{(pid=92056) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:01:08.113672 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [None req-688c9c50-feb4-40a0-ba0e-5516e41fe00a admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'a4cade4474ff444d9734a0b688dd2a83'} {{(pid=92057) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:01:08.122131 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [None req-688c9c50-feb4-40a0-ba0e-5516e41fe00a admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'a4cade4474ff444d9734a0b688dd2a83'} {{(pid=92057) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:01:08.122833 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [None req-688c9c50-feb4-40a0-ba0e-5516e41fe00a admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'a4cade4474ff444d9734a0b688dd2a83'} {{(pid=92057) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:01:08.136452 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [None req-688c9c50-feb4-40a0-ba0e-5516e41fe00a admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'a4cade4474ff444d9734a0b688dd2a83'} {{(pid=92057) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:01:08.142812 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 1007/2009] 10.222.0.30 () {58 vars in 1223 bytes} [Mon May 23 13:01:07 2022] DELETE /identity/v3/users/a4cade4474ff444d9734a0b688dd2a83 => generated 0 bytes in 490 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) May 23 13:01:08.162330 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-a43cafbd-43de-4c31-852d-1dec104f5802 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '304365ce2f104a70b78b1f9750d22022', 'request_id': 'req-a43cafbd-43de-4c31-852d-1dec104f5802'} {{(pid=92056) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:01:08.169530 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-a43cafbd-43de-4c31-852d-1dec104f5802 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '304365ce2f104a70b78b1f9750d22022', 'request_id': 'req-a43cafbd-43de-4c31-852d-1dec104f5802'} {{(pid=92056) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:01:08.180370 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-a43cafbd-43de-4c31-852d-1dec104f5802 admin admin] The token cache is being invalidate because project 304365ce2f104a70b78b1f9750d22022 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=92056) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} May 23 13:01:08.180656 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-a43cafbd-43de-4c31-852d-1dec104f5802 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=92056) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:01:08.181327 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-a43cafbd-43de-4c31-852d-1dec104f5802 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=92056) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:01:08.183959 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 1003/2010] 10.222.0.30 () {58 vars in 1232 bytes} [Mon May 23 13:01:07 2022] DELETE /identity/v3/projects/304365ce2f104a70b78b1f9750d22022 => generated 0 bytes in 365 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) May 23 13:01:08.190668 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9a0c03d-9c55-418a-8e91-74a29233fc82 None None] REQUEST_METHOD: `POST` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:01:08.190963 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9a0c03d-9c55-418a-8e91-74a29233fc82 None None] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:01:08.191270 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9a0c03d-9c55-418a-8e91-74a29233fc82 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:01:08.209553 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.auth.core [None req-e9a0c03d-9c55-418a-8e91-74a29233fc82 None None] MFA Rules not processed for user `0f7119f8fa5c4445977671140ded91a4`. Rule list: `[]` (Enabled: `True`). {{(pid=92057) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 13:01:08.210805 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.fernet_utils [None req-e9a0c03d-9c55-418a-8e91-74a29233fc82 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92057) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:01:08.219496 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=81,keystone:DELETE=17,keystone:INSERT=2 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 13:01:08.251239 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a43cafbd-43de-4c31-852d-1dec104f5802 admin admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:01:08.258227 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.fernet_utils [None req-a43cafbd-43de-4c31-852d-1dec104f5802 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92056) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:01:08.296655 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 1008/2011] 10.222.0.30 () {58 vars in 957 bytes} [Mon May 23 13:01:08 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 107 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) May 23 13:01:08.304825 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e78c4e74-cbe5-44d6-9223-f7b393bf82ec None None] REQUEST_METHOD: `POST` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:01:08.305090 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e78c4e74-cbe5-44d6-9223-f7b393bf82ec None None] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:01:08.305376 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e78c4e74-cbe5-44d6-9223-f7b393bf82ec None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:01:08.325467 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.auth.core [None req-e78c4e74-cbe5-44d6-9223-f7b393bf82ec None None] MFA Rules not processed for user `0f7119f8fa5c4445977671140ded91a4`. Rule list: `[]` (Enabled: `True`). {{(pid=92057) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 13:01:08.326711 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.fernet_utils [None req-e78c4e74-cbe5-44d6-9223-f7b393bf82ec None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92057) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:01:08.335563 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 1009/2012] 10.222.0.30 () {58 vars in 957 bytes} [Mon May 23 13:01:08 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) May 23 13:01:08.340235 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a43cafbd-43de-4c31-852d-1dec104f5802 admin admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:01:08.340911 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e78c4e74-cbe5-44d6-9223-f7b393bf82ec None None] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:01:08.345066 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb41fc26-8994-439f-9ca2-e78540108565 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:01:08.345964 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb41fc26-8994-439f-9ca2-e78540108565 admin admin] REQUEST_METHOD: `DELETE` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:01:08.346253 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb41fc26-8994-439f-9ca2-e78540108565 admin admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:01:08.346535 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb41fc26-8994-439f-9ca2-e78540108565 admin admin] PATH_INFO: `/v3/projects/cb8d79a531c84f92a60c08ffccbeda0f` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:01:08.347738 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb41fc26-8994-439f-9ca2-e78540108565 admin admin] RBAC: Authorizing `identity:delete_project(project_id=cb8d79a531c84f92a60c08ffccbeda0f)` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:01:08.349661 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb41fc26-8994-439f-9ca2-e78540108565 admin admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:01:08.350636 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e78c4e74-cbe5-44d6-9223-f7b393bf82ec None None] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:01:08.355606 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-963e562d-f59c-4d89-a5d2-3262c2b3a33c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:01:08.356259 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-963e562d-f59c-4d89-a5d2-3262c2b3a33c admin admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:01:08.356540 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-963e562d-f59c-4d89-a5d2-3262c2b3a33c admin admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:01:08.356809 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-963e562d-f59c-4d89-a5d2-3262c2b3a33c admin admin] PATH_INFO: `/v3/domains` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:01:08.357517 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-963e562d-f59c-4d89-a5d2-3262c2b3a33c admin admin] RBAC: Adding query filter params (name=Default) {{(pid=92057) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} May 23 13:01:08.357826 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-963e562d-f59c-4d89-a5d2-3262c2b3a33c admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:01:08.359535 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-963e562d-f59c-4d89-a5d2-3262c2b3a33c admin admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:01:08.371973 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 1010/2013] 10.222.0.30 () {58 vars in 1152 bytes} [Mon May 23 13:01:08 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) May 23 13:01:08.381711 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-963e562d-f59c-4d89-a5d2-3262c2b3a33c admin admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:01:08.392859 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-963e562d-f59c-4d89-a5d2-3262c2b3a33c admin admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:01:08.397736 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6c63383-d5bf-4a53-887e-5cf92ce00654 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:01:08.403382 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6c63383-d5bf-4a53-887e-5cf92ce00654 admin admin] REQUEST_METHOD: `POST` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:01:08.403382 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6c63383-d5bf-4a53-887e-5cf92ce00654 admin admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:01:08.403382 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6c63383-d5bf-4a53-887e-5cf92ce00654 admin admin] PATH_INFO: `/v3/projects` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:01:08.403382 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6c63383-d5bf-4a53-887e-5cf92ce00654 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:01:08.403382 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6c63383-d5bf-4a53-887e-5cf92ce00654 admin admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:01:08.506292 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-bb41fc26-8994-439f-9ca2-e78540108565 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'cb8d79a531c84f92a60c08ffccbeda0f', 'request_id': 'req-bb41fc26-8994-439f-9ca2-e78540108565'} {{(pid=92056) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:01:08.527202 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 1011/2014] 10.222.0.30 () {60 vars in 1152 bytes} [Mon May 23 13:01:08 2022] POST /identity/v3/projects => generated 379 bytes in 142 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) May 23 13:01:08.528423 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6c63383-d5bf-4a53-887e-5cf92ce00654 admin admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:01:08.613375 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6c63383-d5bf-4a53-887e-5cf92ce00654 admin admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:01:08.618222 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08c4fc8b-9187-4dda-bb03-32fad65ca983 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:01:08.618898 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08c4fc8b-9187-4dda-bb03-32fad65ca983 admin admin] REQUEST_METHOD: `POST` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:01:08.619164 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08c4fc8b-9187-4dda-bb03-32fad65ca983 admin admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:01:08.619438 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08c4fc8b-9187-4dda-bb03-32fad65ca983 admin admin] PATH_INFO: `/v3/users` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:01:08.620804 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08c4fc8b-9187-4dda-bb03-32fad65ca983 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:01:08.622876 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08c4fc8b-9187-4dda-bb03-32fad65ca983 admin admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:01:08.656659 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-bb41fc26-8994-439f-9ca2-e78540108565 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'cb8d79a531c84f92a60c08ffccbeda0f', 'request_id': 'req-bb41fc26-8994-439f-9ca2-e78540108565'} {{(pid=92056) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:01:08.658439 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=86,keystone:DELETE=12,keystone:INSERT=2 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 13:01:08.666081 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-bb41fc26-8994-439f-9ca2-e78540108565 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'cb8d79a531c84f92a60c08ffccbeda0f', 'request_id': 'req-bb41fc26-8994-439f-9ca2-e78540108565'} {{(pid=92056) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:01:08.676829 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-bb41fc26-8994-439f-9ca2-e78540108565 admin admin] The token cache is being invalidate because project cb8d79a531c84f92a60c08ffccbeda0f was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=92056) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} May 23 13:01:08.677131 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-bb41fc26-8994-439f-9ca2-e78540108565 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=92056) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:01:08.677812 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-bb41fc26-8994-439f-9ca2-e78540108565 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=92056) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:01:08.682954 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 1004/2015] 10.222.0.30 () {58 vars in 1232 bytes} [Mon May 23 13:01:08 2022] DELETE /identity/v3/projects/cb8d79a531c84f92a60c08ffccbeda0f => generated 0 bytes in 429 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) May 23 13:01:08.683881 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb41fc26-8994-439f-9ca2-e78540108565 admin admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:01:08.685342 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.fernet_utils [None req-bb41fc26-8994-439f-9ca2-e78540108565 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92056) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:01:08.751955 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb41fc26-8994-439f-9ca2-e78540108565 admin admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:01:08.757710 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 1012/2016] 10.222.0.30 () {60 vars in 1143 bytes} [Mon May 23 13:01:08 2022] POST /identity/v3/users => generated 290 bytes in 230 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) May 23 13:01:08.758956 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08c4fc8b-9187-4dda-bb03-32fad65ca983 admin admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:01:08.761341 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.fernet_utils [None req-08c4fc8b-9187-4dda-bb03-32fad65ca983 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92057) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:01:08.763574 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ce1ee43-39cf-433d-9b2b-0ed58b394fb2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:01:08.764464 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ce1ee43-39cf-433d-9b2b-0ed58b394fb2 admin admin] REQUEST_METHOD: `DELETE` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:01:08.764758 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ce1ee43-39cf-433d-9b2b-0ed58b394fb2 admin admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:01:08.765064 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ce1ee43-39cf-433d-9b2b-0ed58b394fb2 admin admin] PATH_INFO: `/v3/users/19294839cf0a4fac8c748f76c95e6f38` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:01:08.767689 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ce1ee43-39cf-433d-9b2b-0ed58b394fb2 admin admin] RBAC: Authorizing `identity:delete_user(user_id=19294839cf0a4fac8c748f76c95e6f38)` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:01:08.769633 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ce1ee43-39cf-433d-9b2b-0ed58b394fb2 admin admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:01:08.774872 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08c4fc8b-9187-4dda-bb03-32fad65ca983 admin admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:01:08.781747 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d53c8f27-86c8-4c53-8d59-29c2572acad9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:01:08.782580 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d53c8f27-86c8-4c53-8d59-29c2572acad9 admin admin] REQUEST_METHOD: `DELETE` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:01:08.782870 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d53c8f27-86c8-4c53-8d59-29c2572acad9 admin admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:01:08.783152 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d53c8f27-86c8-4c53-8d59-29c2572acad9 admin admin] PATH_INFO: `/v3/projects/f622a4cb39a14cc789f66eef0a5bf0d1` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:01:08.784372 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d53c8f27-86c8-4c53-8d59-29c2572acad9 admin admin] RBAC: Authorizing `identity:delete_project(project_id=f622a4cb39a14cc789f66eef0a5bf0d1)` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:01:08.786378 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d53c8f27-86c8-4c53-8d59-29c2572acad9 admin admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:01:08.952818 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-4ce1ee43-39cf-433d-9b2b-0ed58b394fb2 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '19294839cf0a4fac8c748f76c95e6f38'} {{(pid=92056) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:01:08.953492 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-4ce1ee43-39cf-433d-9b2b-0ed58b394fb2 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '19294839cf0a4fac8c748f76c95e6f38'} {{(pid=92056) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:01:08.953655 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [None req-d53c8f27-86c8-4c53-8d59-29c2572acad9 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'f622a4cb39a14cc789f66eef0a5bf0d1', 'request_id': 'req-d53c8f27-86c8-4c53-8d59-29c2572acad9'} {{(pid=92057) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:01:08.959116 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [None req-d53c8f27-86c8-4c53-8d59-29c2572acad9 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'f622a4cb39a14cc789f66eef0a5bf0d1', 'request_id': 'req-d53c8f27-86c8-4c53-8d59-29c2572acad9'} {{(pid=92057) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:01:09.036022 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-4ce1ee43-39cf-433d-9b2b-0ed58b394fb2 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '19294839cf0a4fac8c748f76c95e6f38'} {{(pid=92056) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:01:09.048862 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-4ce1ee43-39cf-433d-9b2b-0ed58b394fb2 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '19294839cf0a4fac8c748f76c95e6f38'} {{(pid=92056) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:01:09.055730 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-4ce1ee43-39cf-433d-9b2b-0ed58b394fb2 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '19294839cf0a4fac8c748f76c95e6f38'} {{(pid=92056) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:01:09.062459 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 1005/2017] 10.222.0.30 () {58 vars in 1223 bytes} [Mon May 23 13:01:08 2022] DELETE /identity/v3/users/19294839cf0a4fac8c748f76c95e6f38 => generated 0 bytes in 379 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) May 23 13:01:09.065614 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ce1ee43-39cf-433d-9b2b-0ed58b394fb2 admin admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:01:09.065614 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.fernet_utils [None req-4ce1ee43-39cf-433d-9b2b-0ed58b394fb2 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92056) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:01:09.099192 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [None req-d53c8f27-86c8-4c53-8d59-29c2572acad9 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'f622a4cb39a14cc789f66eef0a5bf0d1', 'request_id': 'req-d53c8f27-86c8-4c53-8d59-29c2572acad9'} {{(pid=92057) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:01:09.115663 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [None req-d53c8f27-86c8-4c53-8d59-29c2572acad9 admin admin] The token cache is being invalidate because project f622a4cb39a14cc789f66eef0a5bf0d1 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=92057) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} May 23 13:01:09.116169 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [None req-d53c8f27-86c8-4c53-8d59-29c2572acad9 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=92057) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:01:09.116947 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [None req-d53c8f27-86c8-4c53-8d59-29c2572acad9 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=92057) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:01:09.118930 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 1013/2018] 10.222.0.30 () {58 vars in 1232 bytes} [Mon May 23 13:01:08 2022] DELETE /identity/v3/projects/f622a4cb39a14cc789f66eef0a5bf0d1 => generated 0 bytes in 360 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) May 23 13:01:09.126733 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1e2cd6a-32ca-45e6-9cca-d01f39a0827f None None] REQUEST_METHOD: `POST` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:01:09.127091 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1e2cd6a-32ca-45e6-9cca-d01f39a0827f None None] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:01:09.127628 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1e2cd6a-32ca-45e6-9cca-d01f39a0827f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:01:09.138666 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ce1ee43-39cf-433d-9b2b-0ed58b394fb2 admin admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:01:09.144498 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30b74f27-d304-4e66-ad6d-4502c5c7cd8c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:01:09.145297 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30b74f27-d304-4e66-ad6d-4502c5c7cd8c admin admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:01:09.145582 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30b74f27-d304-4e66-ad6d-4502c5c7cd8c admin admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:01:09.145934 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30b74f27-d304-4e66-ad6d-4502c5c7cd8c admin admin] PATH_INFO: `/v3/roles` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:01:09.146678 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-30b74f27-d304-4e66-ad6d-4502c5c7cd8c admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:01:09.148625 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-30b74f27-d304-4e66-ad6d-4502c5c7cd8c admin admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:01:09.151501 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.auth.core [None req-c1e2cd6a-32ca-45e6-9cca-d01f39a0827f None None] MFA Rules not processed for user `0f7119f8fa5c4445977671140ded91a4`. Rule list: `[]` (Enabled: `True`). {{(pid=92057) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 13:01:09.152946 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.fernet_utils [None req-c1e2cd6a-32ca-45e6-9cca-d01f39a0827f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92057) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:01:09.165930 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 1014/2019] 10.222.0.30 () {58 vars in 957 bytes} [Mon May 23 13:01:09 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 38 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) May 23 13:01:09.166061 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 1006/2020] 10.222.0.30 () {58 vars in 1121 bytes} [Mon May 23 13:01:09 2022] GET /identity/v3/roles => generated 1420 bytes in 101 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) May 23 13:01:09.166487 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1e2cd6a-32ca-45e6-9cca-d01f39a0827f None None] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:01:09.166487 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.fernet_utils [None req-c1e2cd6a-32ca-45e6-9cca-d01f39a0827f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92057) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:01:09.171669 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83957121-e70c-4835-8425-3524544cd373 None None] REQUEST_METHOD: `POST` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:01:09.171966 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83957121-e70c-4835-8425-3524544cd373 None None] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:01:09.172417 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83957121-e70c-4835-8425-3524544cd373 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:01:09.180308 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1e2cd6a-32ca-45e6-9cca-d01f39a0827f None None] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:01:09.186557 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e11eae0-7b8c-4bf2-8d95-c32611546350 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:01:09.187262 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e11eae0-7b8c-4bf2-8d95-c32611546350 admin admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:01:09.187547 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e11eae0-7b8c-4bf2-8d95-c32611546350 admin admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:01:09.187828 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e11eae0-7b8c-4bf2-8d95-c32611546350 admin admin] PATH_INFO: `/v3/roles` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:01:09.188583 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e11eae0-7b8c-4bf2-8d95-c32611546350 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:01:09.196812 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e11eae0-7b8c-4bf2-8d95-c32611546350 admin admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:01:09.196939 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.auth.core [None req-83957121-e70c-4835-8425-3524544cd373 None None] MFA Rules not processed for user `0f7119f8fa5c4445977671140ded91a4`. Rule list: `[]` (Enabled: `True`). {{(pid=92056) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 13:01:09.196939 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.fernet_utils [None req-83957121-e70c-4835-8425-3524544cd373 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92056) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:01:09.200739 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 1015/2021] 10.222.0.30 () {58 vars in 1121 bytes} [Mon May 23 13:01:09 2022] GET /identity/v3/roles => generated 1420 bytes in 36 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) May 23 13:01:09.203869 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 1007/2022] 10.222.0.30 () {58 vars in 957 bytes} [Mon May 23 13:01:09 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 33 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) May 23 13:01:09.204907 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e11eae0-7b8c-4bf2-8d95-c32611546350 admin admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:01:09.213405 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83957121-e70c-4835-8425-3524544cd373 None None] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:01:09.215051 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e11eae0-7b8c-4bf2-8d95-c32611546350 admin admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:01:09.217567 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83957121-e70c-4835-8425-3524544cd373 None None] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:01:09.219765 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97e64f9d-3593-44fa-b3a2-0c6268042447 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:01:09.220648 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97e64f9d-3593-44fa-b3a2-0c6268042447 admin admin] REQUEST_METHOD: `PUT` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:01:09.220936 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97e64f9d-3593-44fa-b3a2-0c6268042447 admin admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:01:09.221214 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97e64f9d-3593-44fa-b3a2-0c6268042447 admin admin] PATH_INFO: `/v3/projects/c7ef8e9bc68d4e60b8993eac9f160a65/users/c6a6096760734ac49ce2f26c6065b701/roles/80fb22507c9a4ff09d44d8f12f6fff1a` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:01:09.222588 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-628bcf3a-b7b5-4972-a48c-11cebcd84977 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:01:09.225305 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-628bcf3a-b7b5-4972-a48c-11cebcd84977 admin admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:01:09.225305 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-628bcf3a-b7b5-4972-a48c-11cebcd84977 admin admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:01:09.225305 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-628bcf3a-b7b5-4972-a48c-11cebcd84977 admin admin] PATH_INFO: `/v3/domains` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:01:09.225305 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-628bcf3a-b7b5-4972-a48c-11cebcd84977 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=92056) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} May 23 13:01:09.225305 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-628bcf3a-b7b5-4972-a48c-11cebcd84977 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:01:09.226429 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-628bcf3a-b7b5-4972-a48c-11cebcd84977 admin admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:01:09.238268 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97e64f9d-3593-44fa-b3a2-0c6268042447 admin admin] RBAC: Authorizing `identity:create_grant(project_id=c7ef8e9bc68d4e60b8993eac9f160a65, user_id=c6a6096760734ac49ce2f26c6065b701, role_id=80fb22507c9a4ff09d44d8f12f6fff1a)` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:01:09.238806 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 1008/2023] 10.222.0.30 () {58 vars in 1152 bytes} [Mon May 23 13:01:09 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) May 23 13:01:09.241062 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97e64f9d-3593-44fa-b3a2-0c6268042447 admin admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:01:09.243400 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-628bcf3a-b7b5-4972-a48c-11cebcd84977 admin admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:01:09.253998 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-628bcf3a-b7b5-4972-a48c-11cebcd84977 admin admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:01:09.260535 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52756065-b65d-43ed-a184-fb533aee1354 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:01:09.260535 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52756065-b65d-43ed-a184-fb533aee1354 admin admin] REQUEST_METHOD: `POST` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:01:09.260952 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52756065-b65d-43ed-a184-fb533aee1354 admin admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:01:09.261456 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52756065-b65d-43ed-a184-fb533aee1354 admin admin] PATH_INFO: `/v3/projects` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:01:09.262419 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52756065-b65d-43ed-a184-fb533aee1354 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:01:09.264620 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52756065-b65d-43ed-a184-fb533aee1354 admin admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:01:09.316570 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 1016/2024] 10.222.0.30 () {60 vars in 1482 bytes} [Mon May 23 13:01:09 2022] PUT /identity/v3/projects/c7ef8e9bc68d4e60b8993eac9f160a65/users/c6a6096760734ac49ce2f26c6065b701/roles/80fb22507c9a4ff09d44d8f12f6fff1a => generated 0 bytes in 112 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) May 23 13:01:09.323369 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49443422-9ab1-403f-89c5-7243f6a84ea9 None None] REQUEST_METHOD: `POST` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:01:09.323659 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49443422-9ab1-403f-89c5-7243f6a84ea9 None None] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:01:09.324064 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49443422-9ab1-403f-89c5-7243f6a84ea9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:01:09.346020 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.auth.core [None req-49443422-9ab1-403f-89c5-7243f6a84ea9 None None] MFA Rules not processed for user `0f7119f8fa5c4445977671140ded91a4`. Rule list: `[]` (Enabled: `True`). {{(pid=92057) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 13:01:09.346020 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.fernet_utils [None req-49443422-9ab1-403f-89c5-7243f6a84ea9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92057) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:01:09.376963 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 1009/2025] 10.222.0.30 () {60 vars in 1152 bytes} [Mon May 23 13:01:09 2022] POST /identity/v3/projects => generated 379 bytes in 131 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) May 23 13:01:09.378031 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52756065-b65d-43ed-a184-fb533aee1354 admin admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:01:09.412099 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 1017/2026] 10.222.0.30 () {58 vars in 957 bytes} [Mon May 23 13:01:09 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 89 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) May 23 13:01:09.429234 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49443422-9ab1-403f-89c5-7243f6a84ea9 None None] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:01:09.430698 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.fernet_utils [None req-49443422-9ab1-403f-89c5-7243f6a84ea9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92057) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:01:09.443267 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=90,keystone:INSERT=4,keystone:DELETE=6 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 13:01:09.446344 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52756065-b65d-43ed-a184-fb533aee1354 admin admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:01:09.451525 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a376fbcd-297e-4e52-adab-85d97c294ca9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:01:09.452220 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a376fbcd-297e-4e52-adab-85d97c294ca9 admin admin] REQUEST_METHOD: `POST` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:01:09.452501 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a376fbcd-297e-4e52-adab-85d97c294ca9 admin admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:01:09.452775 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a376fbcd-297e-4e52-adab-85d97c294ca9 admin admin] PATH_INFO: `/v3/users` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:01:09.453467 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a376fbcd-297e-4e52-adab-85d97c294ca9 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:01:09.455415 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a376fbcd-297e-4e52-adab-85d97c294ca9 admin admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:01:09.490478 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49443422-9ab1-403f-89c5-7243f6a84ea9 None None] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:01:09.538763 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 1010/2027] 10.222.0.30 () {60 vars in 1143 bytes} [Mon May 23 13:01:09 2022] POST /identity/v3/users => generated 290 bytes in 161 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) May 23 13:01:09.539690 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a376fbcd-297e-4e52-adab-85d97c294ca9 admin admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:01:09.542777 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.fernet_utils [None req-a376fbcd-297e-4e52-adab-85d97c294ca9 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92056) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:01:09.545757 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7dab5b36-9418-4cc3-8a70-27ad85d85990 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2eb83ca38e79449095f3a2f947bbb06a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9c389ae65aaa4542bb5a5c4fd0bacf5b', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:01:09.546513 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7dab5b36-9418-4cc3-8a70-27ad85d85990 service neutron] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:01:09.546794 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7dab5b36-9418-4cc3-8a70-27ad85d85990 service neutron] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:01:09.547086 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7dab5b36-9418-4cc3-8a70-27ad85d85990 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:01:09.555065 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a376fbcd-297e-4e52-adab-85d97c294ca9 admin admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:01:09.556500 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7dab5b36-9418-4cc3-8a70-27ad85d85990 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:01:09.558172 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7dab5b36-9418-4cc3-8a70-27ad85d85990 service neutron] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:01:09.559981 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0087d1c-5f0b-438c-8764-35112c4610bb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:01:09.560865 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0087d1c-5f0b-438c-8764-35112c4610bb admin admin] REQUEST_METHOD: `DELETE` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:01:09.561146 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0087d1c-5f0b-438c-8764-35112c4610bb admin admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:01:09.561422 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0087d1c-5f0b-438c-8764-35112c4610bb admin admin] PATH_INFO: `/v3/projects/1ec206af9f5b42d1bf65059555fe306d` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:01:09.562603 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0087d1c-5f0b-438c-8764-35112c4610bb admin admin] RBAC: Authorizing `identity:delete_project(project_id=1ec206af9f5b42d1bf65059555fe306d)` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:01:09.564619 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0087d1c-5f0b-438c-8764-35112c4610bb admin admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:01:09.567644 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 1018/2028] 10.222.0.30 () {60 vars in 1365 bytes} [Mon May 23 13:01:09 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 139 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) May 23 13:01:09.568333 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7dab5b36-9418-4cc3-8a70-27ad85d85990 service neutron] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:01:09.577443 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7dab5b36-9418-4cc3-8a70-27ad85d85990 service neutron] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:01:09.582196 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29cc1198-9d28-484a-a87b-35820681f80a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:01:09.582866 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29cc1198-9d28-484a-a87b-35820681f80a admin admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:01:09.583135 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29cc1198-9d28-484a-a87b-35820681f80a admin admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:01:09.583410 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29cc1198-9d28-484a-a87b-35820681f80a admin admin] PATH_INFO: `/v3/roles` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:01:09.584127 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-29cc1198-9d28-484a-a87b-35820681f80a admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:01:09.585723 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-29cc1198-9d28-484a-a87b-35820681f80a admin admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:01:09.597716 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 1019/2029] 10.222.0.30 () {58 vars in 1121 bytes} [Mon May 23 13:01:09 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) May 23 13:01:09.601687 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29cc1198-9d28-484a-a87b-35820681f80a admin admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:01:09.612044 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29cc1198-9d28-484a-a87b-35820681f80a admin admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:01:09.616794 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef46fc96-4b41-4182-afe6-4d3734aea7d3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:01:09.617495 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef46fc96-4b41-4182-afe6-4d3734aea7d3 admin admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:01:09.617741 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef46fc96-4b41-4182-afe6-4d3734aea7d3 admin admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:01:09.618067 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef46fc96-4b41-4182-afe6-4d3734aea7d3 admin admin] PATH_INFO: `/v3/roles` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:01:09.618787 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef46fc96-4b41-4182-afe6-4d3734aea7d3 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:01:09.620914 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef46fc96-4b41-4182-afe6-4d3734aea7d3 admin admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:01:09.631029 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 1020/2030] 10.222.0.30 () {58 vars in 1121 bytes} [Mon May 23 13:01:09 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) May 23 13:01:09.635429 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef46fc96-4b41-4182-afe6-4d3734aea7d3 admin admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:01:09.658144 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef46fc96-4b41-4182-afe6-4d3734aea7d3 admin admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:01:09.658144 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b01d623-8757-4737-91ad-8ff85d9c1152 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:01:09.658144 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b01d623-8757-4737-91ad-8ff85d9c1152 admin admin] REQUEST_METHOD: `PUT` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:01:09.658144 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b01d623-8757-4737-91ad-8ff85d9c1152 admin admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:01:09.658530 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b01d623-8757-4737-91ad-8ff85d9c1152 admin admin] PATH_INFO: `/v3/projects/e4a9af8184a1434797a4d055ad1d5fdb/users/db7bf754ea924d889a32cdddba47ac6e/roles/80fb22507c9a4ff09d44d8f12f6fff1a` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:01:09.669171 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b01d623-8757-4737-91ad-8ff85d9c1152 admin admin] RBAC: Authorizing `identity:create_grant(project_id=e4a9af8184a1434797a4d055ad1d5fdb, user_id=db7bf754ea924d889a32cdddba47ac6e, role_id=80fb22507c9a4ff09d44d8f12f6fff1a)` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:01:09.671985 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b01d623-8757-4737-91ad-8ff85d9c1152 admin admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:01:09.763342 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 1021/2031] 10.222.0.30 () {60 vars in 1482 bytes} [Mon May 23 13:01:09 2022] PUT /identity/v3/projects/e4a9af8184a1434797a4d055ad1d5fdb/users/db7bf754ea924d889a32cdddba47ac6e/roles/80fb22507c9a4ff09d44d8f12f6fff1a => generated 0 bytes in 128 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) May 23 13:01:09.775126 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49077e35-3a06-444a-b054-3e69f18e49c5 None None] REQUEST_METHOD: `POST` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:01:09.775442 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49077e35-3a06-444a-b054-3e69f18e49c5 None None] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:01:09.775704 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49077e35-3a06-444a-b054-3e69f18e49c5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:01:09.780609 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-c0087d1c-5f0b-438c-8764-35112c4610bb admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '1ec206af9f5b42d1bf65059555fe306d', 'request_id': 'req-c0087d1c-5f0b-438c-8764-35112c4610bb'} {{(pid=92056) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:01:09.786924 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=87,keystone:DELETE=12,keystone:INSERT=1 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 13:01:09.804595 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.auth.core [None req-49077e35-3a06-444a-b054-3e69f18e49c5 None None] MFA Rules not processed for user `db7bf754ea924d889a32cdddba47ac6e`. Rule list: `[]` (Enabled: `True`). {{(pid=92057) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 13:01:09.805730 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.fernet_utils [None req-49077e35-3a06-444a-b054-3e69f18e49c5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92057) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:01:09.876009 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 1022/2032] 10.222.0.30 () {58 vars in 957 bytes} [Mon May 23 13:01:09 2022] POST /identity/v3/auth/tokens => generated 3246 bytes in 102 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) May 23 13:01:09.880043 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49077e35-3a06-444a-b054-3e69f18e49c5 None None] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:01:09.888394 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-c0087d1c-5f0b-438c-8764-35112c4610bb admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '1ec206af9f5b42d1bf65059555fe306d', 'request_id': 'req-c0087d1c-5f0b-438c-8764-35112c4610bb'} {{(pid=92056) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:01:09.907912 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-c0087d1c-5f0b-438c-8764-35112c4610bb admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '1ec206af9f5b42d1bf65059555fe306d', 'request_id': 'req-c0087d1c-5f0b-438c-8764-35112c4610bb'} {{(pid=92056) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:01:09.910999 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-c0087d1c-5f0b-438c-8764-35112c4610bb admin admin] The token cache is being invalidate because project 1ec206af9f5b42d1bf65059555fe306d was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=92056) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} May 23 13:01:09.911289 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-c0087d1c-5f0b-438c-8764-35112c4610bb admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=92056) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:01:09.911985 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-c0087d1c-5f0b-438c-8764-35112c4610bb admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=92056) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:01:09.913608 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 1011/2033] 10.222.0.30 () {58 vars in 1232 bytes} [Mon May 23 13:01:09 2022] DELETE /identity/v3/projects/1ec206af9f5b42d1bf65059555fe306d => generated 0 bytes in 374 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) May 23 13:01:09.918414 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-243bb178-6578-47c2-91e6-dd1d8ccb6cfb None None] REQUEST_METHOD: `POST` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:01:09.918702 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-243bb178-6578-47c2-91e6-dd1d8ccb6cfb None None] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:01:09.918984 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-243bb178-6578-47c2-91e6-dd1d8ccb6cfb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:01:09.936900 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.auth.core [None req-243bb178-6578-47c2-91e6-dd1d8ccb6cfb None None] MFA Rules not processed for user `0f7119f8fa5c4445977671140ded91a4`. Rule list: `[]` (Enabled: `True`). {{(pid=92056) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 13:01:09.938279 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.fernet_utils [None req-243bb178-6578-47c2-91e6-dd1d8ccb6cfb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92056) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:01:09.956371 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49077e35-3a06-444a-b054-3e69f18e49c5 None None] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:01:09.961431 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c564d0b2-bc7b-4d97-9379-ca1c3ba0f085 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:01:09.962389 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c564d0b2-bc7b-4d97-9379-ca1c3ba0f085 admin admin] REQUEST_METHOD: `POST` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:01:09.962389 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c564d0b2-bc7b-4d97-9379-ca1c3ba0f085 admin admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:01:09.963006 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c564d0b2-bc7b-4d97-9379-ca1c3ba0f085 admin admin] PATH_INFO: `/v3/projects` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:01:09.963373 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c564d0b2-bc7b-4d97-9379-ca1c3ba0f085 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:01:09.966152 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c564d0b2-bc7b-4d97-9379-ca1c3ba0f085 admin admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:01:10.002386 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 1012/2034] 10.222.0.30 () {58 vars in 957 bytes} [Mon May 23 13:01:09 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) May 23 13:01:10.010006 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71b50002-e9d3-484d-8364-c070afc988a8 None None] REQUEST_METHOD: `POST` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:01:10.010298 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71b50002-e9d3-484d-8364-c070afc988a8 None None] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:01:10.010615 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71b50002-e9d3-484d-8364-c070afc988a8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:01:10.029801 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.auth.core [None req-71b50002-e9d3-484d-8364-c070afc988a8 None None] MFA Rules not processed for user `0f7119f8fa5c4445977671140ded91a4`. Rule list: `[]` (Enabled: `True`). {{(pid=92056) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 13:01:10.031025 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.fernet_utils [None req-71b50002-e9d3-484d-8364-c070afc988a8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92056) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:01:10.038861 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 1013/2035] 10.222.0.30 () {58 vars in 957 bytes} [Mon May 23 13:01:10 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 30 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) May 23 13:01:10.043435 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71b50002-e9d3-484d-8364-c070afc988a8 None None] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:01:10.052171 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71b50002-e9d3-484d-8364-c070afc988a8 None None] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:01:10.056795 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f30bf209-a653-41b3-8168-8b94a1367e1d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:01:10.058124 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f30bf209-a653-41b3-8168-8b94a1367e1d admin admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:01:10.058601 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f30bf209-a653-41b3-8168-8b94a1367e1d admin admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:01:10.058882 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f30bf209-a653-41b3-8168-8b94a1367e1d admin admin] PATH_INFO: `/v3/domains` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:01:10.059603 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f30bf209-a653-41b3-8168-8b94a1367e1d admin admin] RBAC: Adding query filter params (name=Default) {{(pid=92056) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} May 23 13:01:10.059893 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f30bf209-a653-41b3-8168-8b94a1367e1d admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:01:10.061491 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f30bf209-a653-41b3-8168-8b94a1367e1d admin admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:01:10.073650 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 1014/2036] 10.222.0.30 () {58 vars in 1152 bytes} [Mon May 23 13:01:10 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) May 23 13:01:10.077835 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f30bf209-a653-41b3-8168-8b94a1367e1d admin admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:01:10.087521 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f30bf209-a653-41b3-8168-8b94a1367e1d admin admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:01:10.093940 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62e7e7f0-9788-41d1-a403-c0f42c29d00f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:01:10.094604 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62e7e7f0-9788-41d1-a403-c0f42c29d00f admin admin] REQUEST_METHOD: `POST` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:01:10.094892 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62e7e7f0-9788-41d1-a403-c0f42c29d00f admin admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:01:10.095174 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62e7e7f0-9788-41d1-a403-c0f42c29d00f admin admin] PATH_INFO: `/v3/projects` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:01:10.096195 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-62e7e7f0-9788-41d1-a403-c0f42c29d00f admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:01:10.098133 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-62e7e7f0-9788-41d1-a403-c0f42c29d00f admin admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:01:10.102398 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 1023/2037] 10.222.0.30 () {60 vars in 1152 bytes} [Mon May 23 13:01:09 2022] POST /identity/v3/projects => generated 381 bytes in 223 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) May 23 13:01:10.107102 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c564d0b2-bc7b-4d97-9379-ca1c3ba0f085 admin admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:01:10.109026 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.fernet_utils [None req-c564d0b2-bc7b-4d97-9379-ca1c3ba0f085 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92057) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:01:10.176475 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c564d0b2-bc7b-4d97-9379-ca1c3ba0f085 admin admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:01:10.181348 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d60aaf7d-cdb9-499f-9472-11a8e0dc9dad admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:01:10.182134 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d60aaf7d-cdb9-499f-9472-11a8e0dc9dad admin admin] REQUEST_METHOD: `POST` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:01:10.182466 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d60aaf7d-cdb9-499f-9472-11a8e0dc9dad admin admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:01:10.182775 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d60aaf7d-cdb9-499f-9472-11a8e0dc9dad admin admin] PATH_INFO: `/v3/users` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:01:10.183497 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d60aaf7d-cdb9-499f-9472-11a8e0dc9dad admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:01:10.185464 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d60aaf7d-cdb9-499f-9472-11a8e0dc9dad admin admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:01:10.205929 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 1015/2038] 10.222.0.30 () {60 vars in 1152 bytes} [Mon May 23 13:01:10 2022] POST /identity/v3/projects => generated 383 bytes in 127 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) May 23 13:01:10.208983 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62e7e7f0-9788-41d1-a403-c0f42c29d00f admin admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:01:10.274679 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62e7e7f0-9788-41d1-a403-c0f42c29d00f admin admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:01:10.280493 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b91467d-8fcc-45c9-9241-af741a81591a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:01:10.281159 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b91467d-8fcc-45c9-9241-af741a81591a admin admin] REQUEST_METHOD: `POST` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:01:10.281443 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b91467d-8fcc-45c9-9241-af741a81591a admin admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:01:10.281721 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b91467d-8fcc-45c9-9241-af741a81591a admin admin] PATH_INFO: `/v3/users` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:01:10.282448 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b91467d-8fcc-45c9-9241-af741a81591a admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:01:10.284425 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b91467d-8fcc-45c9-9241-af741a81591a admin admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:01:10.326149 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 1024/2039] 10.222.0.30 () {60 vars in 1143 bytes} [Mon May 23 13:01:10 2022] POST /identity/v3/users => generated 297 bytes in 220 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) May 23 13:01:10.330257 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d60aaf7d-cdb9-499f-9472-11a8e0dc9dad admin admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:01:10.341048 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d60aaf7d-cdb9-499f-9472-11a8e0dc9dad admin admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:01:10.345684 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-049c42de-3116-4653-8145-eeac60091a71 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:01:10.346462 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-049c42de-3116-4653-8145-eeac60091a71 admin admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:01:10.346809 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-049c42de-3116-4653-8145-eeac60091a71 admin admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:01:10.347179 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-049c42de-3116-4653-8145-eeac60091a71 admin admin] PATH_INFO: `/v3/roles` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:01:10.348050 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-049c42de-3116-4653-8145-eeac60091a71 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:01:10.349697 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-049c42de-3116-4653-8145-eeac60091a71 admin admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:01:10.360667 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 1025/2040] 10.222.0.30 () {58 vars in 1121 bytes} [Mon May 23 13:01:10 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) May 23 13:01:10.364830 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-049c42de-3116-4653-8145-eeac60091a71 admin admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:01:10.375172 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-049c42de-3116-4653-8145-eeac60091a71 admin admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:01:10.379893 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57e0785e-46cc-4ed3-a8ea-f0b10de7b53b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:01:10.380772 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57e0785e-46cc-4ed3-a8ea-f0b10de7b53b admin admin] REQUEST_METHOD: `PUT` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:01:10.381058 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57e0785e-46cc-4ed3-a8ea-f0b10de7b53b admin admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:01:10.381338 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57e0785e-46cc-4ed3-a8ea-f0b10de7b53b admin admin] PATH_INFO: `/v3/projects/24e100fc831d4e53919f466bccb8929e/users/b12ae34252d044e1890682574bf18cbb/roles/bc3c3c9d654940cbaa5177a6be23d1bc` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:01:10.397823 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57e0785e-46cc-4ed3-a8ea-f0b10de7b53b admin admin] RBAC: Authorizing `identity:create_grant(project_id=24e100fc831d4e53919f466bccb8929e, user_id=b12ae34252d044e1890682574bf18cbb, role_id=bc3c3c9d654940cbaa5177a6be23d1bc)` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:01:10.400584 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57e0785e-46cc-4ed3-a8ea-f0b10de7b53b admin admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:01:10.407441 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 1016/2041] 10.222.0.30 () {60 vars in 1143 bytes} [Mon May 23 13:01:10 2022] POST /identity/v3/users => generated 292 bytes in 199 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) May 23 13:01:10.411396 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b91467d-8fcc-45c9-9241-af741a81591a admin admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:01:10.423343 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b91467d-8fcc-45c9-9241-af741a81591a admin admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:01:10.428133 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60255759-8a1c-4f00-8959-759606c39cf0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:01:10.428868 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60255759-8a1c-4f00-8959-759606c39cf0 admin admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:01:10.429154 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60255759-8a1c-4f00-8959-759606c39cf0 admin admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:01:10.429428 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60255759-8a1c-4f00-8959-759606c39cf0 admin admin] PATH_INFO: `/v3/roles` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:01:10.430208 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60255759-8a1c-4f00-8959-759606c39cf0 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:01:10.431812 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60255759-8a1c-4f00-8959-759606c39cf0 admin admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:01:10.445159 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 1017/2042] 10.222.0.30 () {58 vars in 1121 bytes} [Mon May 23 13:01:10 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) May 23 13:01:10.446281 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60255759-8a1c-4f00-8959-759606c39cf0 admin admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:01:10.457853 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60255759-8a1c-4f00-8959-759606c39cf0 admin admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:01:10.462867 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ce28757-ccdd-4e16-988a-acd752a1e63c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:01:10.463530 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ce28757-ccdd-4e16-988a-acd752a1e63c admin admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:01:10.464476 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ce28757-ccdd-4e16-988a-acd752a1e63c admin admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:01:10.464819 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ce28757-ccdd-4e16-988a-acd752a1e63c admin admin] PATH_INFO: `/v3/roles` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:01:10.465550 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ce28757-ccdd-4e16-988a-acd752a1e63c admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:01:10.468741 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ce28757-ccdd-4e16-988a-acd752a1e63c admin admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:01:10.480462 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 1018/2043] 10.222.0.30 () {58 vars in 1121 bytes} [Mon May 23 13:01:10 2022] GET /identity/v3/roles => generated 1420 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) May 23 13:01:10.484328 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ce28757-ccdd-4e16-988a-acd752a1e63c admin admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:01:10.493719 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ce28757-ccdd-4e16-988a-acd752a1e63c admin admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:01:10.498960 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6f14d5e-76ac-460e-8333-0ac0e12688d1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:01:10.499838 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6f14d5e-76ac-460e-8333-0ac0e12688d1 admin admin] REQUEST_METHOD: `PUT` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:01:10.500121 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6f14d5e-76ac-460e-8333-0ac0e12688d1 admin admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:01:10.500394 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6f14d5e-76ac-460e-8333-0ac0e12688d1 admin admin] PATH_INFO: `/v3/projects/95675a44f30b42ef94a8afb1b9167f8c/users/0f675d0e2c8c42b4883586446a4e244e/roles/80fb22507c9a4ff09d44d8f12f6fff1a` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:01:10.505216 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 1026/2044] 10.222.0.30 () {60 vars in 1482 bytes} [Mon May 23 13:01:10 2022] PUT /identity/v3/projects/24e100fc831d4e53919f466bccb8929e/users/b12ae34252d044e1890682574bf18cbb/roles/bc3c3c9d654940cbaa5177a6be23d1bc => generated 0 bytes in 141 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) May 23 13:01:10.509947 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57e0785e-46cc-4ed3-a8ea-f0b10de7b53b admin admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:01:10.515921 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6f14d5e-76ac-460e-8333-0ac0e12688d1 admin admin] RBAC: Authorizing `identity:create_grant(project_id=95675a44f30b42ef94a8afb1b9167f8c, user_id=0f675d0e2c8c42b4883586446a4e244e, role_id=80fb22507c9a4ff09d44d8f12f6fff1a)` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:01:10.518660 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6f14d5e-76ac-460e-8333-0ac0e12688d1 admin admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:01:10.569067 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=97,keystone:INSERT=2,keystone:DELETE=1 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 13:01:10.579777 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57e0785e-46cc-4ed3-a8ea-f0b10de7b53b admin admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:01:10.584374 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae3f5a30-e6db-4ac3-8e55-53f46805b755 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:01:10.585032 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae3f5a30-e6db-4ac3-8e55-53f46805b755 admin admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:01:10.585313 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae3f5a30-e6db-4ac3-8e55-53f46805b755 admin admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:01:10.585585 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae3f5a30-e6db-4ac3-8e55-53f46805b755 admin admin] PATH_INFO: `/v3/roles` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:01:10.586387 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae3f5a30-e6db-4ac3-8e55-53f46805b755 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:01:10.587997 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae3f5a30-e6db-4ac3-8e55-53f46805b755 admin admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:01:10.599176 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 1027/2045] 10.222.0.30 () {58 vars in 1121 bytes} [Mon May 23 13:01:10 2022] GET /identity/v3/roles => generated 1420 bytes in 89 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) May 23 13:01:10.603215 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae3f5a30-e6db-4ac3-8e55-53f46805b755 admin admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:01:10.606271 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 1019/2046] 10.222.0.30 () {60 vars in 1482 bytes} [Mon May 23 13:01:10 2022] PUT /identity/v3/projects/95675a44f30b42ef94a8afb1b9167f8c/users/0f675d0e2c8c42b4883586446a4e244e/roles/80fb22507c9a4ff09d44d8f12f6fff1a => generated 0 bytes in 122 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) May 23 13:01:10.614515 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae3f5a30-e6db-4ac3-8e55-53f46805b755 admin admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:01:10.624188 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00293150-0efb-438c-86eb-960b7f9da0ea None None] REQUEST_METHOD: `POST` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:01:10.624893 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00293150-0efb-438c-86eb-960b7f9da0ea None None] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:01:10.624893 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00293150-0efb-438c-86eb-960b7f9da0ea None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:01:10.654108 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.auth.core [None req-00293150-0efb-438c-86eb-960b7f9da0ea None None] MFA Rules not processed for user `0f675d0e2c8c42b4883586446a4e244e`. Rule list: `[]` (Enabled: `True`). {{(pid=92056) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 13:01:10.655321 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.fernet_utils [None req-00293150-0efb-438c-86eb-960b7f9da0ea None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92056) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:01:10.687258 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39572765-3641-43cc-a515-627678118624 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:01:10.688154 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39572765-3641-43cc-a515-627678118624 admin admin] REQUEST_METHOD: `PUT` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:01:10.688433 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39572765-3641-43cc-a515-627678118624 admin admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:01:10.693432 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39572765-3641-43cc-a515-627678118624 admin admin] PATH_INFO: `/v3/projects/24e100fc831d4e53919f466bccb8929e/users/b12ae34252d044e1890682574bf18cbb/roles/bc3c3c9d654940cbaa5177a6be23d1bc` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:01:10.693432 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39572765-3641-43cc-a515-627678118624 admin admin] RBAC: Authorizing `identity:create_grant(project_id=24e100fc831d4e53919f466bccb8929e, user_id=b12ae34252d044e1890682574bf18cbb, role_id=bc3c3c9d654940cbaa5177a6be23d1bc)` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:01:10.694397 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39572765-3641-43cc-a515-627678118624 admin admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:01:10.708347 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 1028/2047] 10.222.0.30 () {60 vars in 1482 bytes} [Mon May 23 13:01:10 2022] PUT /identity/v3/projects/24e100fc831d4e53919f466bccb8929e/users/b12ae34252d044e1890682574bf18cbb/roles/bc3c3c9d654940cbaa5177a6be23d1bc => generated 0 bytes in 105 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) May 23 13:01:10.719771 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7f24662-b893-4924-ac74-3ea2962a951a None None] REQUEST_METHOD: `POST` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:01:10.720068 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7f24662-b893-4924-ac74-3ea2962a951a None None] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:01:10.720352 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7f24662-b893-4924-ac74-3ea2962a951a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:01:10.749252 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 1020/2048] 10.222.0.30 () {58 vars in 957 bytes} [Mon May 23 13:01:10 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 126 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) May 23 13:01:10.753916 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00293150-0efb-438c-86eb-960b7f9da0ea None None] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:01:10.761948 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.auth.core [None req-c7f24662-b893-4924-ac74-3ea2962a951a None None] MFA Rules not processed for user `b12ae34252d044e1890682574bf18cbb`. Rule list: `[]` (Enabled: `True`). {{(pid=92057) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 13:01:10.763430 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.fernet_utils [None req-c7f24662-b893-4924-ac74-3ea2962a951a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92057) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:01:10.765304 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00293150-0efb-438c-86eb-960b7f9da0ea None None] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:01:10.827070 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-451557b0-da5f-48c4-9e18-01d8c351f894 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:01:10.827662 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-451557b0-da5f-48c4-9e18-01d8c351f894 admin admin] REQUEST_METHOD: `POST` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:01:10.827948 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-451557b0-da5f-48c4-9e18-01d8c351f894 admin admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:01:10.828226 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-451557b0-da5f-48c4-9e18-01d8c351f894 admin admin] PATH_INFO: `/v3/projects` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:01:10.828913 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-451557b0-da5f-48c4-9e18-01d8c351f894 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:01:10.830879 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-451557b0-da5f-48c4-9e18-01d8c351f894 admin admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:01:10.833399 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 1029/2049] 10.222.0.30 () {58 vars in 957 bytes} [Mon May 23 13:01:10 2022] POST /identity/v3/auth/tokens => generated 3315 bytes in 115 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) May 23 13:01:10.845303 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7f24662-b893-4924-ac74-3ea2962a951a None None] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:01:10.846991 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.fernet_utils [None req-c7f24662-b893-4924-ac74-3ea2962a951a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92057) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:01:10.910831 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7f24662-b893-4924-ac74-3ea2962a951a None None] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:01:10.914643 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f362907-6d1b-490d-9c31-8b2d0a492ed5 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1666097f8ed545c6b0e4809c93e4775a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9c389ae65aaa4542bb5a5c4fd0bacf5b', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:01:10.918591 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f362907-6d1b-490d-9c31-8b2d0a492ed5 service cinder] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:01:10.922662 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f362907-6d1b-490d-9c31-8b2d0a492ed5 service cinder] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:01:10.922973 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f362907-6d1b-490d-9c31-8b2d0a492ed5 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:01:10.926780 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 1021/2050] 10.222.0.30 () {60 vars in 1152 bytes} [Mon May 23 13:01:10 2022] POST /identity/v3/projects => generated 385 bytes in 173 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) May 23 13:01:10.930277 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-451557b0-da5f-48c4-9e18-01d8c351f894 admin admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:01:10.939037 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f362907-6d1b-490d-9c31-8b2d0a492ed5 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:01:10.940501 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f362907-6d1b-490d-9c31-8b2d0a492ed5 service cinder] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:01:10.951935 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 1030/2051] 10.222.0.30 () {60 vars in 1365 bytes} [Mon May 23 13:01:10 2022] GET /identity/v3/auth/tokens => generated 3315 bytes in 107 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) May 23 13:01:11.041247 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-451557b0-da5f-48c4-9e18-01d8c351f894 admin admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:01:11.047500 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de167561-8dbf-40c1-b57b-5b5a783a21b0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:01:11.048042 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de167561-8dbf-40c1-b57b-5b5a783a21b0 admin admin] REQUEST_METHOD: `POST` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:01:11.048388 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de167561-8dbf-40c1-b57b-5b5a783a21b0 admin admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:01:11.048756 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de167561-8dbf-40c1-b57b-5b5a783a21b0 admin admin] PATH_INFO: `/v3/users` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:01:11.049562 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de167561-8dbf-40c1-b57b-5b5a783a21b0 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:01:11.051603 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de167561-8dbf-40c1-b57b-5b5a783a21b0 admin admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:01:11.076378 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f362907-6d1b-490d-9c31-8b2d0a492ed5 service cinder] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:01:11.085063 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f362907-6d1b-490d-9c31-8b2d0a492ed5 service cinder] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:01:11.119808 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=88,keystone:INSERT=5,keystone:DELETE=7 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 13:01:11.138656 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12aa55f1-b836-4ff2-9535-547d9a7ca5ca service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1666097f8ed545c6b0e4809c93e4775a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9c389ae65aaa4542bb5a5c4fd0bacf5b', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:01:11.139311 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12aa55f1-b836-4ff2-9535-547d9a7ca5ca service cinder] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:01:11.139596 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12aa55f1-b836-4ff2-9535-547d9a7ca5ca service cinder] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:01:11.139874 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12aa55f1-b836-4ff2-9535-547d9a7ca5ca service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:01:11.141840 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.fernet_utils [None req-12aa55f1-b836-4ff2-9535-547d9a7ca5ca service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92057) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:01:11.163213 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 1022/2052] 10.222.0.30 () {60 vars in 1143 bytes} [Mon May 23 13:01:10 2022] POST /identity/v3/users => generated 293 bytes in 233 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) May 23 13:01:11.168004 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de167561-8dbf-40c1-b57b-5b5a783a21b0 admin admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:01:11.183829 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de167561-8dbf-40c1-b57b-5b5a783a21b0 admin admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:01:11.193704 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d05ee50a-eb54-4472-9e2b-77df5e79b1fc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:01:11.194453 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d05ee50a-eb54-4472-9e2b-77df5e79b1fc admin admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:01:11.194738 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d05ee50a-eb54-4472-9e2b-77df5e79b1fc admin admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:01:11.195013 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d05ee50a-eb54-4472-9e2b-77df5e79b1fc admin admin] PATH_INFO: `/v3/roles` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:01:11.195747 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d05ee50a-eb54-4472-9e2b-77df5e79b1fc admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:01:11.197605 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d05ee50a-eb54-4472-9e2b-77df5e79b1fc admin admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:01:11.212250 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 1023/2053] 10.222.0.30 () {58 vars in 1121 bytes} [Mon May 23 13:01:11 2022] GET /identity/v3/roles => generated 1420 bytes in 46 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) May 23 13:01:11.218375 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d05ee50a-eb54-4472-9e2b-77df5e79b1fc admin admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:01:11.225460 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12aa55f1-b836-4ff2-9535-547d9a7ca5ca service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:01:11.227016 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12aa55f1-b836-4ff2-9535-547d9a7ca5ca service cinder] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:01:11.232040 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d05ee50a-eb54-4472-9e2b-77df5e79b1fc admin admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:01:11.236900 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99ab8ace-8608-42bc-b923-6f22a33324c3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:01:11.237580 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99ab8ace-8608-42bc-b923-6f22a33324c3 admin admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:01:11.238591 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99ab8ace-8608-42bc-b923-6f22a33324c3 admin admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:01:11.238591 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99ab8ace-8608-42bc-b923-6f22a33324c3 admin admin] PATH_INFO: `/v3/roles` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:01:11.239073 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99ab8ace-8608-42bc-b923-6f22a33324c3 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:01:11.240849 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99ab8ace-8608-42bc-b923-6f22a33324c3 admin admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:01:11.250820 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 1031/2054] 10.222.0.30 () {60 vars in 1365 bytes} [Mon May 23 13:01:11 2022] GET /identity/v3/auth/tokens => generated 3246 bytes in 175 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) May 23 13:01:11.252572 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 1024/2055] 10.222.0.30 () {58 vars in 1121 bytes} [Mon May 23 13:01:11 2022] GET /identity/v3/roles => generated 1420 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) May 23 13:01:11.257113 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12aa55f1-b836-4ff2-9535-547d9a7ca5ca service cinder] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:01:11.278245 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12aa55f1-b836-4ff2-9535-547d9a7ca5ca service cinder] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:01:11.285427 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-afe2feb7-4a80-47c5-85f3-dbbd2c7d5710 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:01:11.286473 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afe2feb7-4a80-47c5-85f3-dbbd2c7d5710 admin admin] REQUEST_METHOD: `PUT` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:01:11.286825 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afe2feb7-4a80-47c5-85f3-dbbd2c7d5710 admin admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:01:11.287307 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afe2feb7-4a80-47c5-85f3-dbbd2c7d5710 admin admin] PATH_INFO: `/v3/projects/53f8cb95bdab430d9027d8eeacf0c669/users/af3149805e5146808dd0eaf97cd68c4c/roles/80fb22507c9a4ff09d44d8f12f6fff1a` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:01:11.308957 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-afe2feb7-4a80-47c5-85f3-dbbd2c7d5710 admin admin] RBAC: Authorizing `identity:create_grant(project_id=53f8cb95bdab430d9027d8eeacf0c669, user_id=af3149805e5146808dd0eaf97cd68c4c, role_id=80fb22507c9a4ff09d44d8f12f6fff1a)` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:01:11.311818 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-afe2feb7-4a80-47c5-85f3-dbbd2c7d5710 admin admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:01:11.393190 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 1032/2056] 10.222.0.30 () {60 vars in 1482 bytes} [Mon May 23 13:01:11 2022] PUT /identity/v3/projects/53f8cb95bdab430d9027d8eeacf0c669/users/af3149805e5146808dd0eaf97cd68c4c/roles/80fb22507c9a4ff09d44d8f12f6fff1a => generated 0 bytes in 137 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) May 23 13:01:11.404048 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d33a8287-852b-4888-8f96-0b3e42ab6eec None None] REQUEST_METHOD: `POST` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:01:11.404415 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d33a8287-852b-4888-8f96-0b3e42ab6eec None None] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:01:11.404860 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d33a8287-852b-4888-8f96-0b3e42ab6eec None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:01:11.431567 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.auth.core [None req-d33a8287-852b-4888-8f96-0b3e42ab6eec None None] MFA Rules not processed for user `af3149805e5146808dd0eaf97cd68c4c`. Rule list: `[]` (Enabled: `True`). {{(pid=92056) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 13:01:11.433539 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.fernet_utils [None req-d33a8287-852b-4888-8f96-0b3e42ab6eec None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92056) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:01:11.517955 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 1025/2057] 10.222.0.30 () {58 vars in 957 bytes} [Mon May 23 13:01:11 2022] POST /identity/v3/auth/tokens => generated 3252 bytes in 114 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) May 23 13:01:11.522667 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-afe2feb7-4a80-47c5-85f3-dbbd2c7d5710 admin admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:01:11.612835 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-afe2feb7-4a80-47c5-85f3-dbbd2c7d5710 admin admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:01:11.618560 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45314414-c0fe-4504-889b-7c24e0191bdd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:01:11.619237 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45314414-c0fe-4504-889b-7c24e0191bdd admin admin] REQUEST_METHOD: `POST` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:01:11.620570 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45314414-c0fe-4504-889b-7c24e0191bdd admin admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:01:11.620570 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45314414-c0fe-4504-889b-7c24e0191bdd admin admin] PATH_INFO: `/v3/projects` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:01:11.621418 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45314414-c0fe-4504-889b-7c24e0191bdd admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:01:11.623480 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45314414-c0fe-4504-889b-7c24e0191bdd admin admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:01:11.782380 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 1033/2058] 10.222.0.30 () {60 vars in 1152 bytes} [Mon May 23 13:01:11 2022] POST /identity/v3/projects => generated 385 bytes in 261 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) May 23 13:01:11.788269 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d33a8287-852b-4888-8f96-0b3e42ab6eec None None] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:01:11.795647 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45314414-c0fe-4504-889b-7c24e0191bdd admin admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:01:11.799655 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.fernet_utils [None req-45314414-c0fe-4504-889b-7c24e0191bdd admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92057) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:01:11.804969 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d33a8287-852b-4888-8f96-0b3e42ab6eec None None] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:01:11.906699 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45314414-c0fe-4504-889b-7c24e0191bdd admin admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:01:11.911714 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d371104c-dc15-41d5-97a6-f22e1847cc45 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:01:11.912520 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d371104c-dc15-41d5-97a6-f22e1847cc45 admin admin] REQUEST_METHOD: `DELETE` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:01:11.912847 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d371104c-dc15-41d5-97a6-f22e1847cc45 admin admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:01:11.913123 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d371104c-dc15-41d5-97a6-f22e1847cc45 admin admin] PATH_INFO: `/v3/users/352102dbcd4d4614be1c236842b63d96` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:01:11.918658 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d371104c-dc15-41d5-97a6-f22e1847cc45 admin admin] RBAC: Authorizing `identity:delete_user(user_id=352102dbcd4d4614be1c236842b63d96)` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:01:11.918658 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d371104c-dc15-41d5-97a6-f22e1847cc45 admin admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:01:11.929439 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68b45174-10b6-4235-927c-b55f09181a4b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:01:11.930110 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68b45174-10b6-4235-927c-b55f09181a4b admin admin] REQUEST_METHOD: `POST` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:01:11.930396 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68b45174-10b6-4235-927c-b55f09181a4b admin admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:01:11.930679 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68b45174-10b6-4235-927c-b55f09181a4b admin admin] PATH_INFO: `/v3/users` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:01:11.932005 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68b45174-10b6-4235-927c-b55f09181a4b admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:01:11.935408 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68b45174-10b6-4235-927c-b55f09181a4b admin admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:01:12.036968 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 13:01:12.124453 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 1026/2059] 10.222.0.30 () {60 vars in 1143 bytes} [Mon May 23 13:01:11 2022] POST /identity/v3/users => generated 299 bytes in 336 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) May 23 13:01:12.128193 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68b45174-10b6-4235-927c-b55f09181a4b admin admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:01:12.145664 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68b45174-10b6-4235-927c-b55f09181a4b admin admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:01:12.172709 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f96ad3c3-0682-4ef0-9fb9-707caaee71fd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:01:12.173837 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f96ad3c3-0682-4ef0-9fb9-707caaee71fd admin admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:01:12.174437 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f96ad3c3-0682-4ef0-9fb9-707caaee71fd admin admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:01:12.175282 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f96ad3c3-0682-4ef0-9fb9-707caaee71fd admin admin] PATH_INFO: `/v3/roles` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:01:12.175825 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [None req-d371104c-dc15-41d5-97a6-f22e1847cc45 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '352102dbcd4d4614be1c236842b63d96'} {{(pid=92057) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:01:12.182286 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f96ad3c3-0682-4ef0-9fb9-707caaee71fd admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:01:12.188530 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f96ad3c3-0682-4ef0-9fb9-707caaee71fd admin admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:01:12.230611 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 1027/2060] 10.222.0.30 () {58 vars in 1121 bytes} [Mon May 23 13:01:12 2022] GET /identity/v3/roles => generated 1420 bytes in 103 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) May 23 13:01:12.233828 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f96ad3c3-0682-4ef0-9fb9-707caaee71fd admin admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:01:12.259287 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f96ad3c3-0682-4ef0-9fb9-707caaee71fd admin admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:01:12.266398 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a359c939-b169-48c6-afa1-f7ab6139278b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:01:12.267604 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a359c939-b169-48c6-afa1-f7ab6139278b admin admin] REQUEST_METHOD: `PUT` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:01:12.267989 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a359c939-b169-48c6-afa1-f7ab6139278b admin admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:01:12.268367 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a359c939-b169-48c6-afa1-f7ab6139278b admin admin] PATH_INFO: `/v3/projects/4a3560cee69a4b6a98856668d8724958/users/b3554babb24f418297b82f983ea5cff8/roles/bc3c3c9d654940cbaa5177a6be23d1bc` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:01:12.290164 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a359c939-b169-48c6-afa1-f7ab6139278b admin admin] RBAC: Authorizing `identity:create_grant(project_id=4a3560cee69a4b6a98856668d8724958, user_id=b3554babb24f418297b82f983ea5cff8, role_id=bc3c3c9d654940cbaa5177a6be23d1bc)` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:01:12.293745 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a359c939-b169-48c6-afa1-f7ab6139278b admin admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:01:12.329045 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [None req-d371104c-dc15-41d5-97a6-f22e1847cc45 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '352102dbcd4d4614be1c236842b63d96'} {{(pid=92057) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:01:12.347208 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [None req-d371104c-dc15-41d5-97a6-f22e1847cc45 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '352102dbcd4d4614be1c236842b63d96'} {{(pid=92057) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:01:12.348777 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [None req-d371104c-dc15-41d5-97a6-f22e1847cc45 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '352102dbcd4d4614be1c236842b63d96'} {{(pid=92057) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:01:12.373132 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [None req-d371104c-dc15-41d5-97a6-f22e1847cc45 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '352102dbcd4d4614be1c236842b63d96'} {{(pid=92057) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:01:12.404571 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 1028/2061] 10.222.0.30 () {60 vars in 1482 bytes} [Mon May 23 13:01:12 2022] PUT /identity/v3/projects/4a3560cee69a4b6a98856668d8724958/users/b3554babb24f418297b82f983ea5cff8/roles/bc3c3c9d654940cbaa5177a6be23d1bc => generated 0 bytes in 170 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) May 23 13:01:12.412083 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a359c939-b169-48c6-afa1-f7ab6139278b admin admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:01:12.424603 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.fernet_utils [None req-a359c939-b169-48c6-afa1-f7ab6139278b admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92056) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:01:12.438145 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 1034/2062] 10.222.0.30 () {58 vars in 1223 bytes} [Mon May 23 13:01:11 2022] DELETE /identity/v3/users/352102dbcd4d4614be1c236842b63d96 => generated 0 bytes in 648 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) May 23 13:01:12.447739 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d107e94-ae38-42c5-99f4-e4fc5af31f21 None None] REQUEST_METHOD: `POST` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:01:12.448343 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d107e94-ae38-42c5-99f4-e4fc5af31f21 None None] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:01:12.449451 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d107e94-ae38-42c5-99f4-e4fc5af31f21 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:01:12.492368 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.auth.core [None req-3d107e94-ae38-42c5-99f4-e4fc5af31f21 None None] MFA Rules not processed for user `0f7119f8fa5c4445977671140ded91a4`. Rule list: `[]` (Enabled: `True`). {{(pid=92057) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 13:01:12.494676 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.fernet_utils [None req-3d107e94-ae38-42c5-99f4-e4fc5af31f21 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92057) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:01:12.582865 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a359c939-b169-48c6-afa1-f7ab6139278b admin admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:01:12.612146 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 1035/2063] 10.222.0.30 () {58 vars in 957 bytes} [Mon May 23 13:01:12 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 168 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) May 23 13:01:12.623292 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d107e94-ae38-42c5-99f4-e4fc5af31f21 None None] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:01:12.625078 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.fernet_utils [None req-3d107e94-ae38-42c5-99f4-e4fc5af31f21 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92057) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:01:12.694429 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a98e649-0001-49b5-868d-413279630026 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:01:12.695798 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a98e649-0001-49b5-868d-413279630026 admin admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:01:12.696140 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a98e649-0001-49b5-868d-413279630026 admin admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:01:12.696876 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a98e649-0001-49b5-868d-413279630026 admin admin] PATH_INFO: `/v3/roles` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:01:12.697263 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a98e649-0001-49b5-868d-413279630026 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:01:12.701786 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a98e649-0001-49b5-868d-413279630026 admin admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:01:12.713488 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 1029/2064] 10.222.0.30 () {58 vars in 1121 bytes} [Mon May 23 13:01:12 2022] GET /identity/v3/roles => generated 1420 bytes in 306 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) May 23 13:01:12.718135 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a98e649-0001-49b5-868d-413279630026 admin admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:01:12.720913 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d107e94-ae38-42c5-99f4-e4fc5af31f21 None None] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:01:12.725455 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10efe467-e8dc-489a-950f-e6bb82230129 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2eb83ca38e79449095f3a2f947bbb06a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9c389ae65aaa4542bb5a5c4fd0bacf5b', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:01:12.726263 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10efe467-e8dc-489a-950f-e6bb82230129 service neutron] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:01:12.726767 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10efe467-e8dc-489a-950f-e6bb82230129 service neutron] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:01:12.727118 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10efe467-e8dc-489a-950f-e6bb82230129 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:01:12.731900 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a98e649-0001-49b5-868d-413279630026 admin admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:01:12.738210 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b525f02-12e1-46ed-ac25-d7b87d3ef4a3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:01:12.739209 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b525f02-12e1-46ed-ac25-d7b87d3ef4a3 admin admin] REQUEST_METHOD: `PUT` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:01:12.739428 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10efe467-e8dc-489a-950f-e6bb82230129 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:01:12.739570 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b525f02-12e1-46ed-ac25-d7b87d3ef4a3 admin admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:01:12.739924 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b525f02-12e1-46ed-ac25-d7b87d3ef4a3 admin admin] PATH_INFO: `/v3/projects/4a3560cee69a4b6a98856668d8724958/users/b3554babb24f418297b82f983ea5cff8/roles/bc3c3c9d654940cbaa5177a6be23d1bc` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:01:12.740848 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10efe467-e8dc-489a-950f-e6bb82230129 service neutron] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:01:12.742477 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b525f02-12e1-46ed-ac25-d7b87d3ef4a3 admin admin] RBAC: Authorizing `identity:create_grant(project_id=4a3560cee69a4b6a98856668d8724958, user_id=b3554babb24f418297b82f983ea5cff8, role_id=bc3c3c9d654940cbaa5177a6be23d1bc)` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:01:12.744895 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b525f02-12e1-46ed-ac25-d7b87d3ef4a3 admin admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:01:12.753139 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 1036/2065] 10.222.0.30 () {60 vars in 1365 bytes} [Mon May 23 13:01:12 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 131 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) May 23 13:01:12.760599 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 1030/2066] 10.222.0.30 () {60 vars in 1482 bytes} [Mon May 23 13:01:12 2022] PUT /identity/v3/projects/4a3560cee69a4b6a98856668d8724958/users/b3554babb24f418297b82f983ea5cff8/roles/bc3c3c9d654940cbaa5177a6be23d1bc => generated 0 bytes in 43 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) May 23 13:01:12.771557 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e4d6f5f-177a-4b12-b2cc-c840f499ce08 None None] REQUEST_METHOD: `POST` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:01:12.771852 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e4d6f5f-177a-4b12-b2cc-c840f499ce08 None None] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:01:12.772134 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e4d6f5f-177a-4b12-b2cc-c840f499ce08 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:01:12.801133 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.auth.core [None req-3e4d6f5f-177a-4b12-b2cc-c840f499ce08 None None] MFA Rules not processed for user `b3554babb24f418297b82f983ea5cff8`. Rule list: `[]` (Enabled: `True`). {{(pid=92057) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 13:01:12.802379 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.fernet_utils [None req-3e4d6f5f-177a-4b12-b2cc-c840f499ce08 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92057) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:01:12.884901 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 1037/2067] 10.222.0.30 () {58 vars in 957 bytes} [Mon May 23 13:01:12 2022] POST /identity/v3/auth/tokens => generated 3319 bytes in 114 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) May 23 13:01:12.897977 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b525f02-12e1-46ed-ac25-d7b87d3ef4a3 admin admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:01:12.899730 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.fernet_utils [None req-0b525f02-12e1-46ed-ac25-d7b87d3ef4a3 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92056) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:01:12.960840 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b525f02-12e1-46ed-ac25-d7b87d3ef4a3 admin admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:01:12.965649 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1a59e68-fe24-470f-b7b3-addbd25860ad service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1666097f8ed545c6b0e4809c93e4775a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9c389ae65aaa4542bb5a5c4fd0bacf5b', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:01:12.966327 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1a59e68-fe24-470f-b7b3-addbd25860ad service cinder] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:01:12.966608 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1a59e68-fe24-470f-b7b3-addbd25860ad service cinder] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:01:12.966885 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1a59e68-fe24-470f-b7b3-addbd25860ad service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:01:12.968688 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.fernet_utils [None req-d1a59e68-fe24-470f-b7b3-addbd25860ad service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92056) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:01:13.072020 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1a59e68-fe24-470f-b7b3-addbd25860ad service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:01:13.095193 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1a59e68-fe24-470f-b7b3-addbd25860ad service cinder] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:01:13.108675 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 1031/2068] 10.222.0.30 () {60 vars in 1365 bytes} [Mon May 23 13:01:12 2022] GET /identity/v3/auth/tokens => generated 3250 bytes in 211 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) May 23 13:01:13.327960 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e4d6f5f-177a-4b12-b2cc-c840f499ce08 None None] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:01:13.329787 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.fernet_utils [None req-3e4d6f5f-177a-4b12-b2cc-c840f499ce08 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92057) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:01:13.407803 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e4d6f5f-177a-4b12-b2cc-c840f499ce08 None None] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:01:13.413610 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f7660cc-b460-422b-95bd-658ff633ef4b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:01:13.414498 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f7660cc-b460-422b-95bd-658ff633ef4b admin admin] REQUEST_METHOD: `DELETE` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:01:13.414851 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f7660cc-b460-422b-95bd-658ff633ef4b admin admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:01:13.415306 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f7660cc-b460-422b-95bd-658ff633ef4b admin admin] PATH_INFO: `/v3/projects/e0304d5b3acb4f47b03e048b9b82dc3d` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:01:13.416782 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f7660cc-b460-422b-95bd-658ff633ef4b admin admin] RBAC: Authorizing `identity:delete_project(project_id=e0304d5b3acb4f47b03e048b9b82dc3d)` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:01:13.421191 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f7660cc-b460-422b-95bd-658ff633ef4b admin admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:01:13.446146 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 13:01:13.581333 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [None req-4f7660cc-b460-422b-95bd-658ff633ef4b admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'e0304d5b3acb4f47b03e048b9b82dc3d', 'request_id': 'req-4f7660cc-b460-422b-95bd-658ff633ef4b'} {{(pid=92057) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:01:13.593693 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [None req-4f7660cc-b460-422b-95bd-658ff633ef4b admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'e0304d5b3acb4f47b03e048b9b82dc3d', 'request_id': 'req-4f7660cc-b460-422b-95bd-658ff633ef4b'} {{(pid=92057) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:01:13.642326 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 13:01:13.731786 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [None req-4f7660cc-b460-422b-95bd-658ff633ef4b admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'e0304d5b3acb4f47b03e048b9b82dc3d', 'request_id': 'req-4f7660cc-b460-422b-95bd-658ff633ef4b'} {{(pid=92057) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:01:13.752714 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [None req-4f7660cc-b460-422b-95bd-658ff633ef4b admin admin] The token cache is being invalidate because project e0304d5b3acb4f47b03e048b9b82dc3d was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=92057) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} May 23 13:01:13.753326 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [None req-4f7660cc-b460-422b-95bd-658ff633ef4b admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=92057) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:01:13.754667 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [None req-4f7660cc-b460-422b-95bd-658ff633ef4b admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=92057) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:01:13.758137 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 1038/2069] 10.222.0.30 () {58 vars in 1232 bytes} [Mon May 23 13:01:13 2022] DELETE /identity/v3/projects/e0304d5b3acb4f47b03e048b9b82dc3d => generated 0 bytes in 431 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) May 23 13:01:13.764358 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32251ec8-a2c6-4e01-8f38-93dc29f5eaa0 None None] REQUEST_METHOD: `POST` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:01:13.764864 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32251ec8-a2c6-4e01-8f38-93dc29f5eaa0 None None] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:01:13.765333 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32251ec8-a2c6-4e01-8f38-93dc29f5eaa0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:01:13.784711 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.auth.core [None req-32251ec8-a2c6-4e01-8f38-93dc29f5eaa0 None None] MFA Rules not processed for user `0f7119f8fa5c4445977671140ded91a4`. Rule list: `[]` (Enabled: `True`). {{(pid=92056) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 13:01:13.786224 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.fernet_utils [None req-32251ec8-a2c6-4e01-8f38-93dc29f5eaa0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92056) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:01:13.912910 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 1032/2070] 10.222.0.30 () {58 vars in 957 bytes} [Mon May 23 13:01:13 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 149 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) May 23 13:01:13.922791 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-730945c3-58de-44f7-8e1e-9ca243d8433b None None] REQUEST_METHOD: `POST` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:01:13.923210 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-730945c3-58de-44f7-8e1e-9ca243d8433b None None] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:01:13.923585 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-730945c3-58de-44f7-8e1e-9ca243d8433b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:01:13.967507 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.auth.core [None req-730945c3-58de-44f7-8e1e-9ca243d8433b None None] MFA Rules not processed for user `0f7119f8fa5c4445977671140ded91a4`. Rule list: `[]` (Enabled: `True`). {{(pid=92057) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 13:01:13.974297 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.fernet_utils [None req-730945c3-58de-44f7-8e1e-9ca243d8433b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92057) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:01:13.998160 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 1039/2071] 10.222.0.30 () {58 vars in 957 bytes} [Mon May 23 13:01:13 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 76 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) May 23 13:01:14.001265 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32251ec8-a2c6-4e01-8f38-93dc29f5eaa0 None None] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:01:14.020291 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32251ec8-a2c6-4e01-8f38-93dc29f5eaa0 None None] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:01:14.034433 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1a55a3f-200c-4238-a83f-1095530466a3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:01:14.035367 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1a55a3f-200c-4238-a83f-1095530466a3 admin admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:01:14.035838 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1a55a3f-200c-4238-a83f-1095530466a3 admin admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:01:14.036292 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1a55a3f-200c-4238-a83f-1095530466a3 admin admin] PATH_INFO: `/v3/domains` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:01:14.037255 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1a55a3f-200c-4238-a83f-1095530466a3 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=92056) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} May 23 13:01:14.037712 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1a55a3f-200c-4238-a83f-1095530466a3 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:01:14.040001 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1a55a3f-200c-4238-a83f-1095530466a3 admin admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:01:14.065197 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 1033/2072] 10.222.0.30 () {58 vars in 1152 bytes} [Mon May 23 13:01:14 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 64 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) May 23 13:01:14.069502 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-730945c3-58de-44f7-8e1e-9ca243d8433b None None] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:01:14.088125 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-730945c3-58de-44f7-8e1e-9ca243d8433b None None] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:01:14.094300 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe6be543-80ea-462a-bcfa-2aae7e167368 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:01:14.095301 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe6be543-80ea-462a-bcfa-2aae7e167368 admin admin] REQUEST_METHOD: `POST` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:01:14.095788 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe6be543-80ea-462a-bcfa-2aae7e167368 admin admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:01:14.096217 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe6be543-80ea-462a-bcfa-2aae7e167368 admin admin] PATH_INFO: `/v3/projects` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:01:14.097745 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe6be543-80ea-462a-bcfa-2aae7e167368 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:01:14.100498 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe6be543-80ea-462a-bcfa-2aae7e167368 admin admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:01:14.253985 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 1040/2073] 10.222.0.30 () {60 vars in 1152 bytes} [Mon May 23 13:01:14 2022] POST /identity/v3/projects => generated 371 bytes in 185 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) May 23 13:01:14.258190 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1a55a3f-200c-4238-a83f-1095530466a3 admin admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:01:14.278173 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1a55a3f-200c-4238-a83f-1095530466a3 admin admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:01:14.364734 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc745483-77fe-4900-8288-5f8bf93d797e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:01:14.366127 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc745483-77fe-4900-8288-5f8bf93d797e admin admin] REQUEST_METHOD: `POST` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:01:14.366432 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc745483-77fe-4900-8288-5f8bf93d797e admin admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:01:14.366976 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc745483-77fe-4900-8288-5f8bf93d797e admin admin] PATH_INFO: `/v3/users` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:01:14.367556 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc745483-77fe-4900-8288-5f8bf93d797e admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:01:14.369766 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc745483-77fe-4900-8288-5f8bf93d797e admin admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:01:14.534818 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 1034/2074] 10.222.0.30 () {60 vars in 1143 bytes} [Mon May 23 13:01:14 2022] POST /identity/v3/users => generated 286 bytes in 277 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) May 23 13:01:14.538539 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe6be543-80ea-462a-bcfa-2aae7e167368 admin admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:01:14.556227 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe6be543-80ea-462a-bcfa-2aae7e167368 admin admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:01:14.568142 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-364cde19-d034-475a-b4ad-20d3657f4fda admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:01:14.568937 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-364cde19-d034-475a-b4ad-20d3657f4fda admin admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:01:14.569332 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-364cde19-d034-475a-b4ad-20d3657f4fda admin admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:01:14.569855 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-364cde19-d034-475a-b4ad-20d3657f4fda admin admin] PATH_INFO: `/v3/roles` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:01:14.570690 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-364cde19-d034-475a-b4ad-20d3657f4fda admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:01:14.572543 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-364cde19-d034-475a-b4ad-20d3657f4fda admin admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:01:14.591522 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 1041/2075] 10.222.0.30 () {58 vars in 1121 bytes} [Mon May 23 13:01:14 2022] GET /identity/v3/roles => generated 1420 bytes in 53 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) May 23 13:01:14.597478 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc745483-77fe-4900-8288-5f8bf93d797e admin admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:01:14.619448 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc745483-77fe-4900-8288-5f8bf93d797e admin admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:01:14.633107 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a69c60b7-bf51-4ac0-abbc-2febe0774b2f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:01:14.633797 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a69c60b7-bf51-4ac0-abbc-2febe0774b2f admin admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:01:14.634110 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a69c60b7-bf51-4ac0-abbc-2febe0774b2f admin admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:01:14.642441 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a69c60b7-bf51-4ac0-abbc-2febe0774b2f admin admin] PATH_INFO: `/v3/roles` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:01:14.642441 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a69c60b7-bf51-4ac0-abbc-2febe0774b2f admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:01:14.643544 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a69c60b7-bf51-4ac0-abbc-2febe0774b2f admin admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:01:14.661872 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 1035/2076] 10.222.0.30 () {58 vars in 1121 bytes} [Mon May 23 13:01:14 2022] GET /identity/v3/roles => generated 1420 bytes in 63 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) May 23 13:01:14.667822 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-364cde19-d034-475a-b4ad-20d3657f4fda admin admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:01:14.695925 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-364cde19-d034-475a-b4ad-20d3657f4fda admin admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:01:14.714949 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65e801f8-48f1-4552-828a-7524ae52c138 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:01:14.715940 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65e801f8-48f1-4552-828a-7524ae52c138 admin admin] REQUEST_METHOD: `PUT` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:01:14.716227 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65e801f8-48f1-4552-828a-7524ae52c138 admin admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:01:14.722394 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65e801f8-48f1-4552-828a-7524ae52c138 admin admin] PATH_INFO: `/v3/projects/3b3719ab772449d79dc00dc96487dc36/users/7f5e411f4d8e4f13b662cf2846720cd3/roles/80fb22507c9a4ff09d44d8f12f6fff1a` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:01:14.787444 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65e801f8-48f1-4552-828a-7524ae52c138 admin admin] RBAC: Authorizing `identity:create_grant(project_id=3b3719ab772449d79dc00dc96487dc36, user_id=7f5e411f4d8e4f13b662cf2846720cd3, role_id=80fb22507c9a4ff09d44d8f12f6fff1a)` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:01:14.798349 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65e801f8-48f1-4552-828a-7524ae52c138 admin admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:01:14.894892 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 1042/2077] 10.222.0.30 () {60 vars in 1482 bytes} [Mon May 23 13:01:14 2022] PUT /identity/v3/projects/3b3719ab772449d79dc00dc96487dc36/users/7f5e411f4d8e4f13b662cf2846720cd3/roles/80fb22507c9a4ff09d44d8f12f6fff1a => generated 0 bytes in 230 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) May 23 13:01:14.899954 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7de5fb7d-8a62-465c-9497-70084db8969f None None] REQUEST_METHOD: `POST` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:01:14.900245 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7de5fb7d-8a62-465c-9497-70084db8969f None None] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:01:14.900531 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7de5fb7d-8a62-465c-9497-70084db8969f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:01:14.926435 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.auth.core [None req-7de5fb7d-8a62-465c-9497-70084db8969f None None] MFA Rules not processed for user `0f7119f8fa5c4445977671140ded91a4`. Rule list: `[]` (Enabled: `True`). {{(pid=92056) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 13:01:14.927905 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.fernet_utils [None req-7de5fb7d-8a62-465c-9497-70084db8969f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92056) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:01:15.002125 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 1036/2078] 10.222.0.30 () {58 vars in 957 bytes} [Mon May 23 13:01:14 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 103 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) May 23 13:01:15.013209 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65e801f8-48f1-4552-828a-7524ae52c138 admin admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:01:15.015616 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.fernet_utils [None req-65e801f8-48f1-4552-828a-7524ae52c138 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92057) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:01:15.102557 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65e801f8-48f1-4552-828a-7524ae52c138 admin admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:01:15.113720 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba67ed3a-84bf-47b5-ab50-8f41ca58d51c service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2eb83ca38e79449095f3a2f947bbb06a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9c389ae65aaa4542bb5a5c4fd0bacf5b', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:01:15.114752 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba67ed3a-84bf-47b5-ab50-8f41ca58d51c service neutron] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:01:15.115065 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba67ed3a-84bf-47b5-ab50-8f41ca58d51c service neutron] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:01:15.115488 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba67ed3a-84bf-47b5-ab50-8f41ca58d51c service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:01:15.132149 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba67ed3a-84bf-47b5-ab50-8f41ca58d51c service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:01:15.135144 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba67ed3a-84bf-47b5-ab50-8f41ca58d51c service neutron] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:01:15.149869 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 1043/2079] 10.222.0.30 () {60 vars in 1365 bytes} [Mon May 23 13:01:15 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 138 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) May 23 13:01:15.308700 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 13:01:15.486631 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=100 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 13:01:16.191328 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7de5fb7d-8a62-465c-9497-70084db8969f None None] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:01:16.193078 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.fernet_utils [None req-7de5fb7d-8a62-465c-9497-70084db8969f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92056) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:01:16.217085 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7de5fb7d-8a62-465c-9497-70084db8969f None None] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:01:16.232049 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de63779c-a90b-4651-a649-3d05bb4e3a3a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:01:16.233242 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de63779c-a90b-4651-a649-3d05bb4e3a3a admin admin] REQUEST_METHOD: `DELETE` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:01:16.233681 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de63779c-a90b-4651-a649-3d05bb4e3a3a admin admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:01:16.234139 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de63779c-a90b-4651-a649-3d05bb4e3a3a admin admin] PATH_INFO: `/v3/users/db7bf754ea924d889a32cdddba47ac6e` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:01:16.237367 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de63779c-a90b-4651-a649-3d05bb4e3a3a admin admin] RBAC: Authorizing `identity:delete_user(user_id=db7bf754ea924d889a32cdddba47ac6e)` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:01:16.241948 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de63779c-a90b-4651-a649-3d05bb4e3a3a admin admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:01:16.482888 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=89,keystone:INSERT=3,keystone:DELETE=8 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 13:01:16.570270 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-de63779c-a90b-4651-a649-3d05bb4e3a3a admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'db7bf754ea924d889a32cdddba47ac6e'} {{(pid=92056) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:01:16.571081 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-de63779c-a90b-4651-a649-3d05bb4e3a3a admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'db7bf754ea924d889a32cdddba47ac6e'} {{(pid=92056) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:01:16.642781 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba67ed3a-84bf-47b5-ab50-8f41ca58d51c service neutron] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:01:16.649398 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.fernet_utils [None req-ba67ed3a-84bf-47b5-ab50-8f41ca58d51c service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92057) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:01:16.664565 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-de63779c-a90b-4651-a649-3d05bb4e3a3a admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'db7bf754ea924d889a32cdddba47ac6e'} {{(pid=92056) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:01:16.680387 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-de63779c-a90b-4651-a649-3d05bb4e3a3a admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'db7bf754ea924d889a32cdddba47ac6e'} {{(pid=92056) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:01:16.694334 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-de63779c-a90b-4651-a649-3d05bb4e3a3a admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'db7bf754ea924d889a32cdddba47ac6e'} {{(pid=92056) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:01:16.705153 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 1037/2080] 10.222.0.30 () {58 vars in 1223 bytes} [Mon May 23 13:01:16 2022] DELETE /identity/v3/users/db7bf754ea924d889a32cdddba47ac6e => generated 0 bytes in 514 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) May 23 13:01:16.707498 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0efa1463-9dbe-4cac-9a1a-2a4819a9bf0c None None] REQUEST_METHOD: `POST` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:01:16.709268 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0efa1463-9dbe-4cac-9a1a-2a4819a9bf0c None None] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:01:16.709849 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0efa1463-9dbe-4cac-9a1a-2a4819a9bf0c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:01:16.725342 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba67ed3a-84bf-47b5-ab50-8f41ca58d51c service neutron] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:01:16.742796 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.auth.core [None req-0efa1463-9dbe-4cac-9a1a-2a4819a9bf0c None None] MFA Rules not processed for user `c6a6096760734ac49ce2f26c6065b701`. Rule list: `[]` (Enabled: `True`). {{(pid=92056) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 13:01:16.743999 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.fernet_utils [None req-0efa1463-9dbe-4cac-9a1a-2a4819a9bf0c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92056) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:01:16.833055 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 1038/2081] 10.222.0.30 () {58 vars in 957 bytes} [Mon May 23 13:01:16 2022] POST /identity/v3/auth/tokens => generated 3246 bytes in 127 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) May 23 13:01:16.834641 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acf3b269-a22d-401c-a864-3c2b30a5377d None None] REQUEST_METHOD: `POST` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:01:16.835056 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acf3b269-a22d-401c-a864-3c2b30a5377d None None] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:01:16.835486 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acf3b269-a22d-401c-a864-3c2b30a5377d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:01:16.841389 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78f96d69-6e3e-498a-9fa7-41ef56518ed5 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1666097f8ed545c6b0e4809c93e4775a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9c389ae65aaa4542bb5a5c4fd0bacf5b', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:01:16.842328 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78f96d69-6e3e-498a-9fa7-41ef56518ed5 service cinder] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:01:16.842937 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78f96d69-6e3e-498a-9fa7-41ef56518ed5 service cinder] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:01:16.843218 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78f96d69-6e3e-498a-9fa7-41ef56518ed5 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:01:16.845076 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.fernet_utils [None req-78f96d69-6e3e-498a-9fa7-41ef56518ed5 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92057) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:01:16.858762 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.auth.core [None req-acf3b269-a22d-401c-a864-3c2b30a5377d None None] MFA Rules not processed for user `0f7119f8fa5c4445977671140ded91a4`. Rule list: `[]` (Enabled: `True`). {{(pid=92056) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 13:01:16.860024 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.fernet_utils [None req-acf3b269-a22d-401c-a864-3c2b30a5377d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92056) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:01:16.911103 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-78f96d69-6e3e-498a-9fa7-41ef56518ed5 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:01:16.912724 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-78f96d69-6e3e-498a-9fa7-41ef56518ed5 service cinder] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:01:16.918481 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 1039/2082] 10.222.0.30 () {58 vars in 957 bytes} [Mon May 23 13:01:16 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) May 23 13:01:16.919189 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-acf3b269-a22d-401c-a864-3c2b30a5377d None None] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:01:16.920455 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.fernet_utils [None req-acf3b269-a22d-401c-a864-3c2b30a5377d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92056) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:01:16.931325 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 1044/2083] 10.222.0.30 () {60 vars in 1365 bytes} [Mon May 23 13:01:16 2022] GET /identity/v3/auth/tokens => generated 3252 bytes in 283 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) May 23 13:01:16.931844 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78f96d69-6e3e-498a-9fa7-41ef56518ed5 service cinder] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:01:16.932146 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-acf3b269-a22d-401c-a864-3c2b30a5377d None None] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:01:16.934108 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.fernet_utils [None req-78f96d69-6e3e-498a-9fa7-41ef56518ed5 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92057) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:01:16.937228 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aaba9c43-caa7-4947-93ce-a26fd4f6fd84 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:01:16.937931 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aaba9c43-caa7-4947-93ce-a26fd4f6fd84 admin admin] REQUEST_METHOD: `POST` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:01:16.938308 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aaba9c43-caa7-4947-93ce-a26fd4f6fd84 admin admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:01:16.938462 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aaba9c43-caa7-4947-93ce-a26fd4f6fd84 admin admin] PATH_INFO: `/v3/projects` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:01:16.939290 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aaba9c43-caa7-4947-93ce-a26fd4f6fd84 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:01:16.941345 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aaba9c43-caa7-4947-93ce-a26fd4f6fd84 admin admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:01:17.031088 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78f96d69-6e3e-498a-9fa7-41ef56518ed5 service cinder] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:01:17.036213 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-139f5c62-e5fa-4f8b-830c-851bba176511 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2eb83ca38e79449095f3a2f947bbb06a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9c389ae65aaa4542bb5a5c4fd0bacf5b', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:01:17.038786 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-139f5c62-e5fa-4f8b-830c-851bba176511 service neutron] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:01:17.039455 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-139f5c62-e5fa-4f8b-830c-851bba176511 service neutron] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:01:17.039951 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-139f5c62-e5fa-4f8b-830c-851bba176511 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:01:17.057600 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-139f5c62-e5fa-4f8b-830c-851bba176511 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:01:17.059055 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-139f5c62-e5fa-4f8b-830c-851bba176511 service neutron] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:01:17.066195 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 1040/2084] 10.222.0.30 () {60 vars in 1152 bytes} [Mon May 23 13:01:16 2022] POST /identity/v3/projects => generated 377 bytes in 147 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) May 23 13:01:17.071634 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aaba9c43-caa7-4947-93ce-a26fd4f6fd84 admin admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:01:17.075912 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 1045/2085] 10.222.0.30 () {60 vars in 1365 bytes} [Mon May 23 13:01:16 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 145 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) May 23 13:01:17.110358 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee404543-1fcd-4037-9fff-d2ca343a4709 None None] REQUEST_METHOD: `POST` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:01:17.110679 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee404543-1fcd-4037-9fff-d2ca343a4709 None None] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:01:17.110965 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee404543-1fcd-4037-9fff-d2ca343a4709 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:01:17.171948 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.auth.core [None req-ee404543-1fcd-4037-9fff-d2ca343a4709 None None] MFA Rules not processed for user `7f5e411f4d8e4f13b662cf2846720cd3`. Rule list: `[]` (Enabled: `True`). {{(pid=92057) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 13:01:17.173555 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.fernet_utils [None req-ee404543-1fcd-4037-9fff-d2ca343a4709 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92057) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:01:17.176055 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aaba9c43-caa7-4947-93ce-a26fd4f6fd84 admin admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:01:17.181311 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1fad4b3-e43f-43ff-a3e4-afdb7be9d779 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:01:17.182210 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1fad4b3-e43f-43ff-a3e4-afdb7be9d779 admin admin] REQUEST_METHOD: `POST` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:01:17.182368 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1fad4b3-e43f-43ff-a3e4-afdb7be9d779 admin admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:01:17.184334 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1fad4b3-e43f-43ff-a3e4-afdb7be9d779 admin admin] PATH_INFO: `/v3/users` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:01:17.185111 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1fad4b3-e43f-43ff-a3e4-afdb7be9d779 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:01:17.187672 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1fad4b3-e43f-43ff-a3e4-afdb7be9d779 admin admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:01:17.204080 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 13:01:17.280591 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 1046/2086] 10.222.0.30 () {58 vars in 957 bytes} [Mon May 23 13:01:17 2022] POST /identity/v3/auth/tokens => generated 3238 bytes in 171 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) May 23 13:01:17.328364 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 1041/2087] 10.222.0.30 () {60 vars in 1143 bytes} [Mon May 23 13:01:17 2022] POST /identity/v3/users => generated 295 bytes in 252 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) May 23 13:01:17.328543 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee404543-1fcd-4037-9fff-d2ca343a4709 None None] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:01:17.338887 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee404543-1fcd-4037-9fff-d2ca343a4709 None None] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:01:17.343754 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8882f8a6-fe3f-421c-9d8a-0d5863260a17 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:01:17.344248 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8882f8a6-fe3f-421c-9d8a-0d5863260a17 admin admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:01:17.344453 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8882f8a6-fe3f-421c-9d8a-0d5863260a17 admin admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:01:17.344735 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8882f8a6-fe3f-421c-9d8a-0d5863260a17 admin admin] PATH_INFO: `/v3/roles` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:01:17.345478 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8882f8a6-fe3f-421c-9d8a-0d5863260a17 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:01:17.350953 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8882f8a6-fe3f-421c-9d8a-0d5863260a17 admin admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:01:17.361116 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1fad4b3-e43f-43ff-a3e4-afdb7be9d779 admin admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:01:17.364986 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 1047/2088] 10.222.0.30 () {58 vars in 1121 bytes} [Mon May 23 13:01:17 2022] GET /identity/v3/roles => generated 1420 bytes in 38 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) May 23 13:01:17.366306 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.fernet_utils [None req-c1fad4b3-e43f-43ff-a3e4-afdb7be9d779 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92056) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:01:17.369074 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8882f8a6-fe3f-421c-9d8a-0d5863260a17 admin admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:01:17.381015 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8882f8a6-fe3f-421c-9d8a-0d5863260a17 admin admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:01:17.393023 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d0104b7-9997-448d-839e-dcc56af394ee admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:01:17.394292 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d0104b7-9997-448d-839e-dcc56af394ee admin admin] REQUEST_METHOD: `PUT` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:01:17.394756 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d0104b7-9997-448d-839e-dcc56af394ee admin admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:01:17.395207 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d0104b7-9997-448d-839e-dcc56af394ee admin admin] PATH_INFO: `/v3/projects/6ea0b13fcc8d4dc78571a933bbf475b7/users/a6a7f78a024b43a6b89f4e2fe5c6c28b/roles/bc3c3c9d654940cbaa5177a6be23d1bc` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:01:17.426842 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d0104b7-9997-448d-839e-dcc56af394ee admin admin] RBAC: Authorizing `identity:create_grant(project_id=6ea0b13fcc8d4dc78571a933bbf475b7, user_id=a6a7f78a024b43a6b89f4e2fe5c6c28b, role_id=bc3c3c9d654940cbaa5177a6be23d1bc)` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:01:17.430511 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d0104b7-9997-448d-839e-dcc56af394ee admin admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:01:17.459911 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1fad4b3-e43f-43ff-a3e4-afdb7be9d779 admin admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:01:17.464980 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-713d422d-a8fa-4edf-a683-8922b42be8bc service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1666097f8ed545c6b0e4809c93e4775a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9c389ae65aaa4542bb5a5c4fd0bacf5b', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:01:17.465807 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-713d422d-a8fa-4edf-a683-8922b42be8bc service cinder] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:01:17.466282 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-713d422d-a8fa-4edf-a683-8922b42be8bc service cinder] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:01:17.466576 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-713d422d-a8fa-4edf-a683-8922b42be8bc service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:01:17.479931 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-713d422d-a8fa-4edf-a683-8922b42be8bc service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:01:17.481634 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-713d422d-a8fa-4edf-a683-8922b42be8bc service cinder] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:01:17.498682 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 1042/2089] 10.222.0.30 () {60 vars in 1365 bytes} [Mon May 23 13:01:17 2022] GET /identity/v3/auth/tokens => generated 3238 bytes in 138 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) May 23 13:01:17.528548 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 1048/2090] 10.222.0.30 () {60 vars in 1482 bytes} [Mon May 23 13:01:17 2022] PUT /identity/v3/projects/6ea0b13fcc8d4dc78571a933bbf475b7/users/a6a7f78a024b43a6b89f4e2fe5c6c28b/roles/bc3c3c9d654940cbaa5177a6be23d1bc => generated 0 bytes in 160 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) May 23 13:01:17.532647 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-713d422d-a8fa-4edf-a683-8922b42be8bc service cinder] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:01:17.549214 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-713d422d-a8fa-4edf-a683-8922b42be8bc service cinder] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:01:17.671417 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf369f48-90f5-4609-9b0c-1d5547cdbeb9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:01:17.672175 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf369f48-90f5-4609-9b0c-1d5547cdbeb9 admin admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:01:17.672559 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf369f48-90f5-4609-9b0c-1d5547cdbeb9 admin admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:01:17.672933 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf369f48-90f5-4609-9b0c-1d5547cdbeb9 admin admin] PATH_INFO: `/v3/roles` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:01:17.674601 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf369f48-90f5-4609-9b0c-1d5547cdbeb9 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:01:17.676798 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf369f48-90f5-4609-9b0c-1d5547cdbeb9 admin admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:01:17.695617 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 1043/2091] 10.222.0.30 () {58 vars in 1121 bytes} [Mon May 23 13:01:17 2022] GET /identity/v3/roles => generated 1420 bytes in 161 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) May 23 13:01:17.701673 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d0104b7-9997-448d-839e-dcc56af394ee admin admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:01:17.714630 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d0104b7-9997-448d-839e-dcc56af394ee admin admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:01:17.719481 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d6464d1-254e-4790-945e-413a0b5d9810 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:01:17.720826 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d6464d1-254e-4790-945e-413a0b5d9810 admin admin] REQUEST_METHOD: `PUT` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:01:17.721768 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d6464d1-254e-4790-945e-413a0b5d9810 admin admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:01:17.722077 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d6464d1-254e-4790-945e-413a0b5d9810 admin admin] PATH_INFO: `/v3/projects/6ea0b13fcc8d4dc78571a933bbf475b7/users/a6a7f78a024b43a6b89f4e2fe5c6c28b/roles/bc3c3c9d654940cbaa5177a6be23d1bc` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:01:17.724422 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d6464d1-254e-4790-945e-413a0b5d9810 admin admin] RBAC: Authorizing `identity:create_grant(project_id=6ea0b13fcc8d4dc78571a933bbf475b7, user_id=a6a7f78a024b43a6b89f4e2fe5c6c28b, role_id=bc3c3c9d654940cbaa5177a6be23d1bc)` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:01:17.727528 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d6464d1-254e-4790-945e-413a0b5d9810 admin admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:01:17.736825 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf369f48-90f5-4609-9b0c-1d5547cdbeb9 admin admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:01:17.739068 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.fernet_utils [None req-cf369f48-90f5-4609-9b0c-1d5547cdbeb9 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92056) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:01:17.742642 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 1049/2092] 10.222.0.30 () {60 vars in 1482 bytes} [Mon May 23 13:01:17 2022] PUT /identity/v3/projects/6ea0b13fcc8d4dc78571a933bbf475b7/users/a6a7f78a024b43a6b89f4e2fe5c6c28b/roles/bc3c3c9d654940cbaa5177a6be23d1bc => generated 0 bytes in 41 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) May 23 13:01:17.757375 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf369f48-90f5-4609-9b0c-1d5547cdbeb9 admin admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:01:17.870949 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd997fdd-9b84-4824-98d3-c1bd556915ad admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:01:17.872709 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd997fdd-9b84-4824-98d3-c1bd556915ad admin admin] REQUEST_METHOD: `DELETE` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:01:17.873007 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd997fdd-9b84-4824-98d3-c1bd556915ad admin admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:01:17.873291 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd997fdd-9b84-4824-98d3-c1bd556915ad admin admin] PATH_INFO: `/v3/projects/e4a9af8184a1434797a4d055ad1d5fdb` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:01:17.878540 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd997fdd-9b84-4824-98d3-c1bd556915ad admin admin] RBAC: Authorizing `identity:delete_project(project_id=e4a9af8184a1434797a4d055ad1d5fdb)` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:01:17.881141 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd997fdd-9b84-4824-98d3-c1bd556915ad admin admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:01:18.041553 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-fd997fdd-9b84-4824-98d3-c1bd556915ad admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'e4a9af8184a1434797a4d055ad1d5fdb', 'request_id': 'req-fd997fdd-9b84-4824-98d3-c1bd556915ad'} {{(pid=92056) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:01:18.145130 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-fd997fdd-9b84-4824-98d3-c1bd556915ad admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'e4a9af8184a1434797a4d055ad1d5fdb', 'request_id': 'req-fd997fdd-9b84-4824-98d3-c1bd556915ad'} {{(pid=92056) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:01:18.153545 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-fd997fdd-9b84-4824-98d3-c1bd556915ad admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'e4a9af8184a1434797a4d055ad1d5fdb', 'request_id': 'req-fd997fdd-9b84-4824-98d3-c1bd556915ad'} {{(pid=92056) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:01:18.166357 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-fd997fdd-9b84-4824-98d3-c1bd556915ad admin admin] The token cache is being invalidate because project e4a9af8184a1434797a4d055ad1d5fdb was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=92056) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} May 23 13:01:18.166734 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-fd997fdd-9b84-4824-98d3-c1bd556915ad admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=92056) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:01:18.168180 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-fd997fdd-9b84-4824-98d3-c1bd556915ad admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=92056) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:01:18.170079 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 1044/2093] 10.222.0.30 () {58 vars in 1232 bytes} [Mon May 23 13:01:17 2022] DELETE /identity/v3/projects/e4a9af8184a1434797a4d055ad1d5fdb => generated 0 bytes in 434 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) May 23 13:01:18.174043 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d6464d1-254e-4790-945e-413a0b5d9810 admin admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:01:18.178497 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.fernet_utils [None req-9d6464d1-254e-4790-945e-413a0b5d9810 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92057) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:01:18.202659 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d6464d1-254e-4790-945e-413a0b5d9810 admin admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:01:18.295390 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc323d37-566d-4750-9c5a-8487b2327d51 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:01:18.296435 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc323d37-566d-4750-9c5a-8487b2327d51 admin admin] REQUEST_METHOD: `DELETE` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:01:18.296781 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc323d37-566d-4750-9c5a-8487b2327d51 admin admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:01:18.297062 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc323d37-566d-4750-9c5a-8487b2327d51 admin admin] PATH_INFO: `/v3/users/b12ae34252d044e1890682574bf18cbb` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:01:18.299140 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc323d37-566d-4750-9c5a-8487b2327d51 admin admin] RBAC: Authorizing `identity:delete_user(user_id=b12ae34252d044e1890682574bf18cbb)` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:01:18.301378 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc323d37-566d-4750-9c5a-8487b2327d51 admin admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:01:18.500603 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [None req-fc323d37-566d-4750-9c5a-8487b2327d51 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'b12ae34252d044e1890682574bf18cbb'} {{(pid=92057) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:01:18.591737 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [None req-fc323d37-566d-4750-9c5a-8487b2327d51 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'b12ae34252d044e1890682574bf18cbb'} {{(pid=92057) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:01:18.606633 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [None req-fc323d37-566d-4750-9c5a-8487b2327d51 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'b12ae34252d044e1890682574bf18cbb'} {{(pid=92057) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:01:18.610485 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [None req-fc323d37-566d-4750-9c5a-8487b2327d51 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'b12ae34252d044e1890682574bf18cbb'} {{(pid=92057) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:01:18.637774 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [None req-fc323d37-566d-4750-9c5a-8487b2327d51 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'b12ae34252d044e1890682574bf18cbb'} {{(pid=92057) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:01:18.646408 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 1050/2094] 10.222.0.30 () {58 vars in 1223 bytes} [Mon May 23 13:01:18 2022] DELETE /identity/v3/users/b12ae34252d044e1890682574bf18cbb => generated 0 bytes in 473 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) May 23 13:01:18.963390 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 13:01:19.074347 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd997fdd-9b84-4824-98d3-c1bd556915ad admin admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:01:19.076271 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.fernet_utils [None req-fd997fdd-9b84-4824-98d3-c1bd556915ad admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92056) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:01:19.077007 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc323d37-566d-4750-9c5a-8487b2327d51 admin admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:01:19.078615 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.fernet_utils [None req-fc323d37-566d-4750-9c5a-8487b2327d51 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92057) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:01:19.093274 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd997fdd-9b84-4824-98d3-c1bd556915ad admin admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:01:19.161276 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc323d37-566d-4750-9c5a-8487b2327d51 admin admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:01:19.167585 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b4f34b8-f763-41d8-8fcb-dc33fd0db095 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1666097f8ed545c6b0e4809c93e4775a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9c389ae65aaa4542bb5a5c4fd0bacf5b', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:01:19.168536 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b4f34b8-f763-41d8-8fcb-dc33fd0db095 service cinder] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:01:19.169316 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b4f34b8-f763-41d8-8fcb-dc33fd0db095 service cinder] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:01:19.169801 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b4f34b8-f763-41d8-8fcb-dc33fd0db095 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:01:19.173299 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.fernet_utils [None req-8b4f34b8-f763-41d8-8fcb-dc33fd0db095 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92057) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:01:19.203067 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=98,keystone:DELETE=2 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 13:01:19.208711 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec130fdb-3572-4656-baa7-c23023af0306 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:01:19.216749 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec130fdb-3572-4656-baa7-c23023af0306 admin admin] REQUEST_METHOD: `DELETE` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:01:19.216749 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec130fdb-3572-4656-baa7-c23023af0306 admin admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:01:19.216749 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec130fdb-3572-4656-baa7-c23023af0306 admin admin] PATH_INFO: `/v3/users/bf73cd653d094781b1d2fa4e6d1ad8e9` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:01:19.216749 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec130fdb-3572-4656-baa7-c23023af0306 admin admin] RBAC: Authorizing `identity:delete_user(user_id=bf73cd653d094781b1d2fa4e6d1ad8e9)` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:01:19.216749 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec130fdb-3572-4656-baa7-c23023af0306 admin admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:01:19.314658 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b4f34b8-f763-41d8-8fcb-dc33fd0db095 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:01:19.316278 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b4f34b8-f763-41d8-8fcb-dc33fd0db095 service cinder] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:01:19.342929 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 1051/2095] 10.222.0.30 () {60 vars in 1365 bytes} [Mon May 23 13:01:19 2022] GET /identity/v3/auth/tokens => generated 3319 bytes in 266 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) May 23 13:01:19.421701 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-ec130fdb-3572-4656-baa7-c23023af0306 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'bf73cd653d094781b1d2fa4e6d1ad8e9'} {{(pid=92056) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:01:19.422562 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-ec130fdb-3572-4656-baa7-c23023af0306 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'bf73cd653d094781b1d2fa4e6d1ad8e9'} {{(pid=92056) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:01:19.519996 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-ec130fdb-3572-4656-baa7-c23023af0306 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'bf73cd653d094781b1d2fa4e6d1ad8e9'} {{(pid=92056) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:01:19.533917 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-ec130fdb-3572-4656-baa7-c23023af0306 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'bf73cd653d094781b1d2fa4e6d1ad8e9'} {{(pid=92056) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:01:19.541408 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-ec130fdb-3572-4656-baa7-c23023af0306 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'bf73cd653d094781b1d2fa4e6d1ad8e9'} {{(pid=92056) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:01:19.548631 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 1045/2096] 10.222.0.30 () {58 vars in 1223 bytes} [Mon May 23 13:01:19 2022] DELETE /identity/v3/users/bf73cd653d094781b1d2fa4e6d1ad8e9 => generated 0 bytes in 476 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) May 23 13:01:19.553290 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6950a5ca-4b4a-4e36-aea8-713d1c85aae6 None None] REQUEST_METHOD: `POST` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:01:19.553586 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6950a5ca-4b4a-4e36-aea8-713d1c85aae6 None None] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:01:19.553894 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6950a5ca-4b4a-4e36-aea8-713d1c85aae6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:01:19.573890 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.auth.core [None req-6950a5ca-4b4a-4e36-aea8-713d1c85aae6 None None] MFA Rules not processed for user `0f7119f8fa5c4445977671140ded91a4`. Rule list: `[]` (Enabled: `True`). {{(pid=92057) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 13:01:19.575145 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.fernet_utils [None req-6950a5ca-4b4a-4e36-aea8-713d1c85aae6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92057) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:01:19.588062 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec130fdb-3572-4656-baa7-c23023af0306 admin admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:01:19.590350 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.fernet_utils [None req-ec130fdb-3572-4656-baa7-c23023af0306 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92056) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:01:19.668754 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 1052/2097] 10.222.0.30 () {58 vars in 957 bytes} [Mon May 23 13:01:19 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 116 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) May 23 13:01:19.678281 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6950a5ca-4b4a-4e36-aea8-713d1c85aae6 None None] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:01:19.679897 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.fernet_utils [None req-6950a5ca-4b4a-4e36-aea8-713d1c85aae6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92057) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:01:19.720720 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec130fdb-3572-4656-baa7-c23023af0306 admin admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:01:19.726841 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ee2967f-80a0-4a9c-85e0-fa34cf6f427e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:01:19.728104 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ee2967f-80a0-4a9c-85e0-fa34cf6f427e admin admin] REQUEST_METHOD: `DELETE` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:01:19.728694 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ee2967f-80a0-4a9c-85e0-fa34cf6f427e admin admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:01:19.729981 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ee2967f-80a0-4a9c-85e0-fa34cf6f427e admin admin] PATH_INFO: `/v3/projects/24e100fc831d4e53919f466bccb8929e` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:01:19.731605 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ee2967f-80a0-4a9c-85e0-fa34cf6f427e admin admin] RBAC: Authorizing `identity:delete_project(project_id=24e100fc831d4e53919f466bccb8929e)` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:01:19.734790 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ee2967f-80a0-4a9c-85e0-fa34cf6f427e admin admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:01:19.799944 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6950a5ca-4b4a-4e36-aea8-713d1c85aae6 None None] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:01:19.803542 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b198d1c-c03d-4901-86c7-be1f9532e910 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2eb83ca38e79449095f3a2f947bbb06a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9c389ae65aaa4542bb5a5c4fd0bacf5b', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:01:19.804211 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b198d1c-c03d-4901-86c7-be1f9532e910 service neutron] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:01:19.804474 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b198d1c-c03d-4901-86c7-be1f9532e910 service neutron] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:01:19.804745 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b198d1c-c03d-4901-86c7-be1f9532e910 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:01:19.816148 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b198d1c-c03d-4901-86c7-be1f9532e910 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:01:19.817687 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b198d1c-c03d-4901-86c7-be1f9532e910 service neutron] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:01:19.832800 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 1053/2098] 10.222.0.30 () {60 vars in 1365 bytes} [Mon May 23 13:01:19 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 155 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) May 23 13:01:19.924608 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-6ee2967f-80a0-4a9c-85e0-fa34cf6f427e admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '24e100fc831d4e53919f466bccb8929e', 'request_id': 'req-6ee2967f-80a0-4a9c-85e0-fa34cf6f427e'} {{(pid=92056) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:01:20.053780 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-6ee2967f-80a0-4a9c-85e0-fa34cf6f427e admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '24e100fc831d4e53919f466bccb8929e', 'request_id': 'req-6ee2967f-80a0-4a9c-85e0-fa34cf6f427e'} {{(pid=92056) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:01:20.066725 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-6ee2967f-80a0-4a9c-85e0-fa34cf6f427e admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '24e100fc831d4e53919f466bccb8929e', 'request_id': 'req-6ee2967f-80a0-4a9c-85e0-fa34cf6f427e'} {{(pid=92056) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:01:20.091634 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-6ee2967f-80a0-4a9c-85e0-fa34cf6f427e admin admin] The token cache is being invalidate because project 24e100fc831d4e53919f466bccb8929e was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=92056) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} May 23 13:01:20.091634 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-6ee2967f-80a0-4a9c-85e0-fa34cf6f427e admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=92056) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:01:20.092392 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-6ee2967f-80a0-4a9c-85e0-fa34cf6f427e admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=92056) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:01:20.094471 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 1046/2099] 10.222.0.30 () {58 vars in 1232 bytes} [Mon May 23 13:01:19 2022] DELETE /identity/v3/projects/24e100fc831d4e53919f466bccb8929e => generated 0 bytes in 508 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) May 23 13:01:20.099319 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27e30a8d-9da3-403c-9aed-1fc2ada00b5d None None] REQUEST_METHOD: `POST` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:01:20.099848 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27e30a8d-9da3-403c-9aed-1fc2ada00b5d None None] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:01:20.100293 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27e30a8d-9da3-403c-9aed-1fc2ada00b5d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:01:20.157696 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.auth.core [None req-27e30a8d-9da3-403c-9aed-1fc2ada00b5d None None] MFA Rules not processed for user `0f7119f8fa5c4445977671140ded91a4`. Rule list: `[]` (Enabled: `True`). {{(pid=92057) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 13:01:20.162159 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.fernet_utils [None req-27e30a8d-9da3-403c-9aed-1fc2ada00b5d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92057) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:01:20.292759 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 1054/2100] 10.222.0.30 () {58 vars in 957 bytes} [Mon May 23 13:01:20 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 194 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) May 23 13:01:20.301775 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6123fd3f-390e-4628-9f2c-5c14b04f0945 None None] REQUEST_METHOD: `POST` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:01:20.302424 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6123fd3f-390e-4628-9f2c-5c14b04f0945 None None] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:01:20.302958 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6123fd3f-390e-4628-9f2c-5c14b04f0945 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:01:20.352301 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=90,keystone:INSERT=3,keystone:DELETE=7 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 13:01:20.390345 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.auth.core [None req-6123fd3f-390e-4628-9f2c-5c14b04f0945 None None] MFA Rules not processed for user `0f7119f8fa5c4445977671140ded91a4`. Rule list: `[]` (Enabled: `True`). {{(pid=92056) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 13:01:20.393984 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.fernet_utils [None req-6123fd3f-390e-4628-9f2c-5c14b04f0945 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92056) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:01:20.409816 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 1047/2101] 10.222.0.30 () {58 vars in 957 bytes} [Mon May 23 13:01:20 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 110 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) May 23 13:01:20.415485 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27e30a8d-9da3-403c-9aed-1fc2ada00b5d None None] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:01:20.445185 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27e30a8d-9da3-403c-9aed-1fc2ada00b5d None None] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:01:20.462422 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b56f4dd2-0720-4ab8-a5e6-fb835ed95208 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:01:20.462422 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b56f4dd2-0720-4ab8-a5e6-fb835ed95208 admin admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:01:20.462422 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b56f4dd2-0720-4ab8-a5e6-fb835ed95208 admin admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:01:20.462422 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b56f4dd2-0720-4ab8-a5e6-fb835ed95208 admin admin] PATH_INFO: `/v3/domains` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:01:20.462889 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b56f4dd2-0720-4ab8-a5e6-fb835ed95208 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=92057) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} May 23 13:01:20.462889 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b56f4dd2-0720-4ab8-a5e6-fb835ed95208 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:01:20.462889 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b56f4dd2-0720-4ab8-a5e6-fb835ed95208 admin admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:01:20.509530 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 1055/2102] 10.222.0.30 () {58 vars in 1152 bytes} [Mon May 23 13:01:20 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 98 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) May 23 13:01:20.512621 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6123fd3f-390e-4628-9f2c-5c14b04f0945 None None] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:01:20.527460 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6123fd3f-390e-4628-9f2c-5c14b04f0945 None None] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:01:20.543746 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c225b38-9f5c-42bb-96cf-321e5cc83d19 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:01:20.544526 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c225b38-9f5c-42bb-96cf-321e5cc83d19 admin admin] REQUEST_METHOD: `POST` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:01:20.544884 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c225b38-9f5c-42bb-96cf-321e5cc83d19 admin admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:01:20.545980 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c225b38-9f5c-42bb-96cf-321e5cc83d19 admin admin] PATH_INFO: `/v3/projects` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:01:20.546777 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c225b38-9f5c-42bb-96cf-321e5cc83d19 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:01:20.549220 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c225b38-9f5c-42bb-96cf-321e5cc83d19 admin admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:01:20.578010 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 13:01:20.717249 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 1048/2103] 10.222.0.30 () {60 vars in 1152 bytes} [Mon May 23 13:01:20 2022] POST /identity/v3/projects => generated 383 bytes in 205 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) May 23 13:01:20.717610 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b56f4dd2-0720-4ab8-a5e6-fb835ed95208 admin admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:01:20.739975 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b56f4dd2-0720-4ab8-a5e6-fb835ed95208 admin admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:01:20.787316 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b47bb460-d089-4286-8552-52209a2f2c15 None None] REQUEST_METHOD: `POST` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:01:20.788006 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b47bb460-d089-4286-8552-52209a2f2c15 None None] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:01:20.788437 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b47bb460-d089-4286-8552-52209a2f2c15 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:01:20.825910 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.auth.core [None req-b47bb460-d089-4286-8552-52209a2f2c15 None None] MFA Rules not processed for user `7f5e411f4d8e4f13b662cf2846720cd3`. Rule list: `[]` (Enabled: `True`). {{(pid=92056) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 13:01:20.827214 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.fernet_utils [None req-b47bb460-d089-4286-8552-52209a2f2c15 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92056) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:01:20.913302 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-abf0a9e1-d4c1-481c-813a-62da6043e839 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:01:20.913818 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abf0a9e1-d4c1-481c-813a-62da6043e839 admin admin] REQUEST_METHOD: `POST` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:01:20.914179 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abf0a9e1-d4c1-481c-813a-62da6043e839 admin admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:01:20.914408 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abf0a9e1-d4c1-481c-813a-62da6043e839 admin admin] PATH_INFO: `/v3/users` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:01:20.917191 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-abf0a9e1-d4c1-481c-813a-62da6043e839 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:01:20.920859 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-abf0a9e1-d4c1-481c-813a-62da6043e839 admin admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:01:20.971363 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 1049/2104] 10.222.0.30 () {58 vars in 957 bytes} [Mon May 23 13:01:20 2022] POST /identity/v3/auth/tokens => generated 3238 bytes in 184 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) May 23 13:01:20.986800 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b47bb460-d089-4286-8552-52209a2f2c15 None None] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:01:20.991246 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.fernet_utils [None req-b47bb460-d089-4286-8552-52209a2f2c15 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92056) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:01:21.035004 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b47bb460-d089-4286-8552-52209a2f2c15 None None] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:01:21.050637 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8803cbd8-6f5e-4464-842b-f9f556a5484c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:01:21.056233 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8803cbd8-6f5e-4464-842b-f9f556a5484c admin admin] REQUEST_METHOD: `DELETE` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:01:21.056812 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8803cbd8-6f5e-4464-842b-f9f556a5484c admin admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:01:21.056812 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8803cbd8-6f5e-4464-842b-f9f556a5484c admin admin] PATH_INFO: `/v3/projects/e8e72a1080a44ad3a0ef7918563a2a82` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:01:21.081001 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 1056/2105] 10.222.0.30 () {60 vars in 1143 bytes} [Mon May 23 13:01:20 2022] POST /identity/v3/users => generated 292 bytes in 364 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) May 23 13:01:21.081400 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8803cbd8-6f5e-4464-842b-f9f556a5484c admin admin] RBAC: Authorizing `identity:delete_project(project_id=e8e72a1080a44ad3a0ef7918563a2a82)` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:01:21.083404 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8803cbd8-6f5e-4464-842b-f9f556a5484c admin admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:01:21.085487 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-abf0a9e1-d4c1-481c-813a-62da6043e839 admin admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:01:21.109596 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-abf0a9e1-d4c1-481c-813a-62da6043e839 admin admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:01:21.122968 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4d4c57e-646a-42db-a1a0-4ef59b923ef9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:01:21.125324 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4d4c57e-646a-42db-a1a0-4ef59b923ef9 admin admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:01:21.127009 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4d4c57e-646a-42db-a1a0-4ef59b923ef9 admin admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:01:21.127455 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4d4c57e-646a-42db-a1a0-4ef59b923ef9 admin admin] PATH_INFO: `/v3/roles` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:01:21.131397 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4d4c57e-646a-42db-a1a0-4ef59b923ef9 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:01:21.139221 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4d4c57e-646a-42db-a1a0-4ef59b923ef9 admin admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:01:21.168297 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 1057/2106] 10.222.0.30 () {58 vars in 1121 bytes} [Mon May 23 13:01:21 2022] GET /identity/v3/roles => generated 1420 bytes in 83 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) May 23 13:01:21.169513 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4d4c57e-646a-42db-a1a0-4ef59b923ef9 admin admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:01:21.182295 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4d4c57e-646a-42db-a1a0-4ef59b923ef9 admin admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:01:21.189293 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f273848c-9216-4d60-86c1-0dcdd0f754a0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:01:21.190380 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f273848c-9216-4d60-86c1-0dcdd0f754a0 admin admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:01:21.190819 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f273848c-9216-4d60-86c1-0dcdd0f754a0 admin admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:01:21.191239 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f273848c-9216-4d60-86c1-0dcdd0f754a0 admin admin] PATH_INFO: `/v3/roles` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:01:21.192129 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f273848c-9216-4d60-86c1-0dcdd0f754a0 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:01:21.194008 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f273848c-9216-4d60-86c1-0dcdd0f754a0 admin admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:01:21.240913 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 1058/2107] 10.222.0.30 () {58 vars in 1121 bytes} [Mon May 23 13:01:21 2022] GET /identity/v3/roles => generated 1420 bytes in 72 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) May 23 13:01:21.242379 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f273848c-9216-4d60-86c1-0dcdd0f754a0 admin admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:01:21.258728 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f273848c-9216-4d60-86c1-0dcdd0f754a0 admin admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:01:21.272701 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f294470-02bb-4a88-88af-afb6641cb790 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:01:21.274087 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f294470-02bb-4a88-88af-afb6641cb790 admin admin] REQUEST_METHOD: `PUT` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:01:21.274667 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f294470-02bb-4a88-88af-afb6641cb790 admin admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:01:21.275228 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f294470-02bb-4a88-88af-afb6641cb790 admin admin] PATH_INFO: `/v3/projects/ec74e160a3f74c11b1d25cedcd9be911/users/e54f105d7dcc4dc993ca024fe3a1faf1/roles/80fb22507c9a4ff09d44d8f12f6fff1a` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:01:21.321003 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-8803cbd8-6f5e-4464-842b-f9f556a5484c admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'e8e72a1080a44ad3a0ef7918563a2a82', 'request_id': 'req-8803cbd8-6f5e-4464-842b-f9f556a5484c'} {{(pid=92056) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:01:21.330493 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f294470-02bb-4a88-88af-afb6641cb790 admin admin] RBAC: Authorizing `identity:create_grant(project_id=ec74e160a3f74c11b1d25cedcd9be911, user_id=e54f105d7dcc4dc993ca024fe3a1faf1, role_id=80fb22507c9a4ff09d44d8f12f6fff1a)` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:01:21.335380 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f294470-02bb-4a88-88af-afb6641cb790 admin admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:01:21.426320 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-8803cbd8-6f5e-4464-842b-f9f556a5484c admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'e8e72a1080a44ad3a0ef7918563a2a82', 'request_id': 'req-8803cbd8-6f5e-4464-842b-f9f556a5484c'} {{(pid=92056) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:01:21.442530 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-8803cbd8-6f5e-4464-842b-f9f556a5484c admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'e8e72a1080a44ad3a0ef7918563a2a82', 'request_id': 'req-8803cbd8-6f5e-4464-842b-f9f556a5484c'} {{(pid=92056) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:01:21.502189 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 1059/2108] 10.222.0.30 () {60 vars in 1482 bytes} [Mon May 23 13:01:21 2022] PUT /identity/v3/projects/ec74e160a3f74c11b1d25cedcd9be911/users/e54f105d7dcc4dc993ca024fe3a1faf1/roles/80fb22507c9a4ff09d44d8f12f6fff1a => generated 0 bytes in 251 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) May 23 13:01:21.505437 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11fe3e8d-aceb-4b77-bf0a-8a33a5d64678 None None] REQUEST_METHOD: `POST` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:01:21.505739 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11fe3e8d-aceb-4b77-bf0a-8a33a5d64678 None None] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:01:21.506067 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11fe3e8d-aceb-4b77-bf0a-8a33a5d64678 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:01:21.550661 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-8803cbd8-6f5e-4464-842b-f9f556a5484c admin admin] The token cache is being invalidate because project e8e72a1080a44ad3a0ef7918563a2a82 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=92056) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} May 23 13:01:21.550963 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-8803cbd8-6f5e-4464-842b-f9f556a5484c admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=92056) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:01:21.554271 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-8803cbd8-6f5e-4464-842b-f9f556a5484c admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=92056) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:01:21.558299 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 1050/2109] 10.222.0.30 () {58 vars in 1232 bytes} [Mon May 23 13:01:20 2022] DELETE /identity/v3/projects/e8e72a1080a44ad3a0ef7918563a2a82 => generated 0 bytes in 573 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) May 23 13:01:21.563425 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8803cbd8-6f5e-4464-842b-f9f556a5484c admin admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:01:21.565416 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.fernet_utils [None req-8803cbd8-6f5e-4464-842b-f9f556a5484c admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92056) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:01:21.599043 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.auth.core [None req-11fe3e8d-aceb-4b77-bf0a-8a33a5d64678 None None] MFA Rules not processed for user `e54f105d7dcc4dc993ca024fe3a1faf1`. Rule list: `[]` (Enabled: `True`). {{(pid=92057) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 13:01:21.602194 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.fernet_utils [None req-11fe3e8d-aceb-4b77-bf0a-8a33a5d64678 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92057) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:01:21.653012 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8803cbd8-6f5e-4464-842b-f9f556a5484c admin admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:01:21.753635 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd12b016-9810-4a6b-b56f-f74626839692 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:01:21.766678 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd12b016-9810-4a6b-b56f-f74626839692 admin admin] REQUEST_METHOD: `DELETE` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:01:21.766983 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd12b016-9810-4a6b-b56f-f74626839692 admin admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:01:21.767351 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd12b016-9810-4a6b-b56f-f74626839692 admin admin] PATH_INFO: `/v3/users/3209df37ffc64253a7f86ab40c581bb2` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:01:21.768796 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd12b016-9810-4a6b-b56f-f74626839692 admin admin] RBAC: Authorizing `identity:delete_user(user_id=3209df37ffc64253a7f86ab40c581bb2)` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:01:21.775998 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd12b016-9810-4a6b-b56f-f74626839692 admin admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:01:21.789904 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 1060/2110] 10.222.0.30 () {58 vars in 957 bytes} [Mon May 23 13:01:21 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 287 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) May 23 13:01:21.804127 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11fe3e8d-aceb-4b77-bf0a-8a33a5d64678 None None] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:01:21.806150 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.fernet_utils [None req-11fe3e8d-aceb-4b77-bf0a-8a33a5d64678 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92057) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:01:21.914583 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11fe3e8d-aceb-4b77-bf0a-8a33a5d64678 None None] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:01:21.921516 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-602dcc26-c744-4f0a-989f-efc99e163411 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1666097f8ed545c6b0e4809c93e4775a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9c389ae65aaa4542bb5a5c4fd0bacf5b', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:01:21.921516 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-602dcc26-c744-4f0a-989f-efc99e163411 service cinder] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:01:21.921516 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-602dcc26-c744-4f0a-989f-efc99e163411 service cinder] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:01:21.921516 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-602dcc26-c744-4f0a-989f-efc99e163411 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:01:21.944088 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-602dcc26-c744-4f0a-989f-efc99e163411 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:01:21.945714 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-602dcc26-c744-4f0a-989f-efc99e163411 service cinder] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:01:21.959947 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 1061/2111] 10.222.0.30 () {60 vars in 1365 bytes} [Mon May 23 13:01:21 2022] GET /identity/v3/auth/tokens => generated 3250 bytes in 158 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) May 23 13:01:22.050820 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-bd12b016-9810-4a6b-b56f-f74626839692 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '3209df37ffc64253a7f86ab40c581bb2'} {{(pid=92056) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:01:22.052792 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-bd12b016-9810-4a6b-b56f-f74626839692 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '3209df37ffc64253a7f86ab40c581bb2'} {{(pid=92056) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:01:22.168163 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-bd12b016-9810-4a6b-b56f-f74626839692 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '3209df37ffc64253a7f86ab40c581bb2'} {{(pid=92056) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:01:22.180384 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-bd12b016-9810-4a6b-b56f-f74626839692 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '3209df37ffc64253a7f86ab40c581bb2'} {{(pid=92056) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:01:22.187782 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-bd12b016-9810-4a6b-b56f-f74626839692 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '3209df37ffc64253a7f86ab40c581bb2'} {{(pid=92056) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:01:22.193998 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 1051/2112] 10.222.0.30 () {58 vars in 1223 bytes} [Mon May 23 13:01:21 2022] DELETE /identity/v3/users/3209df37ffc64253a7f86ab40c581bb2 => generated 0 bytes in 631 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) May 23 13:01:22.805905 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=95,keystone:INSERT=2,keystone:DELETE=3 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 13:01:22.876328 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-602dcc26-c744-4f0a-989f-efc99e163411 service cinder] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:01:22.896493 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-602dcc26-c744-4f0a-989f-efc99e163411 service cinder] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:01:23.008423 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6725258-cb60-4eef-9500-7dc9e4b278de admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:01:23.010071 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6725258-cb60-4eef-9500-7dc9e4b278de admin admin] REQUEST_METHOD: `DELETE` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:01:23.011594 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6725258-cb60-4eef-9500-7dc9e4b278de admin admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:01:23.012075 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6725258-cb60-4eef-9500-7dc9e4b278de admin admin] PATH_INFO: `/v3/projects/b319cf5737a54c86949cf39ba9f98fac` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:01:23.014544 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6725258-cb60-4eef-9500-7dc9e4b278de admin admin] RBAC: Authorizing `identity:delete_project(project_id=b319cf5737a54c86949cf39ba9f98fac)` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:01:23.016747 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6725258-cb60-4eef-9500-7dc9e4b278de admin admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:01:23.176541 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [None req-e6725258-cb60-4eef-9500-7dc9e4b278de admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'b319cf5737a54c86949cf39ba9f98fac', 'request_id': 'req-e6725258-cb60-4eef-9500-7dc9e4b278de'} {{(pid=92057) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:01:23.191399 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [None req-e6725258-cb60-4eef-9500-7dc9e4b278de admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'b319cf5737a54c86949cf39ba9f98fac', 'request_id': 'req-e6725258-cb60-4eef-9500-7dc9e4b278de'} {{(pid=92057) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:01:23.223527 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=100 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 13:01:23.308784 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [None req-e6725258-cb60-4eef-9500-7dc9e4b278de admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'b319cf5737a54c86949cf39ba9f98fac', 'request_id': 'req-e6725258-cb60-4eef-9500-7dc9e4b278de'} {{(pid=92057) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:01:23.330198 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [None req-e6725258-cb60-4eef-9500-7dc9e4b278de admin admin] The token cache is being invalidate because project b319cf5737a54c86949cf39ba9f98fac was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=92057) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} May 23 13:01:23.330495 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [None req-e6725258-cb60-4eef-9500-7dc9e4b278de admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=92057) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:01:23.331184 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [None req-e6725258-cb60-4eef-9500-7dc9e4b278de admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=92057) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:01:23.332998 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 1062/2113] 10.222.0.30 () {58 vars in 1232 bytes} [Mon May 23 13:01:22 2022] DELETE /identity/v3/projects/b319cf5737a54c86949cf39ba9f98fac => generated 0 bytes in 457 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) May 23 13:01:23.337986 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0452a2de-71d9-4b39-800d-bde7399b0e5d None None] REQUEST_METHOD: `POST` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:01:23.338836 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0452a2de-71d9-4b39-800d-bde7399b0e5d None None] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:01:23.339258 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0452a2de-71d9-4b39-800d-bde7399b0e5d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:01:23.381991 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.auth.core [None req-0452a2de-71d9-4b39-800d-bde7399b0e5d None None] MFA Rules not processed for user `0f7119f8fa5c4445977671140ded91a4`. Rule list: `[]` (Enabled: `True`). {{(pid=92056) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 13:01:23.393752 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.fernet_utils [None req-0452a2de-71d9-4b39-800d-bde7399b0e5d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92056) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:01:23.560936 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 1052/2114] 10.222.0.30 () {58 vars in 957 bytes} [Mon May 23 13:01:23 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 217 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) May 23 13:01:23.565388 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23ffb7c9-416a-4004-9f20-991c9d163463 None None] REQUEST_METHOD: `POST` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:01:23.574565 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23ffb7c9-416a-4004-9f20-991c9d163463 None None] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:01:23.574840 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23ffb7c9-416a-4004-9f20-991c9d163463 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:01:23.615356 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.auth.core [None req-23ffb7c9-416a-4004-9f20-991c9d163463 None None] MFA Rules not processed for user `0f7119f8fa5c4445977671140ded91a4`. Rule list: `[]` (Enabled: `True`). {{(pid=92057) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 13:01:23.619468 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.fernet_utils [None req-23ffb7c9-416a-4004-9f20-991c9d163463 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92057) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:01:23.650345 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 1063/2115] 10.222.0.30 () {58 vars in 957 bytes} [Mon May 23 13:01:23 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 88 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) May 23 13:01:23.654374 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0452a2de-71d9-4b39-800d-bde7399b0e5d None None] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:01:23.675434 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0452a2de-71d9-4b39-800d-bde7399b0e5d None None] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:01:23.681106 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8630d6ee-79b2-4be3-a4a6-856c020bb844 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:01:23.682067 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8630d6ee-79b2-4be3-a4a6-856c020bb844 admin admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:01:23.683097 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8630d6ee-79b2-4be3-a4a6-856c020bb844 admin admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:01:23.683670 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8630d6ee-79b2-4be3-a4a6-856c020bb844 admin admin] PATH_INFO: `/v3/domains` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:01:23.684999 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8630d6ee-79b2-4be3-a4a6-856c020bb844 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=92056) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} May 23 13:01:23.685531 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8630d6ee-79b2-4be3-a4a6-856c020bb844 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:01:23.687777 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8630d6ee-79b2-4be3-a4a6-856c020bb844 admin admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:01:23.705650 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 1053/2116] 10.222.0.30 () {58 vars in 1152 bytes} [Mon May 23 13:01:23 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 52 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) May 23 13:01:23.710893 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23ffb7c9-416a-4004-9f20-991c9d163463 None None] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:01:23.722832 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23ffb7c9-416a-4004-9f20-991c9d163463 None None] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:01:23.740618 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bde7ef33-85ef-4cf8-b8e7-5d16228a9f03 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:01:23.741282 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bde7ef33-85ef-4cf8-b8e7-5d16228a9f03 admin admin] REQUEST_METHOD: `POST` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:01:23.741789 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bde7ef33-85ef-4cf8-b8e7-5d16228a9f03 admin admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:01:23.742221 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bde7ef33-85ef-4cf8-b8e7-5d16228a9f03 admin admin] PATH_INFO: `/v3/projects` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:01:23.743209 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bde7ef33-85ef-4cf8-b8e7-5d16228a9f03 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:01:23.746317 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bde7ef33-85ef-4cf8-b8e7-5d16228a9f03 admin admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:01:23.896913 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 1064/2117] 10.222.0.30 () {60 vars in 1152 bytes} [Mon May 23 13:01:23 2022] POST /identity/v3/projects => generated 373 bytes in 187 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) May 23 13:01:23.901023 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8630d6ee-79b2-4be3-a4a6-856c020bb844 admin admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:01:23.924672 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8630d6ee-79b2-4be3-a4a6-856c020bb844 admin admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:01:24.028523 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d01eba69-2fb0-4365-8b79-df218efd9978 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:01:24.029281 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d01eba69-2fb0-4365-8b79-df218efd9978 admin admin] REQUEST_METHOD: `POST` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:01:24.029576 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d01eba69-2fb0-4365-8b79-df218efd9978 admin admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:01:24.029851 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d01eba69-2fb0-4365-8b79-df218efd9978 admin admin] PATH_INFO: `/v3/users` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:01:24.030745 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d01eba69-2fb0-4365-8b79-df218efd9978 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:01:24.033008 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d01eba69-2fb0-4365-8b79-df218efd9978 admin admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:01:24.170464 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 1054/2118] 10.222.0.30 () {60 vars in 1143 bytes} [Mon May 23 13:01:23 2022] POST /identity/v3/users => generated 287 bytes in 268 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) May 23 13:01:24.172978 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bde7ef33-85ef-4cf8-b8e7-5d16228a9f03 admin admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:01:24.185402 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bde7ef33-85ef-4cf8-b8e7-5d16228a9f03 admin admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:01:24.190410 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9bb2e121-ca32-408c-bf7f-e725eb16a34a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:01:24.191073 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bb2e121-ca32-408c-bf7f-e725eb16a34a admin admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:01:24.195893 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bb2e121-ca32-408c-bf7f-e725eb16a34a admin admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:01:24.195893 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bb2e121-ca32-408c-bf7f-e725eb16a34a admin admin] PATH_INFO: `/v3/roles` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:01:24.195893 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9bb2e121-ca32-408c-bf7f-e725eb16a34a admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:01:24.195893 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9bb2e121-ca32-408c-bf7f-e725eb16a34a admin admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:01:24.206757 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 1065/2119] 10.222.0.30 () {58 vars in 1121 bytes} [Mon May 23 13:01:24 2022] GET /identity/v3/roles => generated 1420 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) May 23 13:01:24.210593 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d01eba69-2fb0-4365-8b79-df218efd9978 admin admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:01:24.212011 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=93,keystone:INSERT=2,keystone:DELETE=5 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 13:01:24.222719 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d01eba69-2fb0-4365-8b79-df218efd9978 admin admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:01:24.231091 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e6068d9-cbc0-45c5-aca6-f00ac9bae4a5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:01:24.235661 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e6068d9-cbc0-45c5-aca6-f00ac9bae4a5 admin admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:01:24.235951 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e6068d9-cbc0-45c5-aca6-f00ac9bae4a5 admin admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:01:24.236229 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e6068d9-cbc0-45c5-aca6-f00ac9bae4a5 admin admin] PATH_INFO: `/v3/roles` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:01:24.236985 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e6068d9-cbc0-45c5-aca6-f00ac9bae4a5 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:01:24.241506 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e6068d9-cbc0-45c5-aca6-f00ac9bae4a5 admin admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:01:24.279475 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 1055/2120] 10.222.0.30 () {58 vars in 1121 bytes} [Mon May 23 13:01:24 2022] GET /identity/v3/roles => generated 1420 bytes in 65 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) May 23 13:01:24.283651 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9bb2e121-ca32-408c-bf7f-e725eb16a34a admin admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:01:24.295416 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9bb2e121-ca32-408c-bf7f-e725eb16a34a admin admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:01:24.309231 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9dbb25e-0e65-4880-8f70-a490845b1397 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:01:24.309231 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9dbb25e-0e65-4880-8f70-a490845b1397 admin admin] REQUEST_METHOD: `PUT` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:01:24.309231 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9dbb25e-0e65-4880-8f70-a490845b1397 admin admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:01:24.309574 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9dbb25e-0e65-4880-8f70-a490845b1397 admin admin] PATH_INFO: `/v3/projects/841ef16df18f4714ba1a61a24c4fa366/users/bd105c42e14b430c81d058d4dac15ff4/roles/80fb22507c9a4ff09d44d8f12f6fff1a` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:01:24.332937 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9dbb25e-0e65-4880-8f70-a490845b1397 admin admin] RBAC: Authorizing `identity:create_grant(project_id=841ef16df18f4714ba1a61a24c4fa366, user_id=bd105c42e14b430c81d058d4dac15ff4, role_id=80fb22507c9a4ff09d44d8f12f6fff1a)` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:01:24.341498 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9dbb25e-0e65-4880-8f70-a490845b1397 admin admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:01:24.445929 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 1066/2121] 10.222.0.30 () {60 vars in 1482 bytes} [Mon May 23 13:01:24 2022] PUT /identity/v3/projects/841ef16df18f4714ba1a61a24c4fa366/users/bd105c42e14b430c81d058d4dac15ff4/roles/80fb22507c9a4ff09d44d8f12f6fff1a => generated 0 bytes in 159 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) May 23 13:01:24.454024 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b17072da-d6e2-4fc7-8b4f-42cd9bcadc39 None None] REQUEST_METHOD: `POST` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:01:24.454563 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b17072da-d6e2-4fc7-8b4f-42cd9bcadc39 None None] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:01:24.454688 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b17072da-d6e2-4fc7-8b4f-42cd9bcadc39 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:01:24.499412 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.auth.core [None req-b17072da-d6e2-4fc7-8b4f-42cd9bcadc39 None None] MFA Rules not processed for user `bd105c42e14b430c81d058d4dac15ff4`. Rule list: `[]` (Enabled: `True`). {{(pid=92056) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 13:01:24.500798 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.fernet_utils [None req-b17072da-d6e2-4fc7-8b4f-42cd9bcadc39 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92056) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:01:24.709437 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 1056/2122] 10.222.0.30 () {58 vars in 957 bytes} [Mon May 23 13:01:24 2022] POST /identity/v3/auth/tokens => generated 3240 bytes in 257 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) May 23 13:01:24.714449 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9dbb25e-0e65-4880-8f70-a490845b1397 admin admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:01:24.894381 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9dbb25e-0e65-4880-8f70-a490845b1397 admin admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:01:24.902265 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4586124-9420-4a5a-a305-a098cdc5ad07 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:01:24.903448 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4586124-9420-4a5a-a305-a098cdc5ad07 admin admin] REQUEST_METHOD: `POST` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:01:24.904257 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4586124-9420-4a5a-a305-a098cdc5ad07 admin admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:01:24.905016 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4586124-9420-4a5a-a305-a098cdc5ad07 admin admin] PATH_INFO: `/v3/projects` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:01:24.906551 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4586124-9420-4a5a-a305-a098cdc5ad07 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:01:24.909262 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4586124-9420-4a5a-a305-a098cdc5ad07 admin admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:01:25.068841 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 1067/2123] 10.222.0.30 () {60 vars in 1152 bytes} [Mon May 23 13:01:24 2022] POST /identity/v3/projects => generated 373 bytes in 353 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) May 23 13:01:25.070311 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b17072da-d6e2-4fc7-8b4f-42cd9bcadc39 None None] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:01:25.083945 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b17072da-d6e2-4fc7-8b4f-42cd9bcadc39 None None] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:01:25.159390 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fbbc7b5c-dbb4-466a-875c-91043a8c36ef admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:01:25.160404 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbbc7b5c-dbb4-466a-875c-91043a8c36ef admin admin] REQUEST_METHOD: `POST` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:01:25.161041 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbbc7b5c-dbb4-466a-875c-91043a8c36ef admin admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:01:25.161505 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbbc7b5c-dbb4-466a-875c-91043a8c36ef admin admin] PATH_INFO: `/v3/users` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:01:25.162449 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fbbc7b5c-dbb4-466a-875c-91043a8c36ef admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:01:25.165320 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fbbc7b5c-dbb4-466a-875c-91043a8c36ef admin admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:01:25.297120 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 1057/2124] 10.222.0.30 () {60 vars in 1143 bytes} [Mon May 23 13:01:25 2022] POST /identity/v3/users => generated 293 bytes in 227 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) May 23 13:01:25.301412 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4586124-9420-4a5a-a305-a098cdc5ad07 admin admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:01:25.324780 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4586124-9420-4a5a-a305-a098cdc5ad07 admin admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:01:25.330604 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1349d0f-a563-4123-bd8f-319760846389 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:01:25.331867 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1349d0f-a563-4123-bd8f-319760846389 admin admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:01:25.332367 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1349d0f-a563-4123-bd8f-319760846389 admin admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:01:25.332877 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1349d0f-a563-4123-bd8f-319760846389 admin admin] PATH_INFO: `/v3/roles` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:01:25.333738 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1349d0f-a563-4123-bd8f-319760846389 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:01:25.335507 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1349d0f-a563-4123-bd8f-319760846389 admin admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:01:25.364288 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 1068/2125] 10.222.0.30 () {58 vars in 1121 bytes} [Mon May 23 13:01:25 2022] GET /identity/v3/roles => generated 1420 bytes in 63 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) May 23 13:01:25.368591 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fbbc7b5c-dbb4-466a-875c-91043a8c36ef admin admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:01:25.390289 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fbbc7b5c-dbb4-466a-875c-91043a8c36ef admin admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:01:25.399203 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45a9d2b6-5c35-452a-8b9f-6af0945c8a4b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:01:25.400416 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45a9d2b6-5c35-452a-8b9f-6af0945c8a4b admin admin] REQUEST_METHOD: `PUT` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:01:25.400852 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45a9d2b6-5c35-452a-8b9f-6af0945c8a4b admin admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:01:25.403201 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45a9d2b6-5c35-452a-8b9f-6af0945c8a4b admin admin] PATH_INFO: `/v3/projects/19c37ded87a845558f6800794483ed3c/users/5b79e59244cd415588d420abfb652466/roles/bc3c3c9d654940cbaa5177a6be23d1bc` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:01:25.436192 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45a9d2b6-5c35-452a-8b9f-6af0945c8a4b admin admin] RBAC: Authorizing `identity:create_grant(project_id=19c37ded87a845558f6800794483ed3c, user_id=5b79e59244cd415588d420abfb652466, role_id=bc3c3c9d654940cbaa5177a6be23d1bc)` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:01:25.440092 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45a9d2b6-5c35-452a-8b9f-6af0945c8a4b admin admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:01:25.480520 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 13:01:25.536546 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 1058/2126] 10.222.0.30 () {60 vars in 1482 bytes} [Mon May 23 13:01:25 2022] PUT /identity/v3/projects/19c37ded87a845558f6800794483ed3c/users/5b79e59244cd415588d420abfb652466/roles/bc3c3c9d654940cbaa5177a6be23d1bc => generated 0 bytes in 168 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) May 23 13:01:25.542523 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1349d0f-a563-4123-bd8f-319760846389 admin admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:01:25.553936 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1349d0f-a563-4123-bd8f-319760846389 admin admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:01:25.705926 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe946056-257a-408e-b541-a536127593ab admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:01:25.707383 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe946056-257a-408e-b541-a536127593ab admin admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:01:25.707886 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe946056-257a-408e-b541-a536127593ab admin admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:01:25.712191 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe946056-257a-408e-b541-a536127593ab admin admin] PATH_INFO: `/v3/roles` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:01:25.712191 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe946056-257a-408e-b541-a536127593ab admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:01:25.712191 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe946056-257a-408e-b541-a536127593ab admin admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:01:25.733030 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 1069/2127] 10.222.0.30 () {58 vars in 1121 bytes} [Mon May 23 13:01:25 2022] GET /identity/v3/roles => generated 1420 bytes in 188 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) May 23 13:01:25.734337 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45a9d2b6-5c35-452a-8b9f-6af0945c8a4b admin admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:01:25.753185 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45a9d2b6-5c35-452a-8b9f-6af0945c8a4b admin admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:01:25.759784 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2699ff4-e88c-47d9-9f1f-c2c7950836cb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:01:25.763157 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2699ff4-e88c-47d9-9f1f-c2c7950836cb admin admin] REQUEST_METHOD: `PUT` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:01:25.763940 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2699ff4-e88c-47d9-9f1f-c2c7950836cb admin admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:01:25.764234 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2699ff4-e88c-47d9-9f1f-c2c7950836cb admin admin] PATH_INFO: `/v3/projects/19c37ded87a845558f6800794483ed3c/users/5b79e59244cd415588d420abfb652466/roles/bc3c3c9d654940cbaa5177a6be23d1bc` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:01:25.766606 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2699ff4-e88c-47d9-9f1f-c2c7950836cb admin admin] RBAC: Authorizing `identity:create_grant(project_id=19c37ded87a845558f6800794483ed3c, user_id=5b79e59244cd415588d420abfb652466, role_id=bc3c3c9d654940cbaa5177a6be23d1bc)` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:01:25.773511 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2699ff4-e88c-47d9-9f1f-c2c7950836cb admin admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:01:25.788594 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 1059/2128] 10.222.0.30 () {60 vars in 1482 bytes} [Mon May 23 13:01:25 2022] PUT /identity/v3/projects/19c37ded87a845558f6800794483ed3c/users/5b79e59244cd415588d420abfb652466/roles/bc3c3c9d654940cbaa5177a6be23d1bc => generated 0 bytes in 55 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) May 23 13:01:25.798996 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0525dc96-6804-4ed4-a222-73760d976e3d None None] REQUEST_METHOD: `POST` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:01:25.799297 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0525dc96-6804-4ed4-a222-73760d976e3d None None] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:01:25.799607 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0525dc96-6804-4ed4-a222-73760d976e3d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:01:25.832908 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.auth.core [None req-0525dc96-6804-4ed4-a222-73760d976e3d None None] MFA Rules not processed for user `5b79e59244cd415588d420abfb652466`. Rule list: `[]` (Enabled: `True`). {{(pid=92057) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 13:01:25.834352 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.fernet_utils [None req-0525dc96-6804-4ed4-a222-73760d976e3d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92057) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:01:25.929518 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 1070/2129] 10.222.0.30 () {58 vars in 957 bytes} [Mon May 23 13:01:25 2022] POST /identity/v3/auth/tokens => generated 3307 bytes in 132 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) May 23 13:01:25.942872 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2699ff4-e88c-47d9-9f1f-c2c7950836cb admin admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:01:25.944569 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.fernet_utils [None req-c2699ff4-e88c-47d9-9f1f-c2c7950836cb admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92056) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:01:26.096045 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2699ff4-e88c-47d9-9f1f-c2c7950836cb admin admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:01:26.111067 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79edf9c3-b489-4fac-a698-ff48cd36a895 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1666097f8ed545c6b0e4809c93e4775a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9c389ae65aaa4542bb5a5c4fd0bacf5b', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:01:26.116311 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79edf9c3-b489-4fac-a698-ff48cd36a895 service cinder] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:01:26.119959 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79edf9c3-b489-4fac-a698-ff48cd36a895 service cinder] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:01:26.119959 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79edf9c3-b489-4fac-a698-ff48cd36a895 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:01:26.145237 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0525dc96-6804-4ed4-a222-73760d976e3d None None] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:01:26.146282 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79edf9c3-b489-4fac-a698-ff48cd36a895 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:01:26.147339 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.fernet_utils [None req-0525dc96-6804-4ed4-a222-73760d976e3d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92057) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:01:26.153794 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79edf9c3-b489-4fac-a698-ff48cd36a895 service cinder] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:01:26.168617 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 1060/2130] 10.222.0.30 () {60 vars in 1365 bytes} [Mon May 23 13:01:25 2022] GET /identity/v3/auth/tokens => generated 3307 bytes in 226 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) May 23 13:01:26.306471 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0525dc96-6804-4ed4-a222-73760d976e3d None None] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:01:26.314966 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8af20259-5652-4782-85c9-d90ee3e5b7eb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:01:26.323548 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8af20259-5652-4782-85c9-d90ee3e5b7eb admin admin] REQUEST_METHOD: `DELETE` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:01:26.324039 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8af20259-5652-4782-85c9-d90ee3e5b7eb admin admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:01:26.324468 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8af20259-5652-4782-85c9-d90ee3e5b7eb admin admin] PATH_INFO: `/v3/users/0f675d0e2c8c42b4883586446a4e244e` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:01:26.327461 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8af20259-5652-4782-85c9-d90ee3e5b7eb admin admin] RBAC: Authorizing `identity:delete_user(user_id=0f675d0e2c8c42b4883586446a4e244e)` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:01:26.329902 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8af20259-5652-4782-85c9-d90ee3e5b7eb admin admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:01:26.442519 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79edf9c3-b489-4fac-a698-ff48cd36a895 service cinder] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:01:26.445565 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.fernet_utils [None req-79edf9c3-b489-4fac-a698-ff48cd36a895 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92056) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:01:26.472946 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79edf9c3-b489-4fac-a698-ff48cd36a895 service cinder] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:01:26.480675 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-abb93f14-9395-498d-bd4a-7459ec3f7c93 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:01:26.482681 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abb93f14-9395-498d-bd4a-7459ec3f7c93 admin admin] REQUEST_METHOD: `DELETE` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:01:26.483112 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abb93f14-9395-498d-bd4a-7459ec3f7c93 admin admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:01:26.483538 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abb93f14-9395-498d-bd4a-7459ec3f7c93 admin admin] PATH_INFO: `/v3/users/a9b80726b3f14828b8815e30fc7e13c5` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:01:26.505589 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-abb93f14-9395-498d-bd4a-7459ec3f7c93 admin admin] RBAC: Authorizing `identity:delete_user(user_id=a9b80726b3f14828b8815e30fc7e13c5)` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:01:26.514407 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-abb93f14-9395-498d-bd4a-7459ec3f7c93 admin admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:01:26.516897 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=94,keystone:DELETE=5,keystone:INSERT=1 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 13:01:26.614425 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [None req-8af20259-5652-4782-85c9-d90ee3e5b7eb admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '0f675d0e2c8c42b4883586446a4e244e'} {{(pid=92057) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:01:26.711132 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [None req-8af20259-5652-4782-85c9-d90ee3e5b7eb admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '0f675d0e2c8c42b4883586446a4e244e'} {{(pid=92057) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:01:26.730880 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [None req-8af20259-5652-4782-85c9-d90ee3e5b7eb admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '0f675d0e2c8c42b4883586446a4e244e'} {{(pid=92057) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:01:26.732891 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [None req-8af20259-5652-4782-85c9-d90ee3e5b7eb admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '0f675d0e2c8c42b4883586446a4e244e'} {{(pid=92057) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:01:26.754584 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [None req-8af20259-5652-4782-85c9-d90ee3e5b7eb admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '0f675d0e2c8c42b4883586446a4e244e'} {{(pid=92057) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:01:26.766924 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 1071/2131] 10.222.0.30 () {58 vars in 1223 bytes} [Mon May 23 13:01:26 2022] DELETE /identity/v3/users/0f675d0e2c8c42b4883586446a4e244e => generated 0 bytes in 622 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) May 23 13:01:26.768879 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e611fd6-1a6a-46f2-b1e2-7b244ea8cf38 None None] REQUEST_METHOD: `POST` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:01:26.769406 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e611fd6-1a6a-46f2-b1e2-7b244ea8cf38 None None] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:01:26.769936 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e611fd6-1a6a-46f2-b1e2-7b244ea8cf38 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:01:26.797718 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.auth.core [None req-4e611fd6-1a6a-46f2-b1e2-7b244ea8cf38 None None] MFA Rules not processed for user `a6a7f78a024b43a6b89f4e2fe5c6c28b`. Rule list: `[]` (Enabled: `True`). {{(pid=92057) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 13:01:26.799229 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.fernet_utils [None req-4e611fd6-1a6a-46f2-b1e2-7b244ea8cf38 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92057) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:01:26.825372 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-abb93f14-9395-498d-bd4a-7459ec3f7c93 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'a9b80726b3f14828b8815e30fc7e13c5'} {{(pid=92056) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:01:26.826110 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-abb93f14-9395-498d-bd4a-7459ec3f7c93 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'a9b80726b3f14828b8815e30fc7e13c5'} {{(pid=92056) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:01:26.868733 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 1072/2132] 10.222.0.30 () {58 vars in 957 bytes} [Mon May 23 13:01:26 2022] POST /identity/v3/auth/tokens => generated 3311 bytes in 101 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) May 23 13:01:26.871409 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-320e62cb-e967-421c-90cb-a0b8d0ee5412 None None] REQUEST_METHOD: `POST` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:01:26.871791 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-320e62cb-e967-421c-90cb-a0b8d0ee5412 None None] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:01:26.872319 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-320e62cb-e967-421c-90cb-a0b8d0ee5412 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:01:26.883002 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=81,keystone:DELETE=17,keystone:INSERT=2 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 13:01:26.893211 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.auth.core [None req-320e62cb-e967-421c-90cb-a0b8d0ee5412 None None] MFA Rules not processed for user `0f7119f8fa5c4445977671140ded91a4`. Rule list: `[]` (Enabled: `True`). {{(pid=92057) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 13:01:26.894472 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.fernet_utils [None req-320e62cb-e967-421c-90cb-a0b8d0ee5412 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92057) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:01:26.903953 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-abb93f14-9395-498d-bd4a-7459ec3f7c93 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'a9b80726b3f14828b8815e30fc7e13c5'} {{(pid=92056) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:01:26.935530 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-abb93f14-9395-498d-bd4a-7459ec3f7c93 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'a9b80726b3f14828b8815e30fc7e13c5'} {{(pid=92056) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:01:26.954079 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-abb93f14-9395-498d-bd4a-7459ec3f7c93 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'a9b80726b3f14828b8815e30fc7e13c5'} {{(pid=92056) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:01:26.964504 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 1073/2133] 10.222.0.30 () {58 vars in 957 bytes} [Mon May 23 13:01:26 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 95 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) May 23 13:01:26.965668 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-320e62cb-e967-421c-90cb-a0b8d0ee5412 None None] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:01:26.970148 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 1061/2134] 10.222.0.30 () {58 vars in 1223 bytes} [Mon May 23 13:01:26 2022] DELETE /identity/v3/users/a9b80726b3f14828b8815e30fc7e13c5 => generated 0 bytes in 526 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) May 23 13:01:26.971181 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.fernet_utils [None req-320e62cb-e967-421c-90cb-a0b8d0ee5412 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92057) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:01:26.978251 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-abb93f14-9395-498d-bd4a-7459ec3f7c93 admin admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:01:26.980662 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.fernet_utils [None req-abb93f14-9395-498d-bd4a-7459ec3f7c93 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92056) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:01:27.078503 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-abb93f14-9395-498d-bd4a-7459ec3f7c93 admin admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:01:27.083866 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04e27e3b-9c71-43fd-a62d-18b3092b96f4 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2eb83ca38e79449095f3a2f947bbb06a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9c389ae65aaa4542bb5a5c4fd0bacf5b', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:01:27.085536 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04e27e3b-9c71-43fd-a62d-18b3092b96f4 service neutron] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:01:27.086155 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04e27e3b-9c71-43fd-a62d-18b3092b96f4 service neutron] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:01:27.086605 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04e27e3b-9c71-43fd-a62d-18b3092b96f4 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:01:27.113957 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-320e62cb-e967-421c-90cb-a0b8d0ee5412 None None] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:01:27.188399 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04e27e3b-9c71-43fd-a62d-18b3092b96f4 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:01:27.189953 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04e27e3b-9c71-43fd-a62d-18b3092b96f4 service neutron] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:01:27.202092 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 1062/2135] 10.222.0.30 () {60 vars in 1365 bytes} [Mon May 23 13:01:26 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 224 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) May 23 13:01:27.212292 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ae59e8c-55e6-473c-8a1b-a7fcbaf0d47e service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cbe9931d94d44cc7ace0e83a7e5015fc', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9c389ae65aaa4542bb5a5c4fd0bacf5b', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'member', 'reader', 'service', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:01:27.212292 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ae59e8c-55e6-473c-8a1b-a7fcbaf0d47e service nova] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:01:27.212292 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ae59e8c-55e6-473c-8a1b-a7fcbaf0d47e service nova] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:01:27.212292 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ae59e8c-55e6-473c-8a1b-a7fcbaf0d47e service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:01:27.214552 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.fernet_utils [None req-7ae59e8c-55e6-473c-8a1b-a7fcbaf0d47e service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92057) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:01:27.284884 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04e27e3b-9c71-43fd-a62d-18b3092b96f4 service neutron] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:01:27.286377 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.fernet_utils [None req-04e27e3b-9c71-43fd-a62d-18b3092b96f4 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92056) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:01:27.303761 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04e27e3b-9c71-43fd-a62d-18b3092b96f4 service neutron] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:01:27.305605 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ae59e8c-55e6-473c-8a1b-a7fcbaf0d47e service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:01:27.307726 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ae59e8c-55e6-473c-8a1b-a7fcbaf0d47e service nova] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:01:27.316896 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-356c5b75-980a-48d5-9d59-90b933ed4174 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:01:27.318002 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-356c5b75-980a-48d5-9d59-90b933ed4174 admin admin] REQUEST_METHOD: `DELETE` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:01:27.318907 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-356c5b75-980a-48d5-9d59-90b933ed4174 admin admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:01:27.319335 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-356c5b75-980a-48d5-9d59-90b933ed4174 admin admin] PATH_INFO: `/v3/projects/9e3c84deee9a435a9c1e72e8408c18b4` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:01:27.320914 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-356c5b75-980a-48d5-9d59-90b933ed4174 admin admin] RBAC: Authorizing `identity:delete_project(project_id=9e3c84deee9a435a9c1e72e8408c18b4)` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:01:27.324104 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-356c5b75-980a-48d5-9d59-90b933ed4174 admin admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:01:27.326085 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 1074/2136] 10.222.0.30 () {60 vars in 1365 bytes} [Mon May 23 13:01:26 2022] GET /identity/v3/auth/tokens => generated 3246 bytes in 361 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) May 23 13:01:27.489601 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ae59e8c-55e6-473c-8a1b-a7fcbaf0d47e service nova] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:01:27.491667 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.fernet_utils [None req-7ae59e8c-55e6-473c-8a1b-a7fcbaf0d47e service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92057) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:01:27.508102 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ae59e8c-55e6-473c-8a1b-a7fcbaf0d47e service nova] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:01:27.510625 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-356c5b75-980a-48d5-9d59-90b933ed4174 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '9e3c84deee9a435a9c1e72e8408c18b4', 'request_id': 'req-356c5b75-980a-48d5-9d59-90b933ed4174'} {{(pid=92056) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:01:27.569261 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7df2b000-ed7a-446b-a374-e691400c57c5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:01:27.570258 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7df2b000-ed7a-446b-a374-e691400c57c5 admin admin] REQUEST_METHOD: `DELETE` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:01:27.570542 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7df2b000-ed7a-446b-a374-e691400c57c5 admin admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:01:27.570814 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7df2b000-ed7a-446b-a374-e691400c57c5 admin admin] PATH_INFO: `/v3/users/bd105c42e14b430c81d058d4dac15ff4` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:01:27.572649 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7df2b000-ed7a-446b-a374-e691400c57c5 admin admin] RBAC: Authorizing `identity:delete_user(user_id=bd105c42e14b430c81d058d4dac15ff4)` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:01:27.583376 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-356c5b75-980a-48d5-9d59-90b933ed4174 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '9e3c84deee9a435a9c1e72e8408c18b4', 'request_id': 'req-356c5b75-980a-48d5-9d59-90b933ed4174'} {{(pid=92056) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:01:27.591257 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7df2b000-ed7a-446b-a374-e691400c57c5 admin admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:01:27.598473 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-356c5b75-980a-48d5-9d59-90b933ed4174 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '9e3c84deee9a435a9c1e72e8408c18b4', 'request_id': 'req-356c5b75-980a-48d5-9d59-90b933ed4174'} {{(pid=92056) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:01:27.614974 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-356c5b75-980a-48d5-9d59-90b933ed4174 admin admin] The token cache is being invalidate because project 9e3c84deee9a435a9c1e72e8408c18b4 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=92056) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} May 23 13:01:27.615404 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-356c5b75-980a-48d5-9d59-90b933ed4174 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=92056) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:01:27.617919 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-356c5b75-980a-48d5-9d59-90b933ed4174 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=92056) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:01:27.622934 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 1063/2137] 10.222.0.30 () {58 vars in 1232 bytes} [Mon May 23 13:01:27 2022] DELETE /identity/v3/projects/9e3c84deee9a435a9c1e72e8408c18b4 => generated 0 bytes in 335 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) May 23 13:01:27.624498 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17f77b9d-84a8-4826-a7e3-dfd11f64becf None None] REQUEST_METHOD: `POST` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:01:27.624955 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17f77b9d-84a8-4826-a7e3-dfd11f64becf None None] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:01:27.625344 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17f77b9d-84a8-4826-a7e3-dfd11f64becf None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:01:27.649045 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.auth.core [None req-17f77b9d-84a8-4826-a7e3-dfd11f64becf None None] MFA Rules not processed for user `0f7119f8fa5c4445977671140ded91a4`. Rule list: `[]` (Enabled: `True`). {{(pid=92056) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 13:01:27.650457 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.fernet_utils [None req-17f77b9d-84a8-4826-a7e3-dfd11f64becf None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92056) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:01:27.659983 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 1064/2138] 10.222.0.30 () {58 vars in 957 bytes} [Mon May 23 13:01:27 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 36 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) May 23 13:01:27.667685 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-211e458e-53fe-4986-9adf-eac544a2360d None None] REQUEST_METHOD: `POST` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:01:27.667981 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-211e458e-53fe-4986-9adf-eac544a2360d None None] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:01:27.668267 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-211e458e-53fe-4986-9adf-eac544a2360d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:01:27.688318 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.auth.core [None req-211e458e-53fe-4986-9adf-eac544a2360d None None] MFA Rules not processed for user `0f7119f8fa5c4445977671140ded91a4`. Rule list: `[]` (Enabled: `True`). {{(pid=92056) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 13:01:27.689565 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.fernet_utils [None req-211e458e-53fe-4986-9adf-eac544a2360d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92056) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:01:27.707769 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 1065/2139] 10.222.0.30 () {58 vars in 957 bytes} [Mon May 23 13:01:27 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 41 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) May 23 13:01:27.712551 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-211e458e-53fe-4986-9adf-eac544a2360d None None] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:01:27.724882 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-211e458e-53fe-4986-9adf-eac544a2360d None None] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:01:27.737183 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c1938b6-995f-4f29-888e-0170d8063252 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:01:27.738041 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c1938b6-995f-4f29-888e-0170d8063252 admin admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:01:27.738425 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c1938b6-995f-4f29-888e-0170d8063252 admin admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:01:27.738798 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c1938b6-995f-4f29-888e-0170d8063252 admin admin] PATH_INFO: `/v3/domains` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:01:27.739702 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c1938b6-995f-4f29-888e-0170d8063252 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=92056) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} May 23 13:01:27.740101 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c1938b6-995f-4f29-888e-0170d8063252 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:01:27.741852 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c1938b6-995f-4f29-888e-0170d8063252 admin admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:01:27.765871 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 1066/2140] 10.222.0.30 () {58 vars in 1152 bytes} [Mon May 23 13:01:27 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 54 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) May 23 13:01:27.767032 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c1938b6-995f-4f29-888e-0170d8063252 admin admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:01:27.769758 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.fernet_utils [None req-1c1938b6-995f-4f29-888e-0170d8063252 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92056) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:01:27.797138 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c1938b6-995f-4f29-888e-0170d8063252 admin admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:01:27.804089 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [None req-7df2b000-ed7a-446b-a374-e691400c57c5 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'bd105c42e14b430c81d058d4dac15ff4'} {{(pid=92057) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:01:27.811962 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ee97f17-2d74-4374-99ba-98a3502188dd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:01:27.813009 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ee97f17-2d74-4374-99ba-98a3502188dd admin admin] REQUEST_METHOD: `DELETE` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:01:27.813356 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ee97f17-2d74-4374-99ba-98a3502188dd admin admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:01:27.813696 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ee97f17-2d74-4374-99ba-98a3502188dd admin admin] PATH_INFO: `/v3/projects/95675a44f30b42ef94a8afb1b9167f8c` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:01:27.816465 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ee97f17-2d74-4374-99ba-98a3502188dd admin admin] RBAC: Authorizing `identity:delete_project(project_id=95675a44f30b42ef94a8afb1b9167f8c)` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:01:27.818929 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ee97f17-2d74-4374-99ba-98a3502188dd admin admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:01:27.893820 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [None req-7df2b000-ed7a-446b-a374-e691400c57c5 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'bd105c42e14b430c81d058d4dac15ff4'} {{(pid=92057) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:01:27.901485 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [None req-7df2b000-ed7a-446b-a374-e691400c57c5 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'bd105c42e14b430c81d058d4dac15ff4'} {{(pid=92057) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:01:27.902271 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [None req-7df2b000-ed7a-446b-a374-e691400c57c5 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'bd105c42e14b430c81d058d4dac15ff4'} {{(pid=92057) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:01:27.914271 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [None req-7df2b000-ed7a-446b-a374-e691400c57c5 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'bd105c42e14b430c81d058d4dac15ff4'} {{(pid=92057) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:01:27.924517 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 1075/2141] 10.222.0.30 () {58 vars in 1223 bytes} [Mon May 23 13:01:27 2022] DELETE /identity/v3/users/bd105c42e14b430c81d058d4dac15ff4 => generated 0 bytes in 433 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) May 23 13:01:27.924517 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7df2b000-ed7a-446b-a374-e691400c57c5 admin admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:01:27.924517 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.fernet_utils [None req-7df2b000-ed7a-446b-a374-e691400c57c5 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92057) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:01:27.945087 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-1ee97f17-2d74-4374-99ba-98a3502188dd admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '95675a44f30b42ef94a8afb1b9167f8c', 'request_id': 'req-1ee97f17-2d74-4374-99ba-98a3502188dd'} {{(pid=92056) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:01:27.999079 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7df2b000-ed7a-446b-a374-e691400c57c5 admin admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:01:28.034902 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-1ee97f17-2d74-4374-99ba-98a3502188dd admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '95675a44f30b42ef94a8afb1b9167f8c', 'request_id': 'req-1ee97f17-2d74-4374-99ba-98a3502188dd'} {{(pid=92056) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:01:28.055946 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-1ee97f17-2d74-4374-99ba-98a3502188dd admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '95675a44f30b42ef94a8afb1b9167f8c', 'request_id': 'req-1ee97f17-2d74-4374-99ba-98a3502188dd'} {{(pid=92056) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:01:28.066679 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6d8d8b1-5ede-4a08-b462-bcd20af20374 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:01:28.067367 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6d8d8b1-5ede-4a08-b462-bcd20af20374 admin admin] REQUEST_METHOD: `POST` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:01:28.067656 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6d8d8b1-5ede-4a08-b462-bcd20af20374 admin admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:01:28.067938 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6d8d8b1-5ede-4a08-b462-bcd20af20374 admin admin] PATH_INFO: `/v3/projects` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:01:28.068654 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6d8d8b1-5ede-4a08-b462-bcd20af20374 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:01:28.070632 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6d8d8b1-5ede-4a08-b462-bcd20af20374 admin admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:01:28.074745 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-1ee97f17-2d74-4374-99ba-98a3502188dd admin admin] The token cache is being invalidate because project 95675a44f30b42ef94a8afb1b9167f8c was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=92056) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} May 23 13:01:28.075030 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-1ee97f17-2d74-4374-99ba-98a3502188dd admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=92056) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:01:28.075884 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-1ee97f17-2d74-4374-99ba-98a3502188dd admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=92056) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:01:28.077466 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 1067/2142] 10.222.0.30 () {58 vars in 1232 bytes} [Mon May 23 13:01:27 2022] DELETE /identity/v3/projects/95675a44f30b42ef94a8afb1b9167f8c => generated 0 bytes in 311 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) May 23 13:01:28.079149 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66ebdaa3-0e16-4665-8998-bcd9f985e6c4 None None] REQUEST_METHOD: `POST` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:01:28.079437 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66ebdaa3-0e16-4665-8998-bcd9f985e6c4 None None] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:01:28.079745 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66ebdaa3-0e16-4665-8998-bcd9f985e6c4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:01:28.097077 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.auth.core [None req-66ebdaa3-0e16-4665-8998-bcd9f985e6c4 None None] MFA Rules not processed for user `0f7119f8fa5c4445977671140ded91a4`. Rule list: `[]` (Enabled: `True`). {{(pid=92056) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 13:01:28.098372 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.fernet_utils [None req-66ebdaa3-0e16-4665-8998-bcd9f985e6c4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92056) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:01:28.106793 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 1068/2143] 10.222.0.30 () {58 vars in 957 bytes} [Mon May 23 13:01:28 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 29 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) May 23 13:01:28.107766 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66ebdaa3-0e16-4665-8998-bcd9f985e6c4 None None] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:01:28.109008 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.fernet_utils [None req-66ebdaa3-0e16-4665-8998-bcd9f985e6c4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92056) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:01:28.120868 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66ebdaa3-0e16-4665-8998-bcd9f985e6c4 None None] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:01:28.126133 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3968a24-c28a-4003-aef9-d7f24815c4cc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:01:28.126281 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3968a24-c28a-4003-aef9-d7f24815c4cc admin admin] REQUEST_METHOD: `DELETE` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:01:28.126676 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3968a24-c28a-4003-aef9-d7f24815c4cc admin admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:01:28.126951 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3968a24-c28a-4003-aef9-d7f24815c4cc admin admin] PATH_INFO: `/v3/users/af3149805e5146808dd0eaf97cd68c4c` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:01:28.128137 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3968a24-c28a-4003-aef9-d7f24815c4cc admin admin] RBAC: Authorizing `identity:delete_user(user_id=af3149805e5146808dd0eaf97cd68c4c)` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:01:28.130081 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3968a24-c28a-4003-aef9-d7f24815c4cc admin admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:01:28.168274 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 1076/2144] 10.222.0.30 () {60 vars in 1152 bytes} [Mon May 23 13:01:27 2022] POST /identity/v3/projects => generated 387 bytes in 243 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) May 23 13:01:28.168274 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6d8d8b1-5ede-4a08-b462-bcd20af20374 admin admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:01:28.168767 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.fernet_utils [None req-c6d8d8b1-5ede-4a08-b462-bcd20af20374 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92057) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:01:28.235572 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6d8d8b1-5ede-4a08-b462-bcd20af20374 admin admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:01:28.240005 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b426d194-2126-40ac-b9b3-edc56edc6bd9 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2eb83ca38e79449095f3a2f947bbb06a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9c389ae65aaa4542bb5a5c4fd0bacf5b', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:01:28.240770 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b426d194-2126-40ac-b9b3-edc56edc6bd9 service neutron] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:01:28.241175 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b426d194-2126-40ac-b9b3-edc56edc6bd9 service neutron] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:01:28.241695 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b426d194-2126-40ac-b9b3-edc56edc6bd9 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:01:28.308618 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b426d194-2126-40ac-b9b3-edc56edc6bd9 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:01:28.310395 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b426d194-2126-40ac-b9b3-edc56edc6bd9 service neutron] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:01:28.332260 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 1077/2145] 10.222.0.30 () {60 vars in 1365 bytes} [Mon May 23 13:01:28 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 166 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) May 23 13:01:28.335180 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b426d194-2126-40ac-b9b3-edc56edc6bd9 service neutron] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:01:28.344922 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.fernet_utils [None req-b426d194-2126-40ac-b9b3-edc56edc6bd9 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92057) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:01:28.366339 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b426d194-2126-40ac-b9b3-edc56edc6bd9 service neutron] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:01:28.375647 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fdccdf1b-4cb2-447b-ba5f-06a4d7b52131 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:01:28.379233 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdccdf1b-4cb2-447b-ba5f-06a4d7b52131 admin admin] REQUEST_METHOD: `POST` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:01:28.379233 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdccdf1b-4cb2-447b-ba5f-06a4d7b52131 admin admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:01:28.379233 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdccdf1b-4cb2-447b-ba5f-06a4d7b52131 admin admin] PATH_INFO: `/v3/users` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:01:28.379233 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fdccdf1b-4cb2-447b-ba5f-06a4d7b52131 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:01:28.381406 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fdccdf1b-4cb2-447b-ba5f-06a4d7b52131 admin admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:01:28.396771 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-b3968a24-c28a-4003-aef9-d7f24815c4cc admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'af3149805e5146808dd0eaf97cd68c4c'} {{(pid=92056) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:01:28.398263 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-b3968a24-c28a-4003-aef9-d7f24815c4cc admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'af3149805e5146808dd0eaf97cd68c4c'} {{(pid=92056) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:01:28.406206 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 13:01:28.499804 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-b3968a24-c28a-4003-aef9-d7f24815c4cc admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'af3149805e5146808dd0eaf97cd68c4c'} {{(pid=92056) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:01:28.524873 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-b3968a24-c28a-4003-aef9-d7f24815c4cc admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'af3149805e5146808dd0eaf97cd68c4c'} {{(pid=92056) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:01:28.528242 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 1078/2146] 10.222.0.30 () {60 vars in 1143 bytes} [Mon May 23 13:01:28 2022] POST /identity/v3/users => generated 294 bytes in 195 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) May 23 13:01:28.532551 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fdccdf1b-4cb2-447b-ba5f-06a4d7b52131 admin admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:01:28.537927 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.fernet_utils [None req-fdccdf1b-4cb2-447b-ba5f-06a4d7b52131 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92057) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:01:28.538572 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-b3968a24-c28a-4003-aef9-d7f24815c4cc admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'af3149805e5146808dd0eaf97cd68c4c'} {{(pid=92056) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:01:28.545860 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 1069/2147] 10.222.0.30 () {58 vars in 1223 bytes} [Mon May 23 13:01:28 2022] DELETE /identity/v3/users/af3149805e5146808dd0eaf97cd68c4c => generated 0 bytes in 438 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) May 23 13:01:28.554797 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fdccdf1b-4cb2-447b-ba5f-06a4d7b52131 admin admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:01:28.644180 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=91,keystone:DELETE=7,keystone:INSERT=2 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 13:01:28.664472 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-028fca6c-36bc-4ab6-8943-a43fbcf3d61a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:01:28.665171 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-028fca6c-36bc-4ab6-8943-a43fbcf3d61a admin admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:01:28.665473 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-028fca6c-36bc-4ab6-8943-a43fbcf3d61a admin admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:01:28.665737 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-028fca6c-36bc-4ab6-8943-a43fbcf3d61a admin admin] PATH_INFO: `/v3/roles` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:01:28.666516 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-028fca6c-36bc-4ab6-8943-a43fbcf3d61a admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:01:28.670280 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-028fca6c-36bc-4ab6-8943-a43fbcf3d61a admin admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:01:28.688089 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 1079/2148] 10.222.0.30 () {58 vars in 1121 bytes} [Mon May 23 13:01:28 2022] GET /identity/v3/roles => generated 1420 bytes in 156 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) May 23 13:01:28.689066 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3968a24-c28a-4003-aef9-d7f24815c4cc admin admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:01:28.702250 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3968a24-c28a-4003-aef9-d7f24815c4cc admin admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:01:28.708484 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e57e050-20c6-40ed-b882-68d1570b03a1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:01:28.709389 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e57e050-20c6-40ed-b882-68d1570b03a1 admin admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:01:28.710098 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e57e050-20c6-40ed-b882-68d1570b03a1 admin admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:01:28.710520 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e57e050-20c6-40ed-b882-68d1570b03a1 admin admin] PATH_INFO: `/v3/roles` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:01:28.719244 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e57e050-20c6-40ed-b882-68d1570b03a1 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:01:28.719244 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e57e050-20c6-40ed-b882-68d1570b03a1 admin admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:01:28.730286 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 1070/2149] 10.222.0.30 () {58 vars in 1121 bytes} [Mon May 23 13:01:28 2022] GET /identity/v3/roles => generated 1420 bytes in 42 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) May 23 13:01:28.732117 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-028fca6c-36bc-4ab6-8943-a43fbcf3d61a admin admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:01:28.746005 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-028fca6c-36bc-4ab6-8943-a43fbcf3d61a admin admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:01:28.752448 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06c727d6-9d28-4af0-ab2a-f3b9568ab581 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:01:28.754237 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06c727d6-9d28-4af0-ab2a-f3b9568ab581 admin admin] REQUEST_METHOD: `PUT` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:01:28.755135 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06c727d6-9d28-4af0-ab2a-f3b9568ab581 admin admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:01:28.755420 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06c727d6-9d28-4af0-ab2a-f3b9568ab581 admin admin] PATH_INFO: `/v3/projects/5b4d791d4aca45429094aa0fea349041/users/f78ab5496c1e4008b83fa3866601c06c/roles/80fb22507c9a4ff09d44d8f12f6fff1a` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:01:28.782120 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06c727d6-9d28-4af0-ab2a-f3b9568ab581 admin admin] RBAC: Authorizing `identity:create_grant(project_id=5b4d791d4aca45429094aa0fea349041, user_id=f78ab5496c1e4008b83fa3866601c06c, role_id=80fb22507c9a4ff09d44d8f12f6fff1a)` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:01:28.785741 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06c727d6-9d28-4af0-ab2a-f3b9568ab581 admin admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:01:28.873593 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 1080/2150] 10.222.0.30 () {60 vars in 1482 bytes} [Mon May 23 13:01:28 2022] PUT /identity/v3/projects/5b4d791d4aca45429094aa0fea349041/users/f78ab5496c1e4008b83fa3866601c06c/roles/80fb22507c9a4ff09d44d8f12f6fff1a => generated 0 bytes in 142 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) May 23 13:01:28.879630 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79175148-ae2f-488b-9f57-8d3cf7d7b570 None None] REQUEST_METHOD: `POST` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:01:28.880050 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79175148-ae2f-488b-9f57-8d3cf7d7b570 None None] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:01:28.880403 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79175148-ae2f-488b-9f57-8d3cf7d7b570 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:01:28.899727 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.auth.core [None req-79175148-ae2f-488b-9f57-8d3cf7d7b570 None None] MFA Rules not processed for user `0f7119f8fa5c4445977671140ded91a4`. Rule list: `[]` (Enabled: `True`). {{(pid=92056) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 13:01:28.900910 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.fernet_utils [None req-79175148-ae2f-488b-9f57-8d3cf7d7b570 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92056) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:01:28.981333 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 1071/2151] 10.222.0.30 () {58 vars in 957 bytes} [Mon May 23 13:01:28 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 103 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) May 23 13:01:29.000054 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06c727d6-9d28-4af0-ab2a-f3b9568ab581 admin admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:01:29.002639 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.fernet_utils [None req-06c727d6-9d28-4af0-ab2a-f3b9568ab581 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92057) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:01:29.118978 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06c727d6-9d28-4af0-ab2a-f3b9568ab581 admin admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:01:29.126294 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04092a10-0637-4c46-9503-f894e7a9c2a7 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2eb83ca38e79449095f3a2f947bbb06a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9c389ae65aaa4542bb5a5c4fd0bacf5b', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:01:29.127093 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04092a10-0637-4c46-9503-f894e7a9c2a7 service neutron] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:01:29.127454 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04092a10-0637-4c46-9503-f894e7a9c2a7 service neutron] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:01:29.127806 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04092a10-0637-4c46-9503-f894e7a9c2a7 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:01:29.142865 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04092a10-0637-4c46-9503-f894e7a9c2a7 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:01:29.147794 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04092a10-0637-4c46-9503-f894e7a9c2a7 service neutron] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:01:29.148667 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79175148-ae2f-488b-9f57-8d3cf7d7b570 None None] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:01:29.150328 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.fernet_utils [None req-79175148-ae2f-488b-9f57-8d3cf7d7b570 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92056) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:01:29.161505 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 1081/2152] 10.222.0.30 () {60 vars in 1365 bytes} [Mon May 23 13:01:28 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 162 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) May 23 13:01:29.173288 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79175148-ae2f-488b-9f57-8d3cf7d7b570 None None] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:01:29.186564 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8cacb692-0b6a-457b-b247-9d2d500bc5b0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:01:29.188102 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cacb692-0b6a-457b-b247-9d2d500bc5b0 admin admin] REQUEST_METHOD: `DELETE` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:01:29.188541 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cacb692-0b6a-457b-b247-9d2d500bc5b0 admin admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:01:29.188966 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cacb692-0b6a-457b-b247-9d2d500bc5b0 admin admin] PATH_INFO: `/v3/projects/841ef16df18f4714ba1a61a24c4fa366` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:01:29.191288 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8cacb692-0b6a-457b-b247-9d2d500bc5b0 admin admin] RBAC: Authorizing `identity:delete_project(project_id=841ef16df18f4714ba1a61a24c4fa366)` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:01:29.193411 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8cacb692-0b6a-457b-b247-9d2d500bc5b0 admin admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:01:29.298547 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04092a10-0637-4c46-9503-f894e7a9c2a7 service neutron] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:01:29.300141 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.fernet_utils [None req-04092a10-0637-4c46-9503-f894e7a9c2a7 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92057) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:01:29.325645 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04092a10-0637-4c46-9503-f894e7a9c2a7 service neutron] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:01:29.334319 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6302d2a-e1eb-4a66-bc40-0611fb335d4e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:01:29.335578 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6302d2a-e1eb-4a66-bc40-0611fb335d4e admin admin] REQUEST_METHOD: `DELETE` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:01:29.336102 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6302d2a-e1eb-4a66-bc40-0611fb335d4e admin admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:01:29.336553 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6302d2a-e1eb-4a66-bc40-0611fb335d4e admin admin] PATH_INFO: `/v3/projects/53f8cb95bdab430d9027d8eeacf0c669` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:01:29.338070 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6302d2a-e1eb-4a66-bc40-0611fb335d4e admin admin] RBAC: Authorizing `identity:delete_project(project_id=53f8cb95bdab430d9027d8eeacf0c669)` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:01:29.340761 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6302d2a-e1eb-4a66-bc40-0611fb335d4e admin admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:01:29.364393 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-8cacb692-0b6a-457b-b247-9d2d500bc5b0 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '841ef16df18f4714ba1a61a24c4fa366', 'request_id': 'req-8cacb692-0b6a-457b-b247-9d2d500bc5b0'} {{(pid=92056) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:01:29.415197 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-8cacb692-0b6a-457b-b247-9d2d500bc5b0 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '841ef16df18f4714ba1a61a24c4fa366', 'request_id': 'req-8cacb692-0b6a-457b-b247-9d2d500bc5b0'} {{(pid=92056) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:01:29.422939 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-8cacb692-0b6a-457b-b247-9d2d500bc5b0 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '841ef16df18f4714ba1a61a24c4fa366', 'request_id': 'req-8cacb692-0b6a-457b-b247-9d2d500bc5b0'} {{(pid=92056) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:01:29.446023 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-8cacb692-0b6a-457b-b247-9d2d500bc5b0 admin admin] The token cache is being invalidate because project 841ef16df18f4714ba1a61a24c4fa366 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=92056) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} May 23 13:01:29.446439 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-8cacb692-0b6a-457b-b247-9d2d500bc5b0 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=92056) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:01:29.447215 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-8cacb692-0b6a-457b-b247-9d2d500bc5b0 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=92056) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:01:29.449718 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 1072/2153] 10.222.0.30 () {58 vars in 1232 bytes} [Mon May 23 13:01:29 2022] DELETE /identity/v3/projects/841ef16df18f4714ba1a61a24c4fa366 => generated 0 bytes in 301 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) May 23 13:01:29.454502 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8cacb692-0b6a-457b-b247-9d2d500bc5b0 admin admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:01:29.456740 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.fernet_utils [None req-8cacb692-0b6a-457b-b247-9d2d500bc5b0 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92056) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:01:29.483348 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [None req-f6302d2a-e1eb-4a66-bc40-0611fb335d4e admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '53f8cb95bdab430d9027d8eeacf0c669', 'request_id': 'req-f6302d2a-e1eb-4a66-bc40-0611fb335d4e'} {{(pid=92057) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:01:29.492601 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [None req-f6302d2a-e1eb-4a66-bc40-0611fb335d4e admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '53f8cb95bdab430d9027d8eeacf0c669', 'request_id': 'req-f6302d2a-e1eb-4a66-bc40-0611fb335d4e'} {{(pid=92057) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:01:29.540631 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8cacb692-0b6a-457b-b247-9d2d500bc5b0 admin admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:01:29.586518 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [None req-f6302d2a-e1eb-4a66-bc40-0611fb335d4e admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '53f8cb95bdab430d9027d8eeacf0c669', 'request_id': 'req-f6302d2a-e1eb-4a66-bc40-0611fb335d4e'} {{(pid=92057) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:01:29.603001 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [None req-f6302d2a-e1eb-4a66-bc40-0611fb335d4e admin admin] The token cache is being invalidate because project 53f8cb95bdab430d9027d8eeacf0c669 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=92057) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} May 23 13:01:29.603303 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [None req-f6302d2a-e1eb-4a66-bc40-0611fb335d4e admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=92057) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:01:29.604079 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [None req-f6302d2a-e1eb-4a66-bc40-0611fb335d4e admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=92057) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:01:29.606542 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 1082/2154] 10.222.0.30 () {58 vars in 1232 bytes} [Mon May 23 13:01:29 2022] DELETE /identity/v3/projects/53f8cb95bdab430d9027d8eeacf0c669 => generated 0 bytes in 309 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) May 23 13:01:29.611063 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6302d2a-e1eb-4a66-bc40-0611fb335d4e admin admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:01:29.612740 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.fernet_utils [None req-f6302d2a-e1eb-4a66-bc40-0611fb335d4e admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92057) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:01:29.636454 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e280b717-9113-4ccb-b0be-6701fdd079c9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:01:29.637987 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e280b717-9113-4ccb-b0be-6701fdd079c9 admin admin] REQUEST_METHOD: `DELETE` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:01:29.638271 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e280b717-9113-4ccb-b0be-6701fdd079c9 admin admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:01:29.638881 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e280b717-9113-4ccb-b0be-6701fdd079c9 admin admin] PATH_INFO: `/v3/users/5b79e59244cd415588d420abfb652466` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:01:29.641000 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e280b717-9113-4ccb-b0be-6701fdd079c9 admin admin] RBAC: Authorizing `identity:delete_user(user_id=5b79e59244cd415588d420abfb652466)` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:01:29.643036 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e280b717-9113-4ccb-b0be-6701fdd079c9 admin admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:01:29.710437 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 13:01:29.719533 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6302d2a-e1eb-4a66-bc40-0611fb335d4e admin admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:01:29.724673 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a149e6a6-d9ce-4fca-9f43-340788b9285e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:01:29.725764 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a149e6a6-d9ce-4fca-9f43-340788b9285e admin admin] REQUEST_METHOD: `DELETE` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:01:29.726083 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a149e6a6-d9ce-4fca-9f43-340788b9285e admin admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:01:29.726363 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a149e6a6-d9ce-4fca-9f43-340788b9285e admin admin] PATH_INFO: `/v3/users/b3554babb24f418297b82f983ea5cff8` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:01:29.727920 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a149e6a6-d9ce-4fca-9f43-340788b9285e admin admin] RBAC: Authorizing `identity:delete_user(user_id=b3554babb24f418297b82f983ea5cff8)` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:01:29.731101 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a149e6a6-d9ce-4fca-9f43-340788b9285e admin admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:01:29.931583 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-e280b717-9113-4ccb-b0be-6701fdd079c9 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '5b79e59244cd415588d420abfb652466'} {{(pid=92056) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:01:29.935245 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-e280b717-9113-4ccb-b0be-6701fdd079c9 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '5b79e59244cd415588d420abfb652466'} {{(pid=92056) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:01:30.009029 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-e280b717-9113-4ccb-b0be-6701fdd079c9 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '5b79e59244cd415588d420abfb652466'} {{(pid=92056) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:01:30.034318 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-e280b717-9113-4ccb-b0be-6701fdd079c9 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '5b79e59244cd415588d420abfb652466'} {{(pid=92056) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:01:30.042792 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-e280b717-9113-4ccb-b0be-6701fdd079c9 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '5b79e59244cd415588d420abfb652466'} {{(pid=92056) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:01:30.050722 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 1073/2155] 10.222.0.30 () {58 vars in 1223 bytes} [Mon May 23 13:01:29 2022] DELETE /identity/v3/users/5b79e59244cd415588d420abfb652466 => generated 0 bytes in 597 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) May 23 13:01:30.154680 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [None req-a149e6a6-d9ce-4fca-9f43-340788b9285e admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'b3554babb24f418297b82f983ea5cff8'} {{(pid=92057) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:01:30.605222 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [None req-a149e6a6-d9ce-4fca-9f43-340788b9285e admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'b3554babb24f418297b82f983ea5cff8'} {{(pid=92057) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:01:30.618984 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [None req-a149e6a6-d9ce-4fca-9f43-340788b9285e admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'b3554babb24f418297b82f983ea5cff8'} {{(pid=92057) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:01:30.619772 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [None req-a149e6a6-d9ce-4fca-9f43-340788b9285e admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'b3554babb24f418297b82f983ea5cff8'} {{(pid=92057) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:01:30.662934 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [None req-a149e6a6-d9ce-4fca-9f43-340788b9285e admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'b3554babb24f418297b82f983ea5cff8'} {{(pid=92057) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:01:30.679748 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 1083/2156] 10.222.0.30 () {58 vars in 1223 bytes} [Mon May 23 13:01:29 2022] DELETE /identity/v3/users/b3554babb24f418297b82f983ea5cff8 => generated 0 bytes in 1067 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) May 23 13:01:31.116362 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e280b717-9113-4ccb-b0be-6701fdd079c9 admin admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:01:31.119283 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.fernet_utils [None req-e280b717-9113-4ccb-b0be-6701fdd079c9 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92056) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:01:31.246774 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e280b717-9113-4ccb-b0be-6701fdd079c9 admin admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:01:31.375818 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-519fffab-04f2-4252-8525-69c762744187 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:01:31.377058 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-519fffab-04f2-4252-8525-69c762744187 admin admin] REQUEST_METHOD: `DELETE` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:01:31.377413 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-519fffab-04f2-4252-8525-69c762744187 admin admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:01:31.377837 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-519fffab-04f2-4252-8525-69c762744187 admin admin] PATH_INFO: `/v3/projects/19c37ded87a845558f6800794483ed3c` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:01:31.379277 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-519fffab-04f2-4252-8525-69c762744187 admin admin] RBAC: Authorizing `identity:delete_project(project_id=19c37ded87a845558f6800794483ed3c)` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:01:31.381778 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-519fffab-04f2-4252-8525-69c762744187 admin admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:01:31.546420 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=85,keystone:DELETE=14,keystone:INSERT=1 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 13:01:31.553834 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a149e6a6-d9ce-4fca-9f43-340788b9285e admin admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:01:31.556388 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.fernet_utils [None req-a149e6a6-d9ce-4fca-9f43-340788b9285e admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92057) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:01:31.578690 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-519fffab-04f2-4252-8525-69c762744187 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '19c37ded87a845558f6800794483ed3c', 'request_id': 'req-519fffab-04f2-4252-8525-69c762744187'} {{(pid=92056) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:01:31.583062 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a149e6a6-d9ce-4fca-9f43-340788b9285e admin admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:01:31.678264 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-891a17f6-1965-401b-b050-19a0a597fc61 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:01:31.679185 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-891a17f6-1965-401b-b050-19a0a597fc61 admin admin] REQUEST_METHOD: `DELETE` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:01:31.679471 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-891a17f6-1965-401b-b050-19a0a597fc61 admin admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:01:31.679909 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-891a17f6-1965-401b-b050-19a0a597fc61 admin admin] PATH_INFO: `/v3/projects/4a3560cee69a4b6a98856668d8724958` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:01:31.681180 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-891a17f6-1965-401b-b050-19a0a597fc61 admin admin] RBAC: Authorizing `identity:delete_project(project_id=4a3560cee69a4b6a98856668d8724958)` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:01:31.689290 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-891a17f6-1965-401b-b050-19a0a597fc61 admin admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:01:31.696012 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-519fffab-04f2-4252-8525-69c762744187 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '19c37ded87a845558f6800794483ed3c', 'request_id': 'req-519fffab-04f2-4252-8525-69c762744187'} {{(pid=92056) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:01:31.742803 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-519fffab-04f2-4252-8525-69c762744187 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '19c37ded87a845558f6800794483ed3c', 'request_id': 'req-519fffab-04f2-4252-8525-69c762744187'} {{(pid=92056) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:01:31.786640 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=89,keystone:DELETE=7,keystone:INSERT=4 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 13:01:31.813035 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-519fffab-04f2-4252-8525-69c762744187 admin admin] The token cache is being invalidate because project 19c37ded87a845558f6800794483ed3c was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=92056) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} May 23 13:01:31.813944 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-519fffab-04f2-4252-8525-69c762744187 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=92056) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:01:31.815580 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-519fffab-04f2-4252-8525-69c762744187 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=92056) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:01:31.818019 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 1074/2157] 10.222.0.30 () {58 vars in 1232 bytes} [Mon May 23 13:01:31 2022] DELETE /identity/v3/projects/19c37ded87a845558f6800794483ed3c => generated 0 bytes in 702 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) May 23 13:01:31.824369 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e326718-8f7a-4fb1-a41f-73e5fd948f52 None None] REQUEST_METHOD: `POST` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:01:31.824369 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e326718-8f7a-4fb1-a41f-73e5fd948f52 None None] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:01:31.824672 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e326718-8f7a-4fb1-a41f-73e5fd948f52 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:01:31.871884 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.auth.core [None req-3e326718-8f7a-4fb1-a41f-73e5fd948f52 None None] MFA Rules not processed for user `0f7119f8fa5c4445977671140ded91a4`. Rule list: `[]` (Enabled: `True`). {{(pid=92056) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 13:01:31.873430 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.fernet_utils [None req-3e326718-8f7a-4fb1-a41f-73e5fd948f52 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92056) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:01:31.916174 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [None req-891a17f6-1965-401b-b050-19a0a597fc61 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '4a3560cee69a4b6a98856668d8724958', 'request_id': 'req-891a17f6-1965-401b-b050-19a0a597fc61'} {{(pid=92057) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:01:31.926804 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [None req-891a17f6-1965-401b-b050-19a0a597fc61 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '4a3560cee69a4b6a98856668d8724958', 'request_id': 'req-891a17f6-1965-401b-b050-19a0a597fc61'} {{(pid=92057) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:01:31.988754 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [None req-891a17f6-1965-401b-b050-19a0a597fc61 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '4a3560cee69a4b6a98856668d8724958', 'request_id': 'req-891a17f6-1965-401b-b050-19a0a597fc61'} {{(pid=92057) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:01:31.996531 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 1075/2158] 10.222.0.30 () {58 vars in 957 bytes} [Mon May 23 13:01:31 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 169 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) May 23 13:01:31.999275 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8381a5b7-a2aa-4cde-a32e-5ee702616746 None None] REQUEST_METHOD: `POST` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:01:31.999572 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8381a5b7-a2aa-4cde-a32e-5ee702616746 None None] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:01:32.012086 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8381a5b7-a2aa-4cde-a32e-5ee702616746 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:01:32.018613 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [None req-891a17f6-1965-401b-b050-19a0a597fc61 admin admin] The token cache is being invalidate because project 4a3560cee69a4b6a98856668d8724958 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=92057) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} May 23 13:01:32.019077 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [None req-891a17f6-1965-401b-b050-19a0a597fc61 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=92057) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:01:32.020127 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [None req-891a17f6-1965-401b-b050-19a0a597fc61 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=92057) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:01:32.026926 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 1084/2159] 10.222.0.30 () {58 vars in 1232 bytes} [Mon May 23 13:01:31 2022] DELETE /identity/v3/projects/4a3560cee69a4b6a98856668d8724958 => generated 0 bytes in 472 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) May 23 13:01:32.030683 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40b3834c-4c23-454e-887b-2f298bb8afe5 None None] REQUEST_METHOD: `POST` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:01:32.031138 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40b3834c-4c23-454e-887b-2f298bb8afe5 None None] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:01:32.031792 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40b3834c-4c23-454e-887b-2f298bb8afe5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:01:32.052796 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.auth.core [None req-8381a5b7-a2aa-4cde-a32e-5ee702616746 None None] MFA Rules not processed for user `0f7119f8fa5c4445977671140ded91a4`. Rule list: `[]` (Enabled: `True`). {{(pid=92056) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 13:01:32.054249 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.fernet_utils [None req-8381a5b7-a2aa-4cde-a32e-5ee702616746 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92056) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:01:32.078896 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.auth.core [None req-40b3834c-4c23-454e-887b-2f298bb8afe5 None None] MFA Rules not processed for user `0f7119f8fa5c4445977671140ded91a4`. Rule list: `[]` (Enabled: `True`). {{(pid=92057) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 13:01:32.080261 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 1076/2160] 10.222.0.30 () {58 vars in 957 bytes} [Mon May 23 13:01:31 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 83 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) May 23 13:01:32.084899 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8381a5b7-a2aa-4cde-a32e-5ee702616746 None None] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:01:32.098048 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.fernet_utils [None req-40b3834c-4c23-454e-887b-2f298bb8afe5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92057) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:01:32.104355 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8381a5b7-a2aa-4cde-a32e-5ee702616746 None None] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:01:32.115240 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eaff67b4-05b1-49a2-8b04-d6633b22890b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:01:32.119320 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eaff67b4-05b1-49a2-8b04-d6633b22890b admin admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:01:32.119918 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eaff67b4-05b1-49a2-8b04-d6633b22890b admin admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:01:32.120206 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eaff67b4-05b1-49a2-8b04-d6633b22890b admin admin] PATH_INFO: `/v3/domains` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:01:32.121055 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eaff67b4-05b1-49a2-8b04-d6633b22890b admin admin] RBAC: Adding query filter params (name=Default) {{(pid=92056) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} May 23 13:01:32.121723 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eaff67b4-05b1-49a2-8b04-d6633b22890b admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:01:32.124598 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eaff67b4-05b1-49a2-8b04-d6633b22890b admin admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:01:32.133071 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 1085/2161] 10.222.0.30 () {58 vars in 957 bytes} [Mon May 23 13:01:32 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 103 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) May 23 13:01:32.136966 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66a3bd32-d820-40d2-935d-23633c4bb137 None None] REQUEST_METHOD: `POST` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:01:32.138059 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66a3bd32-d820-40d2-935d-23633c4bb137 None None] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:01:32.138944 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66a3bd32-d820-40d2-935d-23633c4bb137 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:01:32.143035 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 1077/2162] 10.222.0.30 () {58 vars in 1152 bytes} [Mon May 23 13:01:32 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 58 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) May 23 13:01:32.147111 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eaff67b4-05b1-49a2-8b04-d6633b22890b admin admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:01:32.163899 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eaff67b4-05b1-49a2-8b04-d6633b22890b admin admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:01:32.171630 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9604ee6-a3b0-42ef-a7a8-0ca47b2c2246 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:01:32.172410 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9604ee6-a3b0-42ef-a7a8-0ca47b2c2246 admin admin] REQUEST_METHOD: `POST` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:01:32.172761 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9604ee6-a3b0-42ef-a7a8-0ca47b2c2246 admin admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:01:32.173145 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9604ee6-a3b0-42ef-a7a8-0ca47b2c2246 admin admin] PATH_INFO: `/v3/projects` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:01:32.174448 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9604ee6-a3b0-42ef-a7a8-0ca47b2c2246 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:01:32.177491 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9604ee6-a3b0-42ef-a7a8-0ca47b2c2246 admin admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:01:32.192297 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.auth.core [None req-66a3bd32-d820-40d2-935d-23633c4bb137 None None] MFA Rules not processed for user `0f7119f8fa5c4445977671140ded91a4`. Rule list: `[]` (Enabled: `True`). {{(pid=92057) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 13:01:32.195874 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.fernet_utils [None req-66a3bd32-d820-40d2-935d-23633c4bb137 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92057) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:01:32.219551 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 1086/2163] 10.222.0.30 () {58 vars in 957 bytes} [Mon May 23 13:01:32 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) May 23 13:01:32.221389 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66a3bd32-d820-40d2-935d-23633c4bb137 None None] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:01:32.243145 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66a3bd32-d820-40d2-935d-23633c4bb137 None None] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:01:32.255955 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2f81a19-5a25-409a-af33-bfe632930e0f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:01:32.256826 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2f81a19-5a25-409a-af33-bfe632930e0f admin admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:01:32.257114 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2f81a19-5a25-409a-af33-bfe632930e0f admin admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:01:32.259529 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2f81a19-5a25-409a-af33-bfe632930e0f admin admin] PATH_INFO: `/v3/domains` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:01:32.260358 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2f81a19-5a25-409a-af33-bfe632930e0f admin admin] RBAC: Adding query filter params (name=Default) {{(pid=92057) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} May 23 13:01:32.260840 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2f81a19-5a25-409a-af33-bfe632930e0f admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:01:32.269427 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2f81a19-5a25-409a-af33-bfe632930e0f admin admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:01:32.291301 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 1087/2164] 10.222.0.30 () {58 vars in 1152 bytes} [Mon May 23 13:01:32 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 62 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) May 23 13:01:32.294121 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2f81a19-5a25-409a-af33-bfe632930e0f admin admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:01:32.314827 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2f81a19-5a25-409a-af33-bfe632930e0f admin admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:01:32.336229 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 1078/2165] 10.222.0.30 () {60 vars in 1152 bytes} [Mon May 23 13:01:32 2022] POST /identity/v3/projects => generated 367 bytes in 183 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) May 23 13:01:32.336749 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4ee0720-9d8e-42af-b44b-d2a491a15b2a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:01:32.336749 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4ee0720-9d8e-42af-b44b-d2a491a15b2a admin admin] REQUEST_METHOD: `POST` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:01:32.336749 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4ee0720-9d8e-42af-b44b-d2a491a15b2a admin admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:01:32.336749 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4ee0720-9d8e-42af-b44b-d2a491a15b2a admin admin] PATH_INFO: `/v3/projects` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:01:32.337088 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4ee0720-9d8e-42af-b44b-d2a491a15b2a admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:01:32.337262 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9604ee6-a3b0-42ef-a7a8-0ca47b2c2246 admin admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:01:32.352020 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4ee0720-9d8e-42af-b44b-d2a491a15b2a admin admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:01:32.436912 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9604ee6-a3b0-42ef-a7a8-0ca47b2c2246 admin admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:01:32.442693 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20a9b431-0586-4973-939a-b626f0f02fcc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:01:32.443455 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20a9b431-0586-4973-939a-b626f0f02fcc admin admin] REQUEST_METHOD: `POST` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:01:32.443868 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20a9b431-0586-4973-939a-b626f0f02fcc admin admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:01:32.444212 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20a9b431-0586-4973-939a-b626f0f02fcc admin admin] PATH_INFO: `/v3/users` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:01:32.444986 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20a9b431-0586-4973-939a-b626f0f02fcc admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:01:32.447271 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20a9b431-0586-4973-939a-b626f0f02fcc admin admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:01:32.476979 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 1088/2166] 10.222.0.30 () {60 vars in 1152 bytes} [Mon May 23 13:01:32 2022] POST /identity/v3/projects => generated 367 bytes in 184 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) May 23 13:01:32.483981 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4ee0720-9d8e-42af-b44b-d2a491a15b2a admin admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:01:32.535826 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 1079/2167] 10.222.0.30 () {60 vars in 1143 bytes} [Mon May 23 13:01:32 2022] POST /identity/v3/users => generated 284 bytes in 199 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) May 23 13:01:32.540978 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20a9b431-0586-4973-939a-b626f0f02fcc admin admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:01:32.561934 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20a9b431-0586-4973-939a-b626f0f02fcc admin admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:01:32.624700 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4ee0720-9d8e-42af-b44b-d2a491a15b2a admin admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:01:32.631035 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c18dee34-1494-40f0-8a2f-d9daa3b92fd9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:01:32.631921 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c18dee34-1494-40f0-8a2f-d9daa3b92fd9 admin admin] REQUEST_METHOD: `POST` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:01:32.632395 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c18dee34-1494-40f0-8a2f-d9daa3b92fd9 admin admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:01:32.632837 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c18dee34-1494-40f0-8a2f-d9daa3b92fd9 admin admin] PATH_INFO: `/v3/users` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:01:32.634002 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c18dee34-1494-40f0-8a2f-d9daa3b92fd9 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:01:32.637617 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c18dee34-1494-40f0-8a2f-d9daa3b92fd9 admin admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:01:32.701283 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-beb8888d-7d30-4f68-bd82-97aab033785f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:01:32.701997 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-beb8888d-7d30-4f68-bd82-97aab033785f admin admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:01:32.702287 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-beb8888d-7d30-4f68-bd82-97aab033785f admin admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:01:32.702568 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-beb8888d-7d30-4f68-bd82-97aab033785f admin admin] PATH_INFO: `/v3/roles` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:01:32.703376 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-beb8888d-7d30-4f68-bd82-97aab033785f admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:01:32.705152 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-beb8888d-7d30-4f68-bd82-97aab033785f admin admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:01:32.734258 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 1080/2168] 10.222.0.30 () {58 vars in 1121 bytes} [Mon May 23 13:01:32 2022] GET /identity/v3/roles => generated 1420 bytes in 189 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) May 23 13:01:32.734258 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-beb8888d-7d30-4f68-bd82-97aab033785f admin admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:01:32.747408 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-beb8888d-7d30-4f68-bd82-97aab033785f admin admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:01:32.758929 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f59e227-fd45-450e-a4d5-2851e6d20248 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:01:32.759758 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f59e227-fd45-450e-a4d5-2851e6d20248 admin admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:01:32.760046 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f59e227-fd45-450e-a4d5-2851e6d20248 admin admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:01:32.760322 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f59e227-fd45-450e-a4d5-2851e6d20248 admin admin] PATH_INFO: `/v3/roles` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:01:32.765410 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f59e227-fd45-450e-a4d5-2851e6d20248 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:01:32.772175 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 1089/2169] 10.222.0.30 () {60 vars in 1143 bytes} [Mon May 23 13:01:32 2022] POST /identity/v3/users => generated 284 bytes in 290 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) May 23 13:01:32.774563 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f59e227-fd45-450e-a4d5-2851e6d20248 admin admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:01:32.775618 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c18dee34-1494-40f0-8a2f-d9daa3b92fd9 admin admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:01:32.797825 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c18dee34-1494-40f0-8a2f-d9daa3b92fd9 admin admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:01:32.802616 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 1081/2170] 10.222.0.30 () {58 vars in 1121 bytes} [Mon May 23 13:01:32 2022] GET /identity/v3/roles => generated 1420 bytes in 66 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) May 23 13:01:32.804872 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f59e227-fd45-450e-a4d5-2851e6d20248 admin admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:01:32.813811 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12f429ad-df50-4a80-b009-5d240c2ceaa4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:01:32.818105 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12f429ad-df50-4a80-b009-5d240c2ceaa4 admin admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:01:32.818105 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12f429ad-df50-4a80-b009-5d240c2ceaa4 admin admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:01:32.818105 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12f429ad-df50-4a80-b009-5d240c2ceaa4 admin admin] PATH_INFO: `/v3/roles` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:01:32.818105 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12f429ad-df50-4a80-b009-5d240c2ceaa4 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:01:32.819624 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12f429ad-df50-4a80-b009-5d240c2ceaa4 admin admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:01:32.820116 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f59e227-fd45-450e-a4d5-2851e6d20248 admin admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:01:32.826961 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33d045fe-276c-4704-8cdd-43f866c130e9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:01:32.827865 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33d045fe-276c-4704-8cdd-43f866c130e9 admin admin] REQUEST_METHOD: `PUT` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:01:32.828145 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33d045fe-276c-4704-8cdd-43f866c130e9 admin admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:01:32.828418 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33d045fe-276c-4704-8cdd-43f866c130e9 admin admin] PATH_INFO: `/v3/projects/9ef27bc16356446eb11ed70c90e7869f/users/af36d5ec94e444a1a903514d9799f687/roles/80fb22507c9a4ff09d44d8f12f6fff1a` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:01:32.835943 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 1090/2171] 10.222.0.30 () {58 vars in 1121 bytes} [Mon May 23 13:01:32 2022] GET /identity/v3/roles => generated 1420 bytes in 61 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) May 23 13:01:32.841063 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12f429ad-df50-4a80-b009-5d240c2ceaa4 admin admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:01:32.847863 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33d045fe-276c-4704-8cdd-43f866c130e9 admin admin] RBAC: Authorizing `identity:create_grant(project_id=9ef27bc16356446eb11ed70c90e7869f, user_id=af36d5ec94e444a1a903514d9799f687, role_id=80fb22507c9a4ff09d44d8f12f6fff1a)` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:01:32.850759 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33d045fe-276c-4704-8cdd-43f866c130e9 admin admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:01:32.858947 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12f429ad-df50-4a80-b009-5d240c2ceaa4 admin admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:01:32.868656 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32f43186-2f25-4627-b394-731d2ea655ea admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:01:32.868656 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32f43186-2f25-4627-b394-731d2ea655ea admin admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:01:32.868656 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32f43186-2f25-4627-b394-731d2ea655ea admin admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:01:32.868656 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32f43186-2f25-4627-b394-731d2ea655ea admin admin] PATH_INFO: `/v3/roles` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:01:32.870104 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32f43186-2f25-4627-b394-731d2ea655ea admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:01:32.873246 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32f43186-2f25-4627-b394-731d2ea655ea admin admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:01:32.885378 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 1091/2172] 10.222.0.30 () {58 vars in 1121 bytes} [Mon May 23 13:01:32 2022] GET /identity/v3/roles => generated 1420 bytes in 46 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) May 23 13:01:32.891341 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32f43186-2f25-4627-b394-731d2ea655ea admin admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:01:32.902899 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32f43186-2f25-4627-b394-731d2ea655ea admin admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:01:32.910066 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0327c7a-56ea-48bd-8b33-7ef42fe5f703 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:01:32.914863 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0327c7a-56ea-48bd-8b33-7ef42fe5f703 admin admin] REQUEST_METHOD: `PUT` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:01:32.914863 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0327c7a-56ea-48bd-8b33-7ef42fe5f703 admin admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:01:32.914863 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0327c7a-56ea-48bd-8b33-7ef42fe5f703 admin admin] PATH_INFO: `/v3/projects/abdeef898e9b49c18cec057dfba5b3af/users/275524da6b1247638abfdb01f8761d1c/roles/80fb22507c9a4ff09d44d8f12f6fff1a` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:01:32.930645 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0327c7a-56ea-48bd-8b33-7ef42fe5f703 admin admin] RBAC: Authorizing `identity:create_grant(project_id=abdeef898e9b49c18cec057dfba5b3af, user_id=275524da6b1247638abfdb01f8761d1c, role_id=80fb22507c9a4ff09d44d8f12f6fff1a)` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:01:32.934924 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0327c7a-56ea-48bd-8b33-7ef42fe5f703 admin admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:01:32.959697 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 1082/2173] 10.222.0.30 () {60 vars in 1482 bytes} [Mon May 23 13:01:32 2022] PUT /identity/v3/projects/9ef27bc16356446eb11ed70c90e7869f/users/af36d5ec94e444a1a903514d9799f687/roles/80fb22507c9a4ff09d44d8f12f6fff1a => generated 0 bytes in 146 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) May 23 13:01:32.962841 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84778ae0-d6eb-4242-a769-6c209c2dc167 None None] REQUEST_METHOD: `POST` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:01:32.963355 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84778ae0-d6eb-4242-a769-6c209c2dc167 None None] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:01:32.963737 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84778ae0-d6eb-4242-a769-6c209c2dc167 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:01:33.035031 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.auth.core [None req-84778ae0-d6eb-4242-a769-6c209c2dc167 None None] MFA Rules not processed for user `af36d5ec94e444a1a903514d9799f687`. Rule list: `[]` (Enabled: `True`). {{(pid=92056) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 13:01:33.044865 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.fernet_utils [None req-84778ae0-d6eb-4242-a769-6c209c2dc167 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92056) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:01:33.055817 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 1092/2174] 10.222.0.30 () {60 vars in 1482 bytes} [Mon May 23 13:01:32 2022] PUT /identity/v3/projects/abdeef898e9b49c18cec057dfba5b3af/users/275524da6b1247638abfdb01f8761d1c/roles/80fb22507c9a4ff09d44d8f12f6fff1a => generated 0 bytes in 156 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) May 23 13:01:33.058010 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e070fe5-de63-40b0-9eee-8b8630edd821 None None] REQUEST_METHOD: `POST` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:01:33.058903 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e070fe5-de63-40b0-9eee-8b8630edd821 None None] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:01:33.059240 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e070fe5-de63-40b0-9eee-8b8630edd821 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:01:33.110055 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.auth.core [None req-8e070fe5-de63-40b0-9eee-8b8630edd821 None None] MFA Rules not processed for user `275524da6b1247638abfdb01f8761d1c`. Rule list: `[]` (Enabled: `True`). {{(pid=92057) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 13:01:33.111458 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.fernet_utils [None req-8e070fe5-de63-40b0-9eee-8b8630edd821 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92057) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:01:33.161620 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 1083/2175] 10.222.0.30 () {58 vars in 957 bytes} [Mon May 23 13:01:32 2022] POST /identity/v3/auth/tokens => generated 3234 bytes in 198 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) May 23 13:01:33.162834 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84778ae0-d6eb-4242-a769-6c209c2dc167 None None] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:01:33.249931 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 1093/2176] 10.222.0.30 () {58 vars in 957 bytes} [Mon May 23 13:01:33 2022] POST /identity/v3/auth/tokens => generated 3234 bytes in 191 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) May 23 13:01:33.267291 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e070fe5-de63-40b0-9eee-8b8630edd821 None None] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:01:33.268878 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.fernet_utils [None req-8e070fe5-de63-40b0-9eee-8b8630edd821 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92057) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:01:33.371759 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84778ae0-d6eb-4242-a769-6c209c2dc167 None None] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:01:33.390660 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23db3309-dfa3-4d94-ae15-0cfc5169eff3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:01:33.391364 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23db3309-dfa3-4d94-ae15-0cfc5169eff3 admin admin] REQUEST_METHOD: `POST` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:01:33.391645 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23db3309-dfa3-4d94-ae15-0cfc5169eff3 admin admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:01:33.391924 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23db3309-dfa3-4d94-ae15-0cfc5169eff3 admin admin] PATH_INFO: `/v3/projects` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:01:33.392800 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-23db3309-dfa3-4d94-ae15-0cfc5169eff3 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:01:33.394864 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-23db3309-dfa3-4d94-ae15-0cfc5169eff3 admin admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:01:33.419098 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e070fe5-de63-40b0-9eee-8b8630edd821 None None] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:01:33.424499 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7204fa50-e712-4638-8822-4cd58448d348 service glance] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '5cea0feed4254ec085854986f99acf30', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9c389ae65aaa4542bb5a5c4fd0bacf5b', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:01:33.425267 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7204fa50-e712-4638-8822-4cd58448d348 service glance] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:01:33.425589 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7204fa50-e712-4638-8822-4cd58448d348 service glance] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:01:33.425963 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7204fa50-e712-4638-8822-4cd58448d348 service glance] PATH_INFO: `/v3/auth/tokens` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:01:33.439549 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7204fa50-e712-4638-8822-4cd58448d348 service glance] RBAC: Authorizing `identity:validate_token()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:01:33.441858 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7204fa50-e712-4638-8822-4cd58448d348 service glance] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:01:33.455698 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 1094/2177] 10.222.0.30 () {60 vars in 1365 bytes} [Mon May 23 13:01:33 2022] GET /identity/v3/auth/tokens => generated 3234 bytes in 189 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) May 23 13:01:33.511877 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 1084/2178] 10.222.0.30 () {60 vars in 1152 bytes} [Mon May 23 13:01:33 2022] POST /identity/v3/projects => generated 367 bytes in 350 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) May 23 13:01:33.513105 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7204fa50-e712-4638-8822-4cd58448d348 service glance] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:01:33.525569 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7204fa50-e712-4638-8822-4cd58448d348 service glance] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:01:33.600361 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7620c10f-d290-40ff-939c-eeeb56013dbf admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:01:33.600953 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7620c10f-d290-40ff-939c-eeeb56013dbf admin admin] REQUEST_METHOD: `POST` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:01:33.601166 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7620c10f-d290-40ff-939c-eeeb56013dbf admin admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:01:33.601499 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7620c10f-d290-40ff-939c-eeeb56013dbf admin admin] PATH_INFO: `/v3/users` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:01:33.602154 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7620c10f-d290-40ff-939c-eeeb56013dbf admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:01:33.604177 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7620c10f-d290-40ff-939c-eeeb56013dbf admin admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:01:33.686083 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 1095/2179] 10.222.0.30 () {60 vars in 1143 bytes} [Mon May 23 13:01:33 2022] POST /identity/v3/users => generated 290 bytes in 174 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) May 23 13:01:33.690206 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23db3309-dfa3-4d94-ae15-0cfc5169eff3 admin admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:01:33.707748 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23db3309-dfa3-4d94-ae15-0cfc5169eff3 admin admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:01:33.717136 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff58f398-c107-4e1e-bac0-2fb202ee6a8e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:01:33.717136 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff58f398-c107-4e1e-bac0-2fb202ee6a8e admin admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:01:33.717136 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff58f398-c107-4e1e-bac0-2fb202ee6a8e admin admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:01:33.717136 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff58f398-c107-4e1e-bac0-2fb202ee6a8e admin admin] PATH_INFO: `/v3/roles` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:01:33.717547 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff58f398-c107-4e1e-bac0-2fb202ee6a8e admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:01:33.718190 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff58f398-c107-4e1e-bac0-2fb202ee6a8e admin admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:01:33.730092 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 1085/2180] 10.222.0.30 () {58 vars in 1121 bytes} [Mon May 23 13:01:33 2022] GET /identity/v3/roles => generated 1420 bytes in 41 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) May 23 13:01:33.734259 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7620c10f-d290-40ff-939c-eeeb56013dbf admin admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:01:33.752721 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7620c10f-d290-40ff-939c-eeeb56013dbf admin admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:01:33.759986 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-539d6433-e4dc-46b3-a04c-6946fc28aaaa admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:01:33.765074 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-539d6433-e4dc-46b3-a04c-6946fc28aaaa admin admin] REQUEST_METHOD: `PUT` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:01:33.765500 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-539d6433-e4dc-46b3-a04c-6946fc28aaaa admin admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:01:33.765901 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-539d6433-e4dc-46b3-a04c-6946fc28aaaa admin admin] PATH_INFO: `/v3/projects/dee1db77df764b8da152352ae3b905d7/users/e456ac07a4484a96a2ac7a4d6f07f7e1/roles/bc3c3c9d654940cbaa5177a6be23d1bc` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:01:33.800018 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-539d6433-e4dc-46b3-a04c-6946fc28aaaa admin admin] RBAC: Authorizing `identity:create_grant(project_id=dee1db77df764b8da152352ae3b905d7, user_id=e456ac07a4484a96a2ac7a4d6f07f7e1, role_id=bc3c3c9d654940cbaa5177a6be23d1bc)` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:01:33.807007 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-539d6433-e4dc-46b3-a04c-6946fc28aaaa admin admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:01:33.944776 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 1096/2181] 10.222.0.30 () {60 vars in 1482 bytes} [Mon May 23 13:01:33 2022] PUT /identity/v3/projects/dee1db77df764b8da152352ae3b905d7/users/e456ac07a4484a96a2ac7a4d6f07f7e1/roles/bc3c3c9d654940cbaa5177a6be23d1bc => generated 0 bytes in 211 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) May 23 13:01:33.950154 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff58f398-c107-4e1e-bac0-2fb202ee6a8e admin admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:01:33.968432 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff58f398-c107-4e1e-bac0-2fb202ee6a8e admin admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:01:34.095100 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1cede01-5afc-4603-a9fd-d83a8d1fd56f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:01:34.095979 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1cede01-5afc-4603-a9fd-d83a8d1fd56f admin admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:01:34.096354 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1cede01-5afc-4603-a9fd-d83a8d1fd56f admin admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:01:34.096765 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1cede01-5afc-4603-a9fd-d83a8d1fd56f admin admin] PATH_INFO: `/v3/roles` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:01:34.097601 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1cede01-5afc-4603-a9fd-d83a8d1fd56f admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:01:34.103808 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1cede01-5afc-4603-a9fd-d83a8d1fd56f admin admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:01:34.112362 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 1086/2182] 10.222.0.30 () {58 vars in 1121 bytes} [Mon May 23 13:01:33 2022] GET /identity/v3/roles => generated 1420 bytes in 163 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) May 23 13:01:34.125926 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-539d6433-e4dc-46b3-a04c-6946fc28aaaa admin admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:01:34.151050 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-539d6433-e4dc-46b3-a04c-6946fc28aaaa admin admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:01:34.167150 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d73b57ab-f467-4ff2-9504-f8bd9c5f890c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:01:34.168171 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d73b57ab-f467-4ff2-9504-f8bd9c5f890c admin admin] REQUEST_METHOD: `PUT` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:01:34.168476 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d73b57ab-f467-4ff2-9504-f8bd9c5f890c admin admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:01:34.168810 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d73b57ab-f467-4ff2-9504-f8bd9c5f890c admin admin] PATH_INFO: `/v3/projects/dee1db77df764b8da152352ae3b905d7/users/e456ac07a4484a96a2ac7a4d6f07f7e1/roles/bc3c3c9d654940cbaa5177a6be23d1bc` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:01:34.173576 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d73b57ab-f467-4ff2-9504-f8bd9c5f890c admin admin] RBAC: Authorizing `identity:create_grant(project_id=dee1db77df764b8da152352ae3b905d7, user_id=e456ac07a4484a96a2ac7a4d6f07f7e1, role_id=bc3c3c9d654940cbaa5177a6be23d1bc)` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:01:34.185809 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d73b57ab-f467-4ff2-9504-f8bd9c5f890c admin admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:01:34.202924 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 1097/2183] 10.222.0.30 () {60 vars in 1482 bytes} [Mon May 23 13:01:34 2022] PUT /identity/v3/projects/dee1db77df764b8da152352ae3b905d7/users/e456ac07a4484a96a2ac7a4d6f07f7e1/roles/bc3c3c9d654940cbaa5177a6be23d1bc => generated 0 bytes in 77 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) May 23 13:01:34.213514 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b37b6e0-c8d7-4eb3-974f-3d38edf3c266 None None] REQUEST_METHOD: `POST` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:01:34.214727 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b37b6e0-c8d7-4eb3-974f-3d38edf3c266 None None] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:01:34.215306 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b37b6e0-c8d7-4eb3-974f-3d38edf3c266 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:01:34.251998 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.auth.core [None req-3b37b6e0-c8d7-4eb3-974f-3d38edf3c266 None None] MFA Rules not processed for user `e456ac07a4484a96a2ac7a4d6f07f7e1`. Rule list: `[]` (Enabled: `True`). {{(pid=92056) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 13:01:34.253250 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.fernet_utils [None req-3b37b6e0-c8d7-4eb3-974f-3d38edf3c266 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92056) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:01:34.375103 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 1087/2184] 10.222.0.30 () {58 vars in 957 bytes} [Mon May 23 13:01:34 2022] POST /identity/v3/auth/tokens => generated 3301 bytes in 163 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) May 23 13:01:34.389698 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d73b57ab-f467-4ff2-9504-f8bd9c5f890c admin admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:01:34.391592 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.fernet_utils [None req-d73b57ab-f467-4ff2-9504-f8bd9c5f890c admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92057) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:01:34.491854 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d73b57ab-f467-4ff2-9504-f8bd9c5f890c admin admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:01:34.496790 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1fe8bbbe-e617-43c4-9ace-98f56775a45e service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1666097f8ed545c6b0e4809c93e4775a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9c389ae65aaa4542bb5a5c4fd0bacf5b', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:01:34.497444 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fe8bbbe-e617-43c4-9ace-98f56775a45e service cinder] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:01:34.497729 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fe8bbbe-e617-43c4-9ace-98f56775a45e service cinder] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:01:34.498056 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fe8bbbe-e617-43c4-9ace-98f56775a45e service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:01:34.507784 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1fe8bbbe-e617-43c4-9ace-98f56775a45e service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:01:34.509228 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1fe8bbbe-e617-43c4-9ace-98f56775a45e service cinder] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:01:34.532333 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 1098/2185] 10.222.0.30 () {60 vars in 1365 bytes} [Mon May 23 13:01:34 2022] GET /identity/v3/auth/tokens => generated 3301 bytes in 143 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) May 23 13:01:34.837550 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:INSERT=4,keystone:DELETE=3,keystone:SELECT=93 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 13:01:35.016868 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=95,keystone:DELETE=5 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 13:01:35.181465 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b37b6e0-c8d7-4eb3-974f-3d38edf3c266 None None] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:01:35.183179 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.fernet_utils [None req-3b37b6e0-c8d7-4eb3-974f-3d38edf3c266 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92056) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:01:35.432127 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b37b6e0-c8d7-4eb3-974f-3d38edf3c266 None None] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:01:35.439278 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65b049a9-7b0d-4095-9975-2084b712b584 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:01:35.440248 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65b049a9-7b0d-4095-9975-2084b712b584 admin admin] REQUEST_METHOD: `DELETE` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:01:35.443727 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65b049a9-7b0d-4095-9975-2084b712b584 admin admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:01:35.444185 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65b049a9-7b0d-4095-9975-2084b712b584 admin admin] PATH_INFO: `/v3/users/e54f105d7dcc4dc993ca024fe3a1faf1` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:01:35.446095 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65b049a9-7b0d-4095-9975-2084b712b584 admin admin] RBAC: Authorizing `identity:delete_user(user_id=e54f105d7dcc4dc993ca024fe3a1faf1)` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:01:35.450278 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65b049a9-7b0d-4095-9975-2084b712b584 admin admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:01:35.724626 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-65b049a9-7b0d-4095-9975-2084b712b584 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'e54f105d7dcc4dc993ca024fe3a1faf1'} {{(pid=92056) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:01:35.725761 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-65b049a9-7b0d-4095-9975-2084b712b584 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'e54f105d7dcc4dc993ca024fe3a1faf1'} {{(pid=92056) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:01:35.834916 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-65b049a9-7b0d-4095-9975-2084b712b584 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'e54f105d7dcc4dc993ca024fe3a1faf1'} {{(pid=92056) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:01:35.847857 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-65b049a9-7b0d-4095-9975-2084b712b584 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'e54f105d7dcc4dc993ca024fe3a1faf1'} {{(pid=92056) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:01:35.854900 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-65b049a9-7b0d-4095-9975-2084b712b584 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'e54f105d7dcc4dc993ca024fe3a1faf1'} {{(pid=92056) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:01:35.861142 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 1088/2186] 10.222.0.30 () {58 vars in 1223 bytes} [Mon May 23 13:01:35 2022] DELETE /identity/v3/users/e54f105d7dcc4dc993ca024fe3a1faf1 => generated 0 bytes in 680 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) May 23 13:01:35.865744 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24847787-42f7-470f-ab19-63ecc6bf1bd9 None None] REQUEST_METHOD: `POST` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:01:35.866109 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24847787-42f7-470f-ab19-63ecc6bf1bd9 None None] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:01:35.866406 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24847787-42f7-470f-ab19-63ecc6bf1bd9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:01:35.909222 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.auth.core [None req-24847787-42f7-470f-ab19-63ecc6bf1bd9 None None] MFA Rules not processed for user `0f7119f8fa5c4445977671140ded91a4`. Rule list: `[]` (Enabled: `True`). {{(pid=92057) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 13:01:35.910643 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.fernet_utils [None req-24847787-42f7-470f-ab19-63ecc6bf1bd9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92057) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:01:35.984819 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 1099/2187] 10.222.0.30 () {58 vars in 957 bytes} [Mon May 23 13:01:35 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 120 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) May 23 13:01:36.005549 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65b049a9-7b0d-4095-9975-2084b712b584 admin admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:01:36.010735 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.fernet_utils [None req-65b049a9-7b0d-4095-9975-2084b712b584 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92056) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:01:36.157921 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65b049a9-7b0d-4095-9975-2084b712b584 admin admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:01:36.161987 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2be0bd74-2b59-401e-ad0a-f646257a78b2 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2eb83ca38e79449095f3a2f947bbb06a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9c389ae65aaa4542bb5a5c4fd0bacf5b', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:01:36.163681 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2be0bd74-2b59-401e-ad0a-f646257a78b2 service neutron] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:01:36.164064 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2be0bd74-2b59-401e-ad0a-f646257a78b2 service neutron] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:01:36.164435 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2be0bd74-2b59-401e-ad0a-f646257a78b2 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:01:36.180546 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2be0bd74-2b59-401e-ad0a-f646257a78b2 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:01:36.180546 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2be0bd74-2b59-401e-ad0a-f646257a78b2 service neutron] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:01:36.190237 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 1089/2188] 10.222.0.30 () {60 vars in 1365 bytes} [Mon May 23 13:01:36 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 185 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) May 23 13:01:37.095789 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24847787-42f7-470f-ab19-63ecc6bf1bd9 None None] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:01:37.097966 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.fernet_utils [None req-24847787-42f7-470f-ab19-63ecc6bf1bd9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92057) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:01:37.100453 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=94,keystone:DELETE=5,keystone:INSERT=1 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 13:01:37.124329 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24847787-42f7-470f-ab19-63ecc6bf1bd9 None None] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:01:37.129839 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb0b319c-7051-497b-b665-ef16b3efe87e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:01:37.131031 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb0b319c-7051-497b-b665-ef16b3efe87e admin admin] REQUEST_METHOD: `DELETE` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:01:37.139396 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb0b319c-7051-497b-b665-ef16b3efe87e admin admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:01:37.139396 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb0b319c-7051-497b-b665-ef16b3efe87e admin admin] PATH_INFO: `/v3/projects/ec74e160a3f74c11b1d25cedcd9be911` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:01:37.146510 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb0b319c-7051-497b-b665-ef16b3efe87e admin admin] RBAC: Authorizing `identity:delete_project(project_id=ec74e160a3f74c11b1d25cedcd9be911)` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:01:37.148468 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb0b319c-7051-497b-b665-ef16b3efe87e admin admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:01:37.306144 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [None req-fb0b319c-7051-497b-b665-ef16b3efe87e admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'ec74e160a3f74c11b1d25cedcd9be911', 'request_id': 'req-fb0b319c-7051-497b-b665-ef16b3efe87e'} {{(pid=92057) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:01:37.313549 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [None req-fb0b319c-7051-497b-b665-ef16b3efe87e admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'ec74e160a3f74c11b1d25cedcd9be911', 'request_id': 'req-fb0b319c-7051-497b-b665-ef16b3efe87e'} {{(pid=92057) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:01:37.402896 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [None req-fb0b319c-7051-497b-b665-ef16b3efe87e admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'ec74e160a3f74c11b1d25cedcd9be911', 'request_id': 'req-fb0b319c-7051-497b-b665-ef16b3efe87e'} {{(pid=92057) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:01:37.417616 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [None req-fb0b319c-7051-497b-b665-ef16b3efe87e admin admin] The token cache is being invalidate because project ec74e160a3f74c11b1d25cedcd9be911 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=92057) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} May 23 13:01:37.417960 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [None req-fb0b319c-7051-497b-b665-ef16b3efe87e admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=92057) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:01:37.418723 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [None req-fb0b319c-7051-497b-b665-ef16b3efe87e admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=92057) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:01:37.420905 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 1100/2189] 10.222.0.30 () {58 vars in 1232 bytes} [Mon May 23 13:01:37 2022] DELETE /identity/v3/projects/ec74e160a3f74c11b1d25cedcd9be911 => generated 0 bytes in 326 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) May 23 13:01:37.426005 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-666fbd52-90dd-47aa-9600-4f7214d7b2ee None None] REQUEST_METHOD: `POST` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:01:37.426493 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-666fbd52-90dd-47aa-9600-4f7214d7b2ee None None] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:01:37.426493 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-666fbd52-90dd-47aa-9600-4f7214d7b2ee None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:01:37.500531 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.auth.core [None req-666fbd52-90dd-47aa-9600-4f7214d7b2ee None None] MFA Rules not processed for user `0f7119f8fa5c4445977671140ded91a4`. Rule list: `[]` (Enabled: `True`). {{(pid=92056) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 13:01:37.501757 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.fernet_utils [None req-666fbd52-90dd-47aa-9600-4f7214d7b2ee None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92056) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:01:37.634582 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 1090/2190] 10.222.0.30 () {58 vars in 957 bytes} [Mon May 23 13:01:37 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 210 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) May 23 13:01:37.635431 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1dc32e8d-dfa1-4adf-8be0-297af77319b3 None None] REQUEST_METHOD: `POST` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:01:37.635936 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1dc32e8d-dfa1-4adf-8be0-297af77319b3 None None] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:01:37.636295 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1dc32e8d-dfa1-4adf-8be0-297af77319b3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:01:37.682189 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.auth.core [None req-1dc32e8d-dfa1-4adf-8be0-297af77319b3 None None] MFA Rules not processed for user `0f7119f8fa5c4445977671140ded91a4`. Rule list: `[]` (Enabled: `True`). {{(pid=92057) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 13:01:37.684901 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.fernet_utils [None req-1dc32e8d-dfa1-4adf-8be0-297af77319b3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92057) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:01:37.714438 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 1101/2191] 10.222.0.30 () {58 vars in 957 bytes} [Mon May 23 13:01:37 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 80 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) May 23 13:01:37.718516 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-666fbd52-90dd-47aa-9600-4f7214d7b2ee None None] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:01:37.744733 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-666fbd52-90dd-47aa-9600-4f7214d7b2ee None None] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:01:37.764967 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b2059b1-0508-4ed8-9bf1-3a12095249fe admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:01:37.765820 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b2059b1-0508-4ed8-9bf1-3a12095249fe admin admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:01:37.766200 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b2059b1-0508-4ed8-9bf1-3a12095249fe admin admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:01:37.766508 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b2059b1-0508-4ed8-9bf1-3a12095249fe admin admin] PATH_INFO: `/v3/domains` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:01:37.767328 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b2059b1-0508-4ed8-9bf1-3a12095249fe admin admin] RBAC: Adding query filter params (name=Default) {{(pid=92056) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} May 23 13:01:37.767638 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b2059b1-0508-4ed8-9bf1-3a12095249fe admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:01:37.781669 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b2059b1-0508-4ed8-9bf1-3a12095249fe admin admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:01:37.787305 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 13:01:37.814611 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 1091/2192] 10.222.0.30 () {58 vars in 1152 bytes} [Mon May 23 13:01:37 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 97 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) May 23 13:01:37.824335 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1dc32e8d-dfa1-4adf-8be0-297af77319b3 None None] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:01:37.840659 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1dc32e8d-dfa1-4adf-8be0-297af77319b3 None None] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:01:37.852077 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e47fd3fb-a019-4a0f-9233-894b39795eb1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:01:37.852815 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e47fd3fb-a019-4a0f-9233-894b39795eb1 admin admin] REQUEST_METHOD: `POST` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:01:37.853039 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e47fd3fb-a019-4a0f-9233-894b39795eb1 admin admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:01:37.853305 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e47fd3fb-a019-4a0f-9233-894b39795eb1 admin admin] PATH_INFO: `/v3/projects` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:01:37.854276 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e47fd3fb-a019-4a0f-9233-894b39795eb1 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:01:37.862980 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e47fd3fb-a019-4a0f-9233-894b39795eb1 admin admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:01:37.920552 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40882fb3-c8b2-4e79-860e-e10954e32374 None None] REQUEST_METHOD: `POST` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:01:37.920844 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40882fb3-c8b2-4e79-860e-e10954e32374 None None] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:01:37.921121 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40882fb3-c8b2-4e79-860e-e10954e32374 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:01:37.949094 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.auth.core [None req-40882fb3-c8b2-4e79-860e-e10954e32374 None None] MFA Rules not processed for user `f78ab5496c1e4008b83fa3866601c06c`. Rule list: `[]` (Enabled: `True`). {{(pid=92056) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 13:01:37.950108 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.fernet_utils [None req-40882fb3-c8b2-4e79-860e-e10954e32374 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92056) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:01:37.978874 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 1102/2193] 10.222.0.30 () {60 vars in 1152 bytes} [Mon May 23 13:01:37 2022] POST /identity/v3/projects => generated 373 bytes in 155 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) May 23 13:01:37.987649 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e47fd3fb-a019-4a0f-9233-894b39795eb1 admin admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:01:38.049967 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 1092/2194] 10.222.0.30 () {58 vars in 957 bytes} [Mon May 23 13:01:37 2022] POST /identity/v3/auth/tokens => generated 3254 bytes in 123 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) May 23 13:01:38.058023 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40882fb3-c8b2-4e79-860e-e10954e32374 None None] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:01:38.059653 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.fernet_utils [None req-40882fb3-c8b2-4e79-860e-e10954e32374 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92056) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:01:38.074421 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e47fd3fb-a019-4a0f-9233-894b39795eb1 admin admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:01:38.079428 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-928296cd-a73c-49ff-b06c-ebf70db6baa8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:01:38.080097 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-928296cd-a73c-49ff-b06c-ebf70db6baa8 admin admin] REQUEST_METHOD: `POST` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:01:38.080382 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-928296cd-a73c-49ff-b06c-ebf70db6baa8 admin admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:01:38.080809 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-928296cd-a73c-49ff-b06c-ebf70db6baa8 admin admin] PATH_INFO: `/v3/users` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:01:38.081560 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-928296cd-a73c-49ff-b06c-ebf70db6baa8 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:01:38.083570 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-928296cd-a73c-49ff-b06c-ebf70db6baa8 admin admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:01:38.135348 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40882fb3-c8b2-4e79-860e-e10954e32374 None None] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:01:38.205621 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 1103/2195] 10.222.0.30 () {60 vars in 1143 bytes} [Mon May 23 13:01:37 2022] POST /identity/v3/users => generated 287 bytes in 219 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) May 23 13:01:38.211325 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-928296cd-a73c-49ff-b06c-ebf70db6baa8 admin admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:01:38.235172 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-928296cd-a73c-49ff-b06c-ebf70db6baa8 admin admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:01:38.250257 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f22c8e0-c6d5-482a-a9dc-d1a3b74fad66 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:01:38.250449 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff29dba2-ec91-49f1-b140-97c458e07562 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cbe9931d94d44cc7ace0e83a7e5015fc', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9c389ae65aaa4542bb5a5c4fd0bacf5b', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'member', 'reader', 'service', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:01:38.251019 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff29dba2-ec91-49f1-b140-97c458e07562 service nova] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:01:38.251281 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff29dba2-ec91-49f1-b140-97c458e07562 service nova] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:01:38.252113 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff29dba2-ec91-49f1-b140-97c458e07562 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:01:38.253019 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f22c8e0-c6d5-482a-a9dc-d1a3b74fad66 admin admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:01:38.253491 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f22c8e0-c6d5-482a-a9dc-d1a3b74fad66 admin admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:01:38.254086 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f22c8e0-c6d5-482a-a9dc-d1a3b74fad66 admin admin] PATH_INFO: `/v3/roles` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:01:38.255491 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f22c8e0-c6d5-482a-a9dc-d1a3b74fad66 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:01:38.257552 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f22c8e0-c6d5-482a-a9dc-d1a3b74fad66 admin admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:01:38.273025 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 1104/2196] 10.222.0.30 () {58 vars in 1121 bytes} [Mon May 23 13:01:38 2022] GET /identity/v3/roles => generated 1420 bytes in 63 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) May 23 13:01:38.278721 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f22c8e0-c6d5-482a-a9dc-d1a3b74fad66 admin admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:01:38.300145 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f22c8e0-c6d5-482a-a9dc-d1a3b74fad66 admin admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:01:38.306743 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1bf349f3-1356-4312-b114-f4bbd4b10fb4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:01:38.307541 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bf349f3-1356-4312-b114-f4bbd4b10fb4 admin admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:01:38.308455 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bf349f3-1356-4312-b114-f4bbd4b10fb4 admin admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:01:38.308922 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bf349f3-1356-4312-b114-f4bbd4b10fb4 admin admin] PATH_INFO: `/v3/roles` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:01:38.310041 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1bf349f3-1356-4312-b114-f4bbd4b10fb4 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:01:38.313115 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1bf349f3-1356-4312-b114-f4bbd4b10fb4 admin admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:01:38.327910 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 1105/2197] 10.222.0.30 () {58 vars in 1121 bytes} [Mon May 23 13:01:38 2022] GET /identity/v3/roles => generated 1420 bytes in 50 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) May 23 13:01:38.342272 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1bf349f3-1356-4312-b114-f4bbd4b10fb4 admin admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:01:38.355177 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1bf349f3-1356-4312-b114-f4bbd4b10fb4 admin admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:01:38.361629 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db558f81-e0f6-484b-bb67-b6331f9c4d42 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:01:38.362841 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db558f81-e0f6-484b-bb67-b6331f9c4d42 admin admin] REQUEST_METHOD: `PUT` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:01:38.363444 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db558f81-e0f6-484b-bb67-b6331f9c4d42 admin admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:01:38.363869 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db558f81-e0f6-484b-bb67-b6331f9c4d42 admin admin] PATH_INFO: `/v3/projects/26397036fab54883aea8a65f83190947/users/01b4e4c7d54c4294ae72f9bd5f9420e1/roles/80fb22507c9a4ff09d44d8f12f6fff1a` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:01:38.384580 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db558f81-e0f6-484b-bb67-b6331f9c4d42 admin admin] RBAC: Authorizing `identity:create_grant(project_id=26397036fab54883aea8a65f83190947, user_id=01b4e4c7d54c4294ae72f9bd5f9420e1, role_id=80fb22507c9a4ff09d44d8f12f6fff1a)` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:01:38.390943 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db558f81-e0f6-484b-bb67-b6331f9c4d42 admin admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:01:38.418317 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff29dba2-ec91-49f1-b140-97c458e07562 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:01:38.420963 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff29dba2-ec91-49f1-b140-97c458e07562 service nova] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:01:38.444370 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 1093/2198] 10.222.0.30 () {60 vars in 1365 bytes} [Mon May 23 13:01:38 2022] GET /identity/v3/auth/tokens => generated 3254 bytes in 387 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) May 23 13:01:38.510375 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 1106/2199] 10.222.0.30 () {60 vars in 1482 bytes} [Mon May 23 13:01:38 2022] PUT /identity/v3/projects/26397036fab54883aea8a65f83190947/users/01b4e4c7d54c4294ae72f9bd5f9420e1/roles/80fb22507c9a4ff09d44d8f12f6fff1a => generated 0 bytes in 161 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) May 23 13:01:38.517864 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3dba816d-1357-40f2-ba06-8d65d6657409 None None] REQUEST_METHOD: `POST` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:01:38.518669 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3dba816d-1357-40f2-ba06-8d65d6657409 None None] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:01:38.519225 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3dba816d-1357-40f2-ba06-8d65d6657409 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:01:38.565205 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.auth.core [None req-3dba816d-1357-40f2-ba06-8d65d6657409 None None] MFA Rules not processed for user `01b4e4c7d54c4294ae72f9bd5f9420e1`. Rule list: `[]` (Enabled: `True`). {{(pid=92056) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 13:01:38.566742 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.fernet_utils [None req-3dba816d-1357-40f2-ba06-8d65d6657409 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92056) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:01:38.596773 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db558f81-e0f6-484b-bb67-b6331f9c4d42 admin admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:01:38.599186 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.fernet_utils [None req-db558f81-e0f6-484b-bb67-b6331f9c4d42 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92057) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:01:38.733740 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 1094/2200] 10.222.0.30 () {58 vars in 957 bytes} [Mon May 23 13:01:38 2022] POST /identity/v3/auth/tokens => generated 3240 bytes in 218 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) May 23 13:01:38.739162 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3dba816d-1357-40f2-ba06-8d65d6657409 None None] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:01:38.763369 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db558f81-e0f6-484b-bb67-b6331f9c4d42 admin admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:01:38.776940 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ede851e-d96e-40a7-9cc5-3602df0495a1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:01:38.778010 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ede851e-d96e-40a7-9cc5-3602df0495a1 admin admin] REQUEST_METHOD: `DELETE` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:01:38.778392 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ede851e-d96e-40a7-9cc5-3602df0495a1 admin admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:01:38.778824 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ede851e-d96e-40a7-9cc5-3602df0495a1 admin admin] PATH_INFO: `/v3/users/af36d5ec94e444a1a903514d9799f687` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:01:38.780814 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ede851e-d96e-40a7-9cc5-3602df0495a1 admin admin] RBAC: Authorizing `identity:delete_user(user_id=af36d5ec94e444a1a903514d9799f687)` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:01:38.793239 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ede851e-d96e-40a7-9cc5-3602df0495a1 admin admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:01:38.837221 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3dba816d-1357-40f2-ba06-8d65d6657409 None None] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:01:38.844933 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b040d27-2e44-41f3-93b9-cd4fa908bfaf admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:01:38.852226 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b040d27-2e44-41f3-93b9-cd4fa908bfaf admin admin] REQUEST_METHOD: `POST` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:01:38.852226 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b040d27-2e44-41f3-93b9-cd4fa908bfaf admin admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:01:38.852226 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b040d27-2e44-41f3-93b9-cd4fa908bfaf admin admin] PATH_INFO: `/v3/projects` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:01:38.852226 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b040d27-2e44-41f3-93b9-cd4fa908bfaf admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:01:38.852226 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b040d27-2e44-41f3-93b9-cd4fa908bfaf admin admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:01:38.941195 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 1095/2201] 10.222.0.30 () {60 vars in 1152 bytes} [Mon May 23 13:01:38 2022] POST /identity/v3/projects => generated 373 bytes in 202 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) May 23 13:01:38.950709 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b040d27-2e44-41f3-93b9-cd4fa908bfaf admin admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:01:39.021275 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b040d27-2e44-41f3-93b9-cd4fa908bfaf admin admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:01:39.028144 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ccbf0b25-6d67-403e-aa97-c53e99aac1ed admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:01:39.028929 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccbf0b25-6d67-403e-aa97-c53e99aac1ed admin admin] REQUEST_METHOD: `POST` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:01:39.029279 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccbf0b25-6d67-403e-aa97-c53e99aac1ed admin admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:01:39.029776 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccbf0b25-6d67-403e-aa97-c53e99aac1ed admin admin] PATH_INFO: `/v3/users` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:01:39.030833 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ccbf0b25-6d67-403e-aa97-c53e99aac1ed admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:01:39.033088 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ccbf0b25-6d67-403e-aa97-c53e99aac1ed admin admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:01:39.063753 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [None req-5ede851e-d96e-40a7-9cc5-3602df0495a1 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'af36d5ec94e444a1a903514d9799f687'} {{(pid=92057) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:01:39.166364 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [None req-5ede851e-d96e-40a7-9cc5-3602df0495a1 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'af36d5ec94e444a1a903514d9799f687'} {{(pid=92057) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:01:39.173097 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:DELETE=12,keystone:SELECT=87,keystone:INSERT=1 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 13:01:39.175035 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 1096/2202] 10.222.0.30 () {60 vars in 1143 bytes} [Mon May 23 13:01:38 2022] POST /identity/v3/users => generated 293 bytes in 224 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) May 23 13:01:39.186227 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [None req-5ede851e-d96e-40a7-9cc5-3602df0495a1 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'af36d5ec94e444a1a903514d9799f687'} {{(pid=92057) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:01:39.186996 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [None req-5ede851e-d96e-40a7-9cc5-3602df0495a1 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'af36d5ec94e444a1a903514d9799f687'} {{(pid=92057) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:01:39.187832 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ccbf0b25-6d67-403e-aa97-c53e99aac1ed admin admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:01:39.205295 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [None req-5ede851e-d96e-40a7-9cc5-3602df0495a1 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'af36d5ec94e444a1a903514d9799f687'} {{(pid=92057) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:01:39.206376 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ccbf0b25-6d67-403e-aa97-c53e99aac1ed admin admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:01:39.212149 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 1107/2203] 10.222.0.30 () {58 vars in 1223 bytes} [Mon May 23 13:01:38 2022] DELETE /identity/v3/users/af36d5ec94e444a1a903514d9799f687 => generated 0 bytes in 616 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) May 23 13:01:39.212636 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa4513ff-e80f-4b5f-8e0b-d65f77915b13 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:01:39.213421 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa4513ff-e80f-4b5f-8e0b-d65f77915b13 admin admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:01:39.214542 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa4513ff-e80f-4b5f-8e0b-d65f77915b13 admin admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:01:39.215115 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa4513ff-e80f-4b5f-8e0b-d65f77915b13 admin admin] PATH_INFO: `/v3/roles` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:01:39.216130 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa4513ff-e80f-4b5f-8e0b-d65f77915b13 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:01:39.222454 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b072e37-4611-4164-822f-cb1659f9b44c None None] REQUEST_METHOD: `POST` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:01:39.222454 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b072e37-4611-4164-822f-cb1659f9b44c None None] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:01:39.222454 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b072e37-4611-4164-822f-cb1659f9b44c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:01:39.222765 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa4513ff-e80f-4b5f-8e0b-d65f77915b13 admin admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:01:39.231120 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 1097/2204] 10.222.0.30 () {58 vars in 1121 bytes} [Mon May 23 13:01:39 2022] GET /identity/v3/roles => generated 1420 bytes in 48 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) May 23 13:01:39.235385 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa4513ff-e80f-4b5f-8e0b-d65f77915b13 admin admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:01:39.235512 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.auth.core [None req-7b072e37-4611-4164-822f-cb1659f9b44c None None] MFA Rules not processed for user `0f7119f8fa5c4445977671140ded91a4`. Rule list: `[]` (Enabled: `True`). {{(pid=92057) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 13:01:39.236745 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.fernet_utils [None req-7b072e37-4611-4164-822f-cb1659f9b44c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92057) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:01:39.237363 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.fernet_utils [None req-fa4513ff-e80f-4b5f-8e0b-d65f77915b13 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92056) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:01:39.243486 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=100 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 13:01:39.251793 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa4513ff-e80f-4b5f-8e0b-d65f77915b13 admin admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:01:39.348682 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 1108/2205] 10.222.0.30 () {58 vars in 957 bytes} [Mon May 23 13:01:39 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 132 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) May 23 13:01:39.354724 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d252f98-64b5-4d02-abb1-854c64dc8372 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:01:39.356501 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d252f98-64b5-4d02-abb1-854c64dc8372 admin admin] REQUEST_METHOD: `PUT` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:01:39.356943 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d252f98-64b5-4d02-abb1-854c64dc8372 admin admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:01:39.357654 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d252f98-64b5-4d02-abb1-854c64dc8372 admin admin] PATH_INFO: `/v3/projects/5126b2bfeaf94cc29ec9745c48bcfbfb/users/28c86932778040d38f208123514d32de/roles/bc3c3c9d654940cbaa5177a6be23d1bc` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:01:39.359320 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b072e37-4611-4164-822f-cb1659f9b44c None None] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:01:39.361663 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.fernet_utils [None req-7b072e37-4611-4164-822f-cb1659f9b44c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92057) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:01:39.376975 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d252f98-64b5-4d02-abb1-854c64dc8372 admin admin] RBAC: Authorizing `identity:create_grant(project_id=5126b2bfeaf94cc29ec9745c48bcfbfb, user_id=28c86932778040d38f208123514d32de, role_id=bc3c3c9d654940cbaa5177a6be23d1bc)` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:01:39.379853 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d252f98-64b5-4d02-abb1-854c64dc8372 admin admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:01:39.436693 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b072e37-4611-4164-822f-cb1659f9b44c None None] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:01:39.440621 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6a24d77-b82c-464e-8deb-75ff47c44511 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2eb83ca38e79449095f3a2f947bbb06a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9c389ae65aaa4542bb5a5c4fd0bacf5b', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:01:39.441280 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6a24d77-b82c-464e-8deb-75ff47c44511 service neutron] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:01:39.441567 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6a24d77-b82c-464e-8deb-75ff47c44511 service neutron] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:01:39.441855 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6a24d77-b82c-464e-8deb-75ff47c44511 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:01:39.452252 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6a24d77-b82c-464e-8deb-75ff47c44511 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:01:39.453706 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6a24d77-b82c-464e-8deb-75ff47c44511 service neutron] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:01:39.463477 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 1109/2206] 10.222.0.30 () {60 vars in 1365 bytes} [Mon May 23 13:01:39 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 105 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) May 23 13:01:39.481938 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 1098/2207] 10.222.0.30 () {60 vars in 1482 bytes} [Mon May 23 13:01:39 2022] PUT /identity/v3/projects/5126b2bfeaf94cc29ec9745c48bcfbfb/users/28c86932778040d38f208123514d32de/roles/bc3c3c9d654940cbaa5177a6be23d1bc => generated 0 bytes in 247 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) May 23 13:01:39.488265 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6a24d77-b82c-464e-8deb-75ff47c44511 service neutron] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:01:39.508696 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6a24d77-b82c-464e-8deb-75ff47c44511 service neutron] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:01:39.599342 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4205f431-c7b4-4ad2-a8ae-5a3aee6f27c7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:01:39.600028 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4205f431-c7b4-4ad2-a8ae-5a3aee6f27c7 admin admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:01:39.600322 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4205f431-c7b4-4ad2-a8ae-5a3aee6f27c7 admin admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:01:39.600595 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4205f431-c7b4-4ad2-a8ae-5a3aee6f27c7 admin admin] PATH_INFO: `/v3/roles` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:01:39.601819 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4205f431-c7b4-4ad2-a8ae-5a3aee6f27c7 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:01:39.604312 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4205f431-c7b4-4ad2-a8ae-5a3aee6f27c7 admin admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:01:39.624352 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 1110/2208] 10.222.0.30 () {58 vars in 1121 bytes} [Mon May 23 13:01:39 2022] GET /identity/v3/roles => generated 1420 bytes in 135 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) May 23 13:01:39.626478 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d252f98-64b5-4d02-abb1-854c64dc8372 admin admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:01:39.652799 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d252f98-64b5-4d02-abb1-854c64dc8372 admin admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:01:39.670499 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d52d92f6-3023-4043-bbf1-28835cf45490 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:01:39.672027 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d52d92f6-3023-4043-bbf1-28835cf45490 admin admin] REQUEST_METHOD: `PUT` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:01:39.672619 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d52d92f6-3023-4043-bbf1-28835cf45490 admin admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:01:39.673145 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d52d92f6-3023-4043-bbf1-28835cf45490 admin admin] PATH_INFO: `/v3/projects/5126b2bfeaf94cc29ec9745c48bcfbfb/users/28c86932778040d38f208123514d32de/roles/bc3c3c9d654940cbaa5177a6be23d1bc` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:01:39.675664 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d52d92f6-3023-4043-bbf1-28835cf45490 admin admin] RBAC: Authorizing `identity:create_grant(project_id=5126b2bfeaf94cc29ec9745c48bcfbfb, user_id=28c86932778040d38f208123514d32de, role_id=bc3c3c9d654940cbaa5177a6be23d1bc)` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:01:39.683450 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d52d92f6-3023-4043-bbf1-28835cf45490 admin admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:01:39.719669 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 1099/2209] 10.222.0.30 () {60 vars in 1482 bytes} [Mon May 23 13:01:39 2022] PUT /identity/v3/projects/5126b2bfeaf94cc29ec9745c48bcfbfb/users/28c86932778040d38f208123514d32de/roles/bc3c3c9d654940cbaa5177a6be23d1bc => generated 0 bytes in 94 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) May 23 13:01:39.733296 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1dd3a321-0e6b-40d8-abad-3d96200e89ea None None] REQUEST_METHOD: `POST` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:01:39.733592 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1dd3a321-0e6b-40d8-abad-3d96200e89ea None None] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:01:39.733898 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1dd3a321-0e6b-40d8-abad-3d96200e89ea None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:01:39.784168 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.auth.core [None req-1dd3a321-0e6b-40d8-abad-3d96200e89ea None None] MFA Rules not processed for user `28c86932778040d38f208123514d32de`. Rule list: `[]` (Enabled: `True`). {{(pid=92057) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 13:01:39.787519 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.fernet_utils [None req-1dd3a321-0e6b-40d8-abad-3d96200e89ea None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92057) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:01:39.870338 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:DELETE=10,keystone:SELECT=88,keystone:INSERT=2 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 13:01:39.903870 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 1111/2210] 10.222.0.30 () {58 vars in 957 bytes} [Mon May 23 13:01:39 2022] POST /identity/v3/auth/tokens => generated 3307 bytes in 172 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) May 23 13:01:39.915954 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d52d92f6-3023-4043-bbf1-28835cf45490 admin admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:01:39.917965 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.fernet_utils [None req-d52d92f6-3023-4043-bbf1-28835cf45490 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92056) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:01:40.056767 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d52d92f6-3023-4043-bbf1-28835cf45490 admin admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:01:40.068580 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53635186-9518-4669-ad20-a55b37859f15 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1666097f8ed545c6b0e4809c93e4775a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9c389ae65aaa4542bb5a5c4fd0bacf5b', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:01:40.072766 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53635186-9518-4669-ad20-a55b37859f15 service cinder] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:01:40.073078 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53635186-9518-4669-ad20-a55b37859f15 service cinder] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:01:40.074775 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53635186-9518-4669-ad20-a55b37859f15 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:01:40.077396 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.fernet_utils [None req-53635186-9518-4669-ad20-a55b37859f15 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92056) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:01:40.138519 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53635186-9518-4669-ad20-a55b37859f15 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:01:40.140411 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53635186-9518-4669-ad20-a55b37859f15 service cinder] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:01:40.151086 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1dd3a321-0e6b-40d8-abad-3d96200e89ea None None] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:01:40.151086 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.fernet_utils [None req-1dd3a321-0e6b-40d8-abad-3d96200e89ea None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92057) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:01:40.151647 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 1100/2211] 10.222.0.30 () {60 vars in 1365 bytes} [Mon May 23 13:01:39 2022] GET /identity/v3/auth/tokens => generated 3240 bytes in 236 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) May 23 13:01:40.238377 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1dd3a321-0e6b-40d8-abad-3d96200e89ea None None] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:01:40.260677 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-767bcf77-71b7-47b3-ae83-72f2a05ccbf9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:01:40.260677 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-767bcf77-71b7-47b3-ae83-72f2a05ccbf9 admin admin] REQUEST_METHOD: `DELETE` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:01:40.260677 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-767bcf77-71b7-47b3-ae83-72f2a05ccbf9 admin admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:01:40.260677 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-767bcf77-71b7-47b3-ae83-72f2a05ccbf9 admin admin] PATH_INFO: `/v3/projects/9ef27bc16356446eb11ed70c90e7869f` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:01:40.261704 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-767bcf77-71b7-47b3-ae83-72f2a05ccbf9 admin admin] RBAC: Authorizing `identity:delete_project(project_id=9ef27bc16356446eb11ed70c90e7869f)` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:01:40.261704 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-767bcf77-71b7-47b3-ae83-72f2a05ccbf9 admin admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:01:40.464383 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [None req-767bcf77-71b7-47b3-ae83-72f2a05ccbf9 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '9ef27bc16356446eb11ed70c90e7869f', 'request_id': 'req-767bcf77-71b7-47b3-ae83-72f2a05ccbf9'} {{(pid=92057) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:01:40.472746 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [None req-767bcf77-71b7-47b3-ae83-72f2a05ccbf9 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '9ef27bc16356446eb11ed70c90e7869f', 'request_id': 'req-767bcf77-71b7-47b3-ae83-72f2a05ccbf9'} {{(pid=92057) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:01:40.541859 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [None req-767bcf77-71b7-47b3-ae83-72f2a05ccbf9 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '9ef27bc16356446eb11ed70c90e7869f', 'request_id': 'req-767bcf77-71b7-47b3-ae83-72f2a05ccbf9'} {{(pid=92057) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:01:40.560782 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [None req-767bcf77-71b7-47b3-ae83-72f2a05ccbf9 admin admin] The token cache is being invalidate because project 9ef27bc16356446eb11ed70c90e7869f was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=92057) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} May 23 13:01:40.561505 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [None req-767bcf77-71b7-47b3-ae83-72f2a05ccbf9 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=92057) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:01:40.562771 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [None req-767bcf77-71b7-47b3-ae83-72f2a05ccbf9 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=92057) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:01:40.566597 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 1112/2212] 10.222.0.30 () {58 vars in 1232 bytes} [Mon May 23 13:01:40 2022] DELETE /identity/v3/projects/9ef27bc16356446eb11ed70c90e7869f => generated 0 bytes in 421 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) May 23 13:01:40.570550 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53635186-9518-4669-ad20-a55b37859f15 service cinder] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:01:40.582771 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53635186-9518-4669-ad20-a55b37859f15 service cinder] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:01:40.660316 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c39aa793-bbd4-4d91-9e8a-d56fc9fd7f18 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:01:40.661977 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c39aa793-bbd4-4d91-9e8a-d56fc9fd7f18 admin admin] REQUEST_METHOD: `DELETE` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:01:40.662298 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c39aa793-bbd4-4d91-9e8a-d56fc9fd7f18 admin admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:01:40.662622 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c39aa793-bbd4-4d91-9e8a-d56fc9fd7f18 admin admin] PATH_INFO: `/v3/users/e456ac07a4484a96a2ac7a4d6f07f7e1` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:01:40.664663 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c39aa793-bbd4-4d91-9e8a-d56fc9fd7f18 admin admin] RBAC: Authorizing `identity:delete_user(user_id=e456ac07a4484a96a2ac7a4d6f07f7e1)` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:01:40.666765 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c39aa793-bbd4-4d91-9e8a-d56fc9fd7f18 admin admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:01:40.957491 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-c39aa793-bbd4-4d91-9e8a-d56fc9fd7f18 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'e456ac07a4484a96a2ac7a4d6f07f7e1'} {{(pid=92056) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:01:40.958483 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-c39aa793-bbd4-4d91-9e8a-d56fc9fd7f18 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'e456ac07a4484a96a2ac7a4d6f07f7e1'} {{(pid=92056) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:01:41.031665 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-c39aa793-bbd4-4d91-9e8a-d56fc9fd7f18 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'e456ac07a4484a96a2ac7a4d6f07f7e1'} {{(pid=92056) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:01:41.068443 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-c39aa793-bbd4-4d91-9e8a-d56fc9fd7f18 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'e456ac07a4484a96a2ac7a4d6f07f7e1'} {{(pid=92056) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:01:41.079560 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-c39aa793-bbd4-4d91-9e8a-d56fc9fd7f18 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'e456ac07a4484a96a2ac7a4d6f07f7e1'} {{(pid=92056) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:01:41.091451 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 1101/2213] 10.222.0.30 () {58 vars in 1223 bytes} [Mon May 23 13:01:40 2022] DELETE /identity/v3/users/e456ac07a4484a96a2ac7a4d6f07f7e1 => generated 0 bytes in 521 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) May 23 13:01:41.306779 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=76,keystone:DELETE=21,keystone:INSERT=3 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 13:01:41.616613 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-767bcf77-71b7-47b3-ae83-72f2a05ccbf9 admin admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:01:41.618219 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.fernet_utils [None req-767bcf77-71b7-47b3-ae83-72f2a05ccbf9 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92057) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:01:41.651488 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-767bcf77-71b7-47b3-ae83-72f2a05ccbf9 admin admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:01:41.818602 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1133f223-52e3-4ad3-87cd-5c29ebb81241 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:01:41.824398 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1133f223-52e3-4ad3-87cd-5c29ebb81241 admin admin] REQUEST_METHOD: `DELETE` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:01:41.826479 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1133f223-52e3-4ad3-87cd-5c29ebb81241 admin admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:01:41.826781 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1133f223-52e3-4ad3-87cd-5c29ebb81241 admin admin] PATH_INFO: `/v3/projects/dee1db77df764b8da152352ae3b905d7` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:01:41.828521 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1133f223-52e3-4ad3-87cd-5c29ebb81241 admin admin] RBAC: Authorizing `identity:delete_project(project_id=dee1db77df764b8da152352ae3b905d7)` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:01:41.831525 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1133f223-52e3-4ad3-87cd-5c29ebb81241 admin admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:01:42.026563 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 13:01:42.039525 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba115fd6-0984-43e2-9bf1-e52d52d247ce None None] REQUEST_METHOD: `POST` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:01:42.040167 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba115fd6-0984-43e2-9bf1-e52d52d247ce None None] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:01:42.040167 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba115fd6-0984-43e2-9bf1-e52d52d247ce None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:01:42.057662 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [None req-1133f223-52e3-4ad3-87cd-5c29ebb81241 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'dee1db77df764b8da152352ae3b905d7', 'request_id': 'req-1133f223-52e3-4ad3-87cd-5c29ebb81241'} {{(pid=92057) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:01:42.064713 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [None req-1133f223-52e3-4ad3-87cd-5c29ebb81241 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'dee1db77df764b8da152352ae3b905d7', 'request_id': 'req-1133f223-52e3-4ad3-87cd-5c29ebb81241'} {{(pid=92057) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:01:42.075216 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.auth.core [None req-ba115fd6-0984-43e2-9bf1-e52d52d247ce None None] MFA Rules not processed for user `f78ab5496c1e4008b83fa3866601c06c`. Rule list: `[]` (Enabled: `True`). {{(pid=92056) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 13:01:42.076642 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.fernet_utils [None req-ba115fd6-0984-43e2-9bf1-e52d52d247ce None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92056) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:01:42.146489 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [None req-1133f223-52e3-4ad3-87cd-5c29ebb81241 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'dee1db77df764b8da152352ae3b905d7', 'request_id': 'req-1133f223-52e3-4ad3-87cd-5c29ebb81241'} {{(pid=92057) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:01:42.147333 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 1102/2214] 10.222.0.30 () {58 vars in 957 bytes} [Mon May 23 13:01:42 2022] POST /identity/v3/auth/tokens => generated 3254 bytes in 109 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) May 23 13:01:42.162510 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [None req-1133f223-52e3-4ad3-87cd-5c29ebb81241 admin admin] The token cache is being invalidate because project dee1db77df764b8da152352ae3b905d7 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=92057) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} May 23 13:01:42.162900 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [None req-1133f223-52e3-4ad3-87cd-5c29ebb81241 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=92057) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:01:42.163614 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [None req-1133f223-52e3-4ad3-87cd-5c29ebb81241 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=92057) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:01:42.165372 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 1113/2215] 10.222.0.30 () {58 vars in 1232 bytes} [Mon May 23 13:01:41 2022] DELETE /identity/v3/projects/dee1db77df764b8da152352ae3b905d7 => generated 0 bytes in 549 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) May 23 13:01:42.177348 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b15d796-24c6-4f50-b256-7f1b7e79e46f None None] REQUEST_METHOD: `POST` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:01:42.178374 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b15d796-24c6-4f50-b256-7f1b7e79e46f None None] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:01:42.178685 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b15d796-24c6-4f50-b256-7f1b7e79e46f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:01:42.207711 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.auth.core [None req-2b15d796-24c6-4f50-b256-7f1b7e79e46f None None] MFA Rules not processed for user `0f7119f8fa5c4445977671140ded91a4`. Rule list: `[]` (Enabled: `True`). {{(pid=92056) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 13:01:42.208638 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.fernet_utils [None req-2b15d796-24c6-4f50-b256-7f1b7e79e46f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92056) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:01:42.302485 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 1103/2216] 10.222.0.30 () {58 vars in 957 bytes} [Mon May 23 13:01:42 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 127 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) May 23 13:01:42.313148 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50251f01-679a-4663-a62c-1d6ad2cf004f None None] REQUEST_METHOD: `POST` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:01:42.313148 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50251f01-679a-4663-a62c-1d6ad2cf004f None None] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:01:42.313148 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50251f01-679a-4663-a62c-1d6ad2cf004f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:01:42.340875 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.auth.core [None req-50251f01-679a-4663-a62c-1d6ad2cf004f None None] MFA Rules not processed for user `0f7119f8fa5c4445977671140ded91a4`. Rule list: `[]` (Enabled: `True`). {{(pid=92057) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 13:01:42.342273 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.fernet_utils [None req-50251f01-679a-4663-a62c-1d6ad2cf004f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92057) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:01:42.352797 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 1114/2217] 10.222.0.30 () {58 vars in 957 bytes} [Mon May 23 13:01:42 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 41 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) May 23 13:01:42.357523 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b15d796-24c6-4f50-b256-7f1b7e79e46f None None] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:01:42.369080 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b15d796-24c6-4f50-b256-7f1b7e79e46f None None] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:01:42.375946 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90c00fd7-a948-4bba-a3bf-8e4e6de02e43 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:01:42.376984 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90c00fd7-a948-4bba-a3bf-8e4e6de02e43 admin admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:01:42.377555 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90c00fd7-a948-4bba-a3bf-8e4e6de02e43 admin admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:01:42.377990 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90c00fd7-a948-4bba-a3bf-8e4e6de02e43 admin admin] PATH_INFO: `/v3/domains` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:01:42.378850 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90c00fd7-a948-4bba-a3bf-8e4e6de02e43 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=92056) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} May 23 13:01:42.379286 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90c00fd7-a948-4bba-a3bf-8e4e6de02e43 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:01:42.382312 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90c00fd7-a948-4bba-a3bf-8e4e6de02e43 admin admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:01:42.402159 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 1104/2218] 10.222.0.30 () {58 vars in 1152 bytes} [Mon May 23 13:01:42 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 45 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) May 23 13:01:42.406856 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50251f01-679a-4663-a62c-1d6ad2cf004f None None] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:01:42.419639 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50251f01-679a-4663-a62c-1d6ad2cf004f None None] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:01:42.426508 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ead07d71-e980-464e-9bbf-46dfe30a02eb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:01:42.427279 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ead07d71-e980-464e-9bbf-46dfe30a02eb admin admin] REQUEST_METHOD: `POST` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:01:42.427636 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ead07d71-e980-464e-9bbf-46dfe30a02eb admin admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:01:42.427989 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ead07d71-e980-464e-9bbf-46dfe30a02eb admin admin] PATH_INFO: `/v3/projects` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:01:42.428873 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ead07d71-e980-464e-9bbf-46dfe30a02eb admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:01:42.431966 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ead07d71-e980-464e-9bbf-46dfe30a02eb admin admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:01:42.520082 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 1115/2219] 10.222.0.30 () {60 vars in 1152 bytes} [Mon May 23 13:01:42 2022] POST /identity/v3/projects => generated 407 bytes in 114 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) May 23 13:01:42.520688 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90c00fd7-a948-4bba-a3bf-8e4e6de02e43 admin admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:01:42.532497 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90c00fd7-a948-4bba-a3bf-8e4e6de02e43 admin admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:01:42.627751 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e519f921-1d98-4b22-a5df-a2dfb884db91 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:01:42.628514 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e519f921-1d98-4b22-a5df-a2dfb884db91 admin admin] REQUEST_METHOD: `POST` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:01:42.628877 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e519f921-1d98-4b22-a5df-a2dfb884db91 admin admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:01:42.629232 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e519f921-1d98-4b22-a5df-a2dfb884db91 admin admin] PATH_INFO: `/v3/users` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:01:42.630156 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e519f921-1d98-4b22-a5df-a2dfb884db91 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:01:42.632187 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e519f921-1d98-4b22-a5df-a2dfb884db91 admin admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:01:42.727447 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 1105/2220] 10.222.0.30 () {60 vars in 1143 bytes} [Mon May 23 13:01:42 2022] POST /identity/v3/users => generated 304 bytes in 204 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) May 23 13:01:42.728682 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ead07d71-e980-464e-9bbf-46dfe30a02eb admin admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:01:42.740435 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ead07d71-e980-464e-9bbf-46dfe30a02eb admin admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:01:42.746676 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5d41088-cf81-402e-b1ee-3fbbedc049eb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:01:42.747185 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5d41088-cf81-402e-b1ee-3fbbedc049eb admin admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:01:42.747473 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5d41088-cf81-402e-b1ee-3fbbedc049eb admin admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:01:42.747749 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5d41088-cf81-402e-b1ee-3fbbedc049eb admin admin] PATH_INFO: `/v3/roles` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:01:42.748487 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a5d41088-cf81-402e-b1ee-3fbbedc049eb admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:01:42.750201 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a5d41088-cf81-402e-b1ee-3fbbedc049eb admin admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:01:42.761871 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 1116/2221] 10.222.0.30 () {58 vars in 1121 bytes} [Mon May 23 13:01:42 2022] GET /identity/v3/roles => generated 1420 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) May 23 13:01:42.766194 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e519f921-1d98-4b22-a5df-a2dfb884db91 admin admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:01:42.778329 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e519f921-1d98-4b22-a5df-a2dfb884db91 admin admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:01:42.783347 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1947ddc2-7114-40d0-9f63-cfc300b8437e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:01:42.784040 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1947ddc2-7114-40d0-9f63-cfc300b8437e admin admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:01:42.784323 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1947ddc2-7114-40d0-9f63-cfc300b8437e admin admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:01:42.784603 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1947ddc2-7114-40d0-9f63-cfc300b8437e admin admin] PATH_INFO: `/v3/roles` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:01:42.785334 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1947ddc2-7114-40d0-9f63-cfc300b8437e admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:01:42.786977 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1947ddc2-7114-40d0-9f63-cfc300b8437e admin admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:01:42.798672 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 1106/2222] 10.222.0.30 () {58 vars in 1121 bytes} [Mon May 23 13:01:42 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) May 23 13:01:42.803119 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5d41088-cf81-402e-b1ee-3fbbedc049eb admin admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:01:42.814156 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5d41088-cf81-402e-b1ee-3fbbedc049eb admin admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:01:42.818987 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b94fb3c3-cf6d-48ce-9eb3-53f896add8e4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:01:42.819886 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b94fb3c3-cf6d-48ce-9eb3-53f896add8e4 admin admin] REQUEST_METHOD: `PUT` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:01:42.820176 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b94fb3c3-cf6d-48ce-9eb3-53f896add8e4 admin admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:01:42.823906 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b94fb3c3-cf6d-48ce-9eb3-53f896add8e4 admin admin] PATH_INFO: `/v3/projects/761edc78b23c4faba7aa88f558d3564b/users/064e2f8a197a492eb3225733fab4adda/roles/80fb22507c9a4ff09d44d8f12f6fff1a` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:01:42.837643 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b94fb3c3-cf6d-48ce-9eb3-53f896add8e4 admin admin] RBAC: Authorizing `identity:create_grant(project_id=761edc78b23c4faba7aa88f558d3564b, user_id=064e2f8a197a492eb3225733fab4adda, role_id=80fb22507c9a4ff09d44d8f12f6fff1a)` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:01:42.841298 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b94fb3c3-cf6d-48ce-9eb3-53f896add8e4 admin admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:01:42.866311 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=91,keystone:DELETE=8,keystone:INSERT=1 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 13:01:42.946555 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 1117/2223] 10.222.0.30 () {60 vars in 1482 bytes} [Mon May 23 13:01:42 2022] PUT /identity/v3/projects/761edc78b23c4faba7aa88f558d3564b/users/064e2f8a197a492eb3225733fab4adda/roles/80fb22507c9a4ff09d44d8f12f6fff1a => generated 0 bytes in 144 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) May 23 13:01:42.949813 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=83,keystone:DELETE=15,keystone:INSERT=2 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 13:01:42.957218 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13df9f68-9a1a-401f-9d53-903f22d3cca7 None None] REQUEST_METHOD: `POST` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:01:42.957510 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13df9f68-9a1a-401f-9d53-903f22d3cca7 None None] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:01:42.957792 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13df9f68-9a1a-401f-9d53-903f22d3cca7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:01:42.988050 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.auth.core [None req-13df9f68-9a1a-401f-9d53-903f22d3cca7 None None] MFA Rules not processed for user `064e2f8a197a492eb3225733fab4adda`. Rule list: `[]` (Enabled: `True`). {{(pid=92056) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 13:01:42.990195 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.fernet_utils [None req-13df9f68-9a1a-401f-9d53-903f22d3cca7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92056) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:01:43.061135 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 1107/2224] 10.222.0.30 () {58 vars in 957 bytes} [Mon May 23 13:01:42 2022] POST /identity/v3/auth/tokens => generated 3274 bytes in 105 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) May 23 13:01:43.065582 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b94fb3c3-cf6d-48ce-9eb3-53f896add8e4 admin admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:01:43.133141 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b94fb3c3-cf6d-48ce-9eb3-53f896add8e4 admin admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:01:43.139465 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f356ffc1-9e13-48eb-9f3a-791e04af6e71 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:01:43.140124 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f356ffc1-9e13-48eb-9f3a-791e04af6e71 admin admin] REQUEST_METHOD: `POST` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:01:43.140407 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f356ffc1-9e13-48eb-9f3a-791e04af6e71 admin admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:01:43.140672 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f356ffc1-9e13-48eb-9f3a-791e04af6e71 admin admin] PATH_INFO: `/v3/projects` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:01:43.141357 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f356ffc1-9e13-48eb-9f3a-791e04af6e71 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:01:43.143295 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f356ffc1-9e13-48eb-9f3a-791e04af6e71 admin admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:01:43.224093 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 1118/2225] 10.222.0.30 () {60 vars in 1152 bytes} [Mon May 23 13:01:43 2022] POST /identity/v3/projects => generated 409 bytes in 159 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) May 23 13:01:43.228226 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13df9f68-9a1a-401f-9d53-903f22d3cca7 None None] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:01:43.239483 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13df9f68-9a1a-401f-9d53-903f22d3cca7 None None] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:01:43.294941 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7bee1120-da2e-450f-b501-152eba5ff154 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:01:43.295601 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bee1120-da2e-450f-b501-152eba5ff154 admin admin] REQUEST_METHOD: `POST` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:01:43.295881 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bee1120-da2e-450f-b501-152eba5ff154 admin admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:01:43.296152 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bee1120-da2e-450f-b501-152eba5ff154 admin admin] PATH_INFO: `/v3/users` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:01:43.296843 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7bee1120-da2e-450f-b501-152eba5ff154 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:01:43.298880 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7bee1120-da2e-450f-b501-152eba5ff154 admin admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:01:43.397752 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 1108/2226] 10.222.0.30 () {60 vars in 1143 bytes} [Mon May 23 13:01:43 2022] POST /identity/v3/users => generated 311 bytes in 170 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) May 23 13:01:43.401811 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f356ffc1-9e13-48eb-9f3a-791e04af6e71 admin admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:01:43.413999 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f356ffc1-9e13-48eb-9f3a-791e04af6e71 admin admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:01:43.418836 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9482a03-2967-4b69-a09d-c1e6718b69dc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:01:43.419485 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9482a03-2967-4b69-a09d-c1e6718b69dc admin admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:01:43.419768 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9482a03-2967-4b69-a09d-c1e6718b69dc admin admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:01:43.420043 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9482a03-2967-4b69-a09d-c1e6718b69dc admin admin] PATH_INFO: `/v3/roles` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:01:43.420776 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9482a03-2967-4b69-a09d-c1e6718b69dc admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:01:43.422393 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9482a03-2967-4b69-a09d-c1e6718b69dc admin admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:01:43.435429 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 1119/2227] 10.222.0.30 () {58 vars in 1121 bytes} [Mon May 23 13:01:43 2022] GET /identity/v3/roles => generated 1420 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) May 23 13:01:43.439959 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7bee1120-da2e-450f-b501-152eba5ff154 admin admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:01:43.450463 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7bee1120-da2e-450f-b501-152eba5ff154 admin admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:01:43.463214 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f60f24d8-5d9e-4167-b4a0-663e71917539 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:01:43.464122 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f60f24d8-5d9e-4167-b4a0-663e71917539 admin admin] REQUEST_METHOD: `PUT` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:01:43.464425 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f60f24d8-5d9e-4167-b4a0-663e71917539 admin admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:01:43.464711 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f60f24d8-5d9e-4167-b4a0-663e71917539 admin admin] PATH_INFO: `/v3/projects/a31b7b2c9fc54259a9e8d676b1dfc52b/users/a008c5d341bc40ada155ef0049fc94a1/roles/bc3c3c9d654940cbaa5177a6be23d1bc` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:01:43.483084 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f60f24d8-5d9e-4167-b4a0-663e71917539 admin admin] RBAC: Authorizing `identity:create_grant(project_id=a31b7b2c9fc54259a9e8d676b1dfc52b, user_id=a008c5d341bc40ada155ef0049fc94a1, role_id=bc3c3c9d654940cbaa5177a6be23d1bc)` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:01:43.485469 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f60f24d8-5d9e-4167-b4a0-663e71917539 admin admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:01:43.616990 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 1109/2228] 10.222.0.30 () {60 vars in 1482 bytes} [Mon May 23 13:01:43 2022] PUT /identity/v3/projects/a31b7b2c9fc54259a9e8d676b1dfc52b/users/a008c5d341bc40ada155ef0049fc94a1/roles/bc3c3c9d654940cbaa5177a6be23d1bc => generated 0 bytes in 175 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) May 23 13:01:43.619049 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9482a03-2967-4b69-a09d-c1e6718b69dc admin admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:01:43.629779 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9482a03-2967-4b69-a09d-c1e6718b69dc admin admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:01:43.687521 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-daa3c560-6ac2-43a1-aaec-d586274f2512 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:01:43.688199 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-daa3c560-6ac2-43a1-aaec-d586274f2512 admin admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:01:43.688492 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-daa3c560-6ac2-43a1-aaec-d586274f2512 admin admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:01:43.688771 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-daa3c560-6ac2-43a1-aaec-d586274f2512 admin admin] PATH_INFO: `/v3/roles` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:01:43.693117 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-daa3c560-6ac2-43a1-aaec-d586274f2512 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:01:43.693117 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-daa3c560-6ac2-43a1-aaec-d586274f2512 admin admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:01:43.702341 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 1120/2229] 10.222.0.30 () {58 vars in 1121 bytes} [Mon May 23 13:01:43 2022] GET /identity/v3/roles => generated 1420 bytes in 84 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) May 23 13:01:43.707113 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f60f24d8-5d9e-4167-b4a0-663e71917539 admin admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:01:43.717327 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f60f24d8-5d9e-4167-b4a0-663e71917539 admin admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:01:43.722175 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e690d1b-ad9c-496d-963c-dbdc3123b4fa admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:01:43.723061 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e690d1b-ad9c-496d-963c-dbdc3123b4fa admin admin] REQUEST_METHOD: `PUT` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:01:43.723343 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e690d1b-ad9c-496d-963c-dbdc3123b4fa admin admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:01:43.723618 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e690d1b-ad9c-496d-963c-dbdc3123b4fa admin admin] PATH_INFO: `/v3/projects/a31b7b2c9fc54259a9e8d676b1dfc52b/users/a008c5d341bc40ada155ef0049fc94a1/roles/bc3c3c9d654940cbaa5177a6be23d1bc` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:01:43.725575 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e690d1b-ad9c-496d-963c-dbdc3123b4fa admin admin] RBAC: Authorizing `identity:create_grant(project_id=a31b7b2c9fc54259a9e8d676b1dfc52b, user_id=a008c5d341bc40ada155ef0049fc94a1, role_id=bc3c3c9d654940cbaa5177a6be23d1bc)` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:01:43.728257 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e690d1b-ad9c-496d-963c-dbdc3123b4fa admin admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:01:43.739377 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 1110/2230] 10.222.0.30 () {60 vars in 1482 bytes} [Mon May 23 13:01:43 2022] PUT /identity/v3/projects/a31b7b2c9fc54259a9e8d676b1dfc52b/users/a008c5d341bc40ada155ef0049fc94a1/roles/bc3c3c9d654940cbaa5177a6be23d1bc => generated 0 bytes in 33 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) May 23 13:01:43.752091 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52d7342a-ef87-418b-8591-9f1d2baccc67 None None] REQUEST_METHOD: `POST` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:01:43.752618 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52d7342a-ef87-418b-8591-9f1d2baccc67 None None] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:01:43.752618 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52d7342a-ef87-418b-8591-9f1d2baccc67 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:01:43.783066 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.auth.core [None req-52d7342a-ef87-418b-8591-9f1d2baccc67 None None] MFA Rules not processed for user `a008c5d341bc40ada155ef0049fc94a1`. Rule list: `[]` (Enabled: `True`). {{(pid=92057) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 13:01:43.784290 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.fernet_utils [None req-52d7342a-ef87-418b-8591-9f1d2baccc67 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92057) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:01:43.856139 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 1121/2231] 10.222.0.30 () {58 vars in 957 bytes} [Mon May 23 13:01:43 2022] POST /identity/v3/auth/tokens => generated 3343 bytes in 105 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) May 23 13:01:43.866947 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e690d1b-ad9c-496d-963c-dbdc3123b4fa admin admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:01:43.868668 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.fernet_utils [None req-5e690d1b-ad9c-496d-963c-dbdc3123b4fa admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92056) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:01:43.931459 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e690d1b-ad9c-496d-963c-dbdc3123b4fa admin admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:01:43.936730 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ddc9eab-2d95-43ff-a4a7-b95ca443000f service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1666097f8ed545c6b0e4809c93e4775a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9c389ae65aaa4542bb5a5c4fd0bacf5b', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:01:43.937382 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ddc9eab-2d95-43ff-a4a7-b95ca443000f service cinder] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:01:43.937664 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ddc9eab-2d95-43ff-a4a7-b95ca443000f service cinder] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:01:43.937978 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ddc9eab-2d95-43ff-a4a7-b95ca443000f service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:01:43.950093 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ddc9eab-2d95-43ff-a4a7-b95ca443000f service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:01:43.952403 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ddc9eab-2d95-43ff-a4a7-b95ca443000f service cinder] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:01:43.963594 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 1111/2232] 10.222.0.30 () {60 vars in 1365 bytes} [Mon May 23 13:01:43 2022] GET /identity/v3/auth/tokens => generated 3343 bytes in 97 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) May 23 13:01:44.202914 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 13:01:44.271704 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=89,keystone:DELETE=9,keystone:INSERT=2 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 13:01:44.334356 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52d7342a-ef87-418b-8591-9f1d2baccc67 None None] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:01:44.344288 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52d7342a-ef87-418b-8591-9f1d2baccc67 None None] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:01:44.348028 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05c0c78b-ed33-4c67-a6a4-53ba0fac1aec service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1666097f8ed545c6b0e4809c93e4775a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9c389ae65aaa4542bb5a5c4fd0bacf5b', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:01:44.348661 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05c0c78b-ed33-4c67-a6a4-53ba0fac1aec service cinder] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:01:44.348940 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05c0c78b-ed33-4c67-a6a4-53ba0fac1aec service cinder] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:01:44.349405 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05c0c78b-ed33-4c67-a6a4-53ba0fac1aec service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:01:44.407277 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05c0c78b-ed33-4c67-a6a4-53ba0fac1aec service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:01:44.408723 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05c0c78b-ed33-4c67-a6a4-53ba0fac1aec service cinder] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:01:44.418783 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 1122/2233] 10.222.0.30 () {60 vars in 1365 bytes} [Mon May 23 13:01:44 2022] GET /identity/v3/auth/tokens => generated 3274 bytes in 85 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) May 23 13:01:44.493091 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ddc9eab-2d95-43ff-a4a7-b95ca443000f service cinder] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:01:44.494578 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.fernet_utils [None req-2ddc9eab-2d95-43ff-a4a7-b95ca443000f service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92056) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:01:44.568849 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ddc9eab-2d95-43ff-a4a7-b95ca443000f service cinder] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:01:44.573975 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad4374f1-2a37-4677-a214-7ac3e79bb62b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:01:44.574835 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad4374f1-2a37-4677-a214-7ac3e79bb62b admin admin] REQUEST_METHOD: `DELETE` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:01:44.575115 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad4374f1-2a37-4677-a214-7ac3e79bb62b admin admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:01:44.575389 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad4374f1-2a37-4677-a214-7ac3e79bb62b admin admin] PATH_INFO: `/v3/users/01b4e4c7d54c4294ae72f9bd5f9420e1` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:01:44.576590 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad4374f1-2a37-4677-a214-7ac3e79bb62b admin admin] RBAC: Authorizing `identity:delete_user(user_id=01b4e4c7d54c4294ae72f9bd5f9420e1)` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:01:44.578575 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad4374f1-2a37-4677-a214-7ac3e79bb62b admin admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:01:44.807351 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-ad4374f1-2a37-4677-a214-7ac3e79bb62b admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '01b4e4c7d54c4294ae72f9bd5f9420e1'} {{(pid=92056) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:01:44.808792 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-ad4374f1-2a37-4677-a214-7ac3e79bb62b admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '01b4e4c7d54c4294ae72f9bd5f9420e1'} {{(pid=92056) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:01:44.880592 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-ad4374f1-2a37-4677-a214-7ac3e79bb62b admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '01b4e4c7d54c4294ae72f9bd5f9420e1'} {{(pid=92056) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:01:44.899886 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-ad4374f1-2a37-4677-a214-7ac3e79bb62b admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '01b4e4c7d54c4294ae72f9bd5f9420e1'} {{(pid=92056) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:01:44.915185 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-ad4374f1-2a37-4677-a214-7ac3e79bb62b admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '01b4e4c7d54c4294ae72f9bd5f9420e1'} {{(pid=92056) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:01:44.924771 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 1112/2234] 10.222.0.30 () {58 vars in 1223 bytes} [Mon May 23 13:01:44 2022] DELETE /identity/v3/users/01b4e4c7d54c4294ae72f9bd5f9420e1 => generated 0 bytes in 432 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) May 23 13:01:44.929314 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b5e2435-4e33-470c-a3b2-5fd380056521 None None] REQUEST_METHOD: `POST` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:01:44.929786 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b5e2435-4e33-470c-a3b2-5fd380056521 None None] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:01:44.930009 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b5e2435-4e33-470c-a3b2-5fd380056521 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:01:44.950147 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.auth.core [None req-4b5e2435-4e33-470c-a3b2-5fd380056521 None None] MFA Rules not processed for user `0f7119f8fa5c4445977671140ded91a4`. Rule list: `[]` (Enabled: `True`). {{(pid=92057) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 13:01:44.951446 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.fernet_utils [None req-4b5e2435-4e33-470c-a3b2-5fd380056521 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92057) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:01:45.017620 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 1123/2235] 10.222.0.30 () {58 vars in 957 bytes} [Mon May 23 13:01:44 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 89 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) May 23 13:01:45.027382 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad4374f1-2a37-4677-a214-7ac3e79bb62b admin admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:01:45.028843 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.fernet_utils [None req-ad4374f1-2a37-4677-a214-7ac3e79bb62b admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92056) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:01:45.100118 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad4374f1-2a37-4677-a214-7ac3e79bb62b admin admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:01:45.103807 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc1ae187-4906-4fed-ba53-6078b399f4f0 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2eb83ca38e79449095f3a2f947bbb06a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9c389ae65aaa4542bb5a5c4fd0bacf5b', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:01:45.104993 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc1ae187-4906-4fed-ba53-6078b399f4f0 service neutron] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:01:45.105269 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc1ae187-4906-4fed-ba53-6078b399f4f0 service neutron] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:01:45.105549 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc1ae187-4906-4fed-ba53-6078b399f4f0 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:01:45.123444 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc1ae187-4906-4fed-ba53-6078b399f4f0 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:01:45.124946 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc1ae187-4906-4fed-ba53-6078b399f4f0 service neutron] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:01:45.135311 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 1113/2236] 10.222.0.30 () {60 vars in 1365 bytes} [Mon May 23 13:01:45 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 109 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) May 23 13:01:45.586805 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b5e2435-4e33-470c-a3b2-5fd380056521 None None] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:01:45.588314 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.fernet_utils [None req-4b5e2435-4e33-470c-a3b2-5fd380056521 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92057) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:01:45.603970 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b5e2435-4e33-470c-a3b2-5fd380056521 None None] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:01:45.610338 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d317071e-2545-48a5-9e78-6d6cd963feab admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:01:45.611300 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d317071e-2545-48a5-9e78-6d6cd963feab admin admin] REQUEST_METHOD: `DELETE` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:01:45.611582 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d317071e-2545-48a5-9e78-6d6cd963feab admin admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:01:45.611857 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d317071e-2545-48a5-9e78-6d6cd963feab admin admin] PATH_INFO: `/v3/projects/26397036fab54883aea8a65f83190947` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:01:45.613078 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d317071e-2545-48a5-9e78-6d6cd963feab admin admin] RBAC: Authorizing `identity:delete_project(project_id=26397036fab54883aea8a65f83190947)` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:01:45.615042 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d317071e-2545-48a5-9e78-6d6cd963feab admin admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:01:45.734325 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [None req-d317071e-2545-48a5-9e78-6d6cd963feab admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '26397036fab54883aea8a65f83190947', 'request_id': 'req-d317071e-2545-48a5-9e78-6d6cd963feab'} {{(pid=92057) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:01:45.740534 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [None req-d317071e-2545-48a5-9e78-6d6cd963feab admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '26397036fab54883aea8a65f83190947', 'request_id': 'req-d317071e-2545-48a5-9e78-6d6cd963feab'} {{(pid=92057) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:01:45.871864 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [None req-d317071e-2545-48a5-9e78-6d6cd963feab admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '26397036fab54883aea8a65f83190947', 'request_id': 'req-d317071e-2545-48a5-9e78-6d6cd963feab'} {{(pid=92057) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:01:45.883733 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [None req-d317071e-2545-48a5-9e78-6d6cd963feab admin admin] The token cache is being invalidate because project 26397036fab54883aea8a65f83190947 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=92057) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} May 23 13:01:45.884021 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [None req-d317071e-2545-48a5-9e78-6d6cd963feab admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=92057) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:01:45.884723 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [None req-d317071e-2545-48a5-9e78-6d6cd963feab admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=92057) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:01:45.887290 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 1124/2237] 10.222.0.30 () {58 vars in 1232 bytes} [Mon May 23 13:01:45 2022] DELETE /identity/v3/projects/26397036fab54883aea8a65f83190947 => generated 0 bytes in 301 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) May 23 13:01:45.891666 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc1ae187-4906-4fed-ba53-6078b399f4f0 service neutron] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:01:45.902156 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc1ae187-4906-4fed-ba53-6078b399f4f0 service neutron] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:01:45.960332 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a8b4edb-8446-4076-bb63-fe1007a53d4a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:01:45.961341 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a8b4edb-8446-4076-bb63-fe1007a53d4a admin admin] REQUEST_METHOD: `DELETE` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:01:45.961670 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a8b4edb-8446-4076-bb63-fe1007a53d4a admin admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:01:45.961827 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a8b4edb-8446-4076-bb63-fe1007a53d4a admin admin] PATH_INFO: `/v3/users/28c86932778040d38f208123514d32de` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:01:45.963042 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a8b4edb-8446-4076-bb63-fe1007a53d4a admin admin] RBAC: Authorizing `identity:delete_user(user_id=28c86932778040d38f208123514d32de)` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:01:45.965706 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a8b4edb-8446-4076-bb63-fe1007a53d4a admin admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:01:46.158290 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-9a8b4edb-8446-4076-bb63-fe1007a53d4a admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '28c86932778040d38f208123514d32de'} {{(pid=92056) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:01:46.168221 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-9a8b4edb-8446-4076-bb63-fe1007a53d4a admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '28c86932778040d38f208123514d32de'} {{(pid=92056) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:01:46.217692 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-9a8b4edb-8446-4076-bb63-fe1007a53d4a admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '28c86932778040d38f208123514d32de'} {{(pid=92056) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:01:46.230013 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-9a8b4edb-8446-4076-bb63-fe1007a53d4a admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '28c86932778040d38f208123514d32de'} {{(pid=92056) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:01:46.236756 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-9a8b4edb-8446-4076-bb63-fe1007a53d4a admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '28c86932778040d38f208123514d32de'} {{(pid=92056) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:01:46.240733 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=91,keystone:DELETE=7,keystone:INSERT=2 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 13:01:46.244843 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 1114/2238] 10.222.0.30 () {58 vars in 1223 bytes} [Mon May 23 13:01:45 2022] DELETE /identity/v3/users/28c86932778040d38f208123514d32de => generated 0 bytes in 354 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) May 23 13:01:46.298119 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 13:01:46.876930 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d317071e-2545-48a5-9e78-6d6cd963feab admin admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:01:46.882871 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.fernet_utils [None req-d317071e-2545-48a5-9e78-6d6cd963feab admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92057) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:01:46.911643 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d317071e-2545-48a5-9e78-6d6cd963feab admin admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:01:46.978045 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1c9d543-3401-477f-a391-e8173c0f298c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:01:46.978923 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1c9d543-3401-477f-a391-e8173c0f298c admin admin] REQUEST_METHOD: `DELETE` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:01:46.979207 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1c9d543-3401-477f-a391-e8173c0f298c admin admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:01:46.979551 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1c9d543-3401-477f-a391-e8173c0f298c admin admin] PATH_INFO: `/v3/projects/5126b2bfeaf94cc29ec9745c48bcfbfb` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:01:46.980807 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1c9d543-3401-477f-a391-e8173c0f298c admin admin] RBAC: Authorizing `identity:delete_project(project_id=5126b2bfeaf94cc29ec9745c48bcfbfb)` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:01:46.982797 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1c9d543-3401-477f-a391-e8173c0f298c admin admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:01:47.200640 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [None req-c1c9d543-3401-477f-a391-e8173c0f298c admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '5126b2bfeaf94cc29ec9745c48bcfbfb', 'request_id': 'req-c1c9d543-3401-477f-a391-e8173c0f298c'} {{(pid=92057) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:01:47.206494 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [None req-c1c9d543-3401-477f-a391-e8173c0f298c admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '5126b2bfeaf94cc29ec9745c48bcfbfb', 'request_id': 'req-c1c9d543-3401-477f-a391-e8173c0f298c'} {{(pid=92057) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:01:47.320119 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [None req-c1c9d543-3401-477f-a391-e8173c0f298c admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '5126b2bfeaf94cc29ec9745c48bcfbfb', 'request_id': 'req-c1c9d543-3401-477f-a391-e8173c0f298c'} {{(pid=92057) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:01:47.333255 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [None req-c1c9d543-3401-477f-a391-e8173c0f298c admin admin] The token cache is being invalidate because project 5126b2bfeaf94cc29ec9745c48bcfbfb was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=92057) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} May 23 13:01:47.333543 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [None req-c1c9d543-3401-477f-a391-e8173c0f298c admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=92057) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:01:47.334201 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [None req-c1c9d543-3401-477f-a391-e8173c0f298c admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=92057) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:01:47.336010 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 1125/2239] 10.222.0.30 () {58 vars in 1232 bytes} [Mon May 23 13:01:46 2022] DELETE /identity/v3/projects/5126b2bfeaf94cc29ec9745c48bcfbfb => generated 0 bytes in 459 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) May 23 13:01:47.341939 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-208e7a9a-41b7-4033-b076-410b85bc6fab None None] REQUEST_METHOD: `POST` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:01:47.342333 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-208e7a9a-41b7-4033-b076-410b85bc6fab None None] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:01:47.342502 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-208e7a9a-41b7-4033-b076-410b85bc6fab None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:01:47.363059 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.auth.core [None req-208e7a9a-41b7-4033-b076-410b85bc6fab None None] MFA Rules not processed for user `0f7119f8fa5c4445977671140ded91a4`. Rule list: `[]` (Enabled: `True`). {{(pid=92056) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 13:01:47.364506 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.fernet_utils [None req-208e7a9a-41b7-4033-b076-410b85bc6fab None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92056) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:01:47.443325 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 1115/2240] 10.222.0.30 () {58 vars in 957 bytes} [Mon May 23 13:01:47 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 103 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) May 23 13:01:47.452133 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-516bef43-b49a-4f99-a030-f65223b108c8 None None] REQUEST_METHOD: `POST` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:01:47.452430 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-516bef43-b49a-4f99-a030-f65223b108c8 None None] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:01:47.452713 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-516bef43-b49a-4f99-a030-f65223b108c8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:01:47.472174 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.auth.core [None req-516bef43-b49a-4f99-a030-f65223b108c8 None None] MFA Rules not processed for user `0f7119f8fa5c4445977671140ded91a4`. Rule list: `[]` (Enabled: `True`). {{(pid=92057) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 13:01:47.473370 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.fernet_utils [None req-516bef43-b49a-4f99-a030-f65223b108c8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92057) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:01:47.481806 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 1126/2241] 10.222.0.30 () {58 vars in 957 bytes} [Mon May 23 13:01:47 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) May 23 13:01:47.486001 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-208e7a9a-41b7-4033-b076-410b85bc6fab None None] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:01:47.495781 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-208e7a9a-41b7-4033-b076-410b85bc6fab None None] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:01:47.500940 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07eda92a-a869-47e2-a984-b9806a40dc72 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:01:47.501644 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07eda92a-a869-47e2-a984-b9806a40dc72 admin admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:01:47.501964 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07eda92a-a869-47e2-a984-b9806a40dc72 admin admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:01:47.502246 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07eda92a-a869-47e2-a984-b9806a40dc72 admin admin] PATH_INFO: `/v3/domains` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:01:47.502987 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07eda92a-a869-47e2-a984-b9806a40dc72 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=92056) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} May 23 13:01:47.503298 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07eda92a-a869-47e2-a984-b9806a40dc72 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:01:47.504927 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07eda92a-a869-47e2-a984-b9806a40dc72 admin admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:01:47.517844 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 1116/2242] 10.222.0.30 () {58 vars in 1152 bytes} [Mon May 23 13:01:47 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) May 23 13:01:47.522389 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-516bef43-b49a-4f99-a030-f65223b108c8 None None] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:01:47.532397 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-516bef43-b49a-4f99-a030-f65223b108c8 None None] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:01:47.539616 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e971e99-60b5-4c4a-91d3-453789d3de62 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:01:47.540965 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e971e99-60b5-4c4a-91d3-453789d3de62 admin admin] REQUEST_METHOD: `POST` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:01:47.541380 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e971e99-60b5-4c4a-91d3-453789d3de62 admin admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:01:47.541867 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e971e99-60b5-4c4a-91d3-453789d3de62 admin admin] PATH_INFO: `/v3/projects` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:01:47.542838 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e971e99-60b5-4c4a-91d3-453789d3de62 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:01:47.544771 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e971e99-60b5-4c4a-91d3-453789d3de62 admin admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:01:47.591662 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 13:01:47.670615 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 1127/2243] 10.222.0.30 () {60 vars in 1152 bytes} [Mon May 23 13:01:47 2022] POST /identity/v3/projects => generated 377 bytes in 149 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) May 23 13:01:47.675044 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07eda92a-a869-47e2-a984-b9806a40dc72 admin admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:01:47.686378 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07eda92a-a869-47e2-a984-b9806a40dc72 admin admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:01:47.749120 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4175c0d2-d9ce-4529-abcb-28e4bb9b6a0f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:01:47.749787 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4175c0d2-d9ce-4529-abcb-28e4bb9b6a0f admin admin] REQUEST_METHOD: `POST` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:01:47.750094 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4175c0d2-d9ce-4529-abcb-28e4bb9b6a0f admin admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:01:47.750366 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4175c0d2-d9ce-4529-abcb-28e4bb9b6a0f admin admin] PATH_INFO: `/v3/users` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:01:47.751047 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4175c0d2-d9ce-4529-abcb-28e4bb9b6a0f admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:01:47.753166 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4175c0d2-d9ce-4529-abcb-28e4bb9b6a0f admin admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:01:47.870545 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 1117/2244] 10.222.0.30 () {60 vars in 1143 bytes} [Mon May 23 13:01:47 2022] POST /identity/v3/users => generated 289 bytes in 196 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) May 23 13:01:47.878059 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e971e99-60b5-4c4a-91d3-453789d3de62 admin admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:01:47.888806 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e971e99-60b5-4c4a-91d3-453789d3de62 admin admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:01:47.893990 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56abae1c-1170-40ae-83ce-f8d5e9f8f0cf admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:01:47.900031 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56abae1c-1170-40ae-83ce-f8d5e9f8f0cf admin admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:01:47.900031 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56abae1c-1170-40ae-83ce-f8d5e9f8f0cf admin admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:01:47.900031 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56abae1c-1170-40ae-83ce-f8d5e9f8f0cf admin admin] PATH_INFO: `/v3/roles` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:01:47.900031 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56abae1c-1170-40ae-83ce-f8d5e9f8f0cf admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:01:47.900031 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56abae1c-1170-40ae-83ce-f8d5e9f8f0cf admin admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:01:47.908261 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 1128/2245] 10.222.0.30 () {58 vars in 1121 bytes} [Mon May 23 13:01:47 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) May 23 13:01:47.912274 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4175c0d2-d9ce-4529-abcb-28e4bb9b6a0f admin admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:01:47.922530 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4175c0d2-d9ce-4529-abcb-28e4bb9b6a0f admin admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:01:47.927238 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67c48650-0d99-45b4-a003-eb61701e8dad admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:01:47.927889 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67c48650-0d99-45b4-a003-eb61701e8dad admin admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:01:47.928199 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67c48650-0d99-45b4-a003-eb61701e8dad admin admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:01:47.928501 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67c48650-0d99-45b4-a003-eb61701e8dad admin admin] PATH_INFO: `/v3/roles` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:01:47.929299 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67c48650-0d99-45b4-a003-eb61701e8dad admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:01:47.930943 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67c48650-0d99-45b4-a003-eb61701e8dad admin admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:01:47.941804 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 1118/2246] 10.222.0.30 () {58 vars in 1121 bytes} [Mon May 23 13:01:47 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) May 23 13:01:47.946082 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56abae1c-1170-40ae-83ce-f8d5e9f8f0cf admin admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:01:47.957122 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56abae1c-1170-40ae-83ce-f8d5e9f8f0cf admin admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:01:47.960375 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 13:01:47.961783 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e8a6de0-cf5f-4f2d-952c-378703a0b799 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:01:47.962707 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e8a6de0-cf5f-4f2d-952c-378703a0b799 admin admin] REQUEST_METHOD: `PUT` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:01:47.962993 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e8a6de0-cf5f-4f2d-952c-378703a0b799 admin admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:01:47.963273 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e8a6de0-cf5f-4f2d-952c-378703a0b799 admin admin] PATH_INFO: `/v3/projects/d177483c7cf24ca7aadd852a66c2e427/users/7ce98e71dfe94015be955ec759a9e312/roles/80fb22507c9a4ff09d44d8f12f6fff1a` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:01:47.979100 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e8a6de0-cf5f-4f2d-952c-378703a0b799 admin admin] RBAC: Authorizing `identity:create_grant(project_id=d177483c7cf24ca7aadd852a66c2e427, user_id=7ce98e71dfe94015be955ec759a9e312, role_id=80fb22507c9a4ff09d44d8f12f6fff1a)` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:01:47.982514 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e8a6de0-cf5f-4f2d-952c-378703a0b799 admin admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:01:48.037518 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 1129/2247] 10.222.0.30 () {60 vars in 1482 bytes} [Mon May 23 13:01:47 2022] PUT /identity/v3/projects/d177483c7cf24ca7aadd852a66c2e427/users/7ce98e71dfe94015be955ec759a9e312/roles/80fb22507c9a4ff09d44d8f12f6fff1a => generated 0 bytes in 92 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) May 23 13:01:48.048534 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9324c439-3912-44ed-b8dd-d511ef55a882 None None] REQUEST_METHOD: `POST` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:01:48.048824 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9324c439-3912-44ed-b8dd-d511ef55a882 None None] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:01:48.049105 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9324c439-3912-44ed-b8dd-d511ef55a882 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:01:48.079243 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.auth.core [None req-9324c439-3912-44ed-b8dd-d511ef55a882 None None] MFA Rules not processed for user `7ce98e71dfe94015be955ec759a9e312`. Rule list: `[]` (Enabled: `True`). {{(pid=92056) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 13:01:48.080474 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.fernet_utils [None req-9324c439-3912-44ed-b8dd-d511ef55a882 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92056) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:01:48.146936 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 1119/2248] 10.222.0.30 () {58 vars in 957 bytes} [Mon May 23 13:01:48 2022] POST /identity/v3/auth/tokens => generated 3244 bytes in 99 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) May 23 13:01:48.151646 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e8a6de0-cf5f-4f2d-952c-378703a0b799 admin admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:01:48.216744 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e8a6de0-cf5f-4f2d-952c-378703a0b799 admin admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:01:48.221439 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d60e1e8-1a65-4516-82b6-237511dbe311 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:01:48.222182 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d60e1e8-1a65-4516-82b6-237511dbe311 admin admin] REQUEST_METHOD: `POST` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:01:48.222735 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d60e1e8-1a65-4516-82b6-237511dbe311 admin admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:01:48.223199 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d60e1e8-1a65-4516-82b6-237511dbe311 admin admin] PATH_INFO: `/v3/projects` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:01:48.224954 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=88,keystone:DELETE=7,keystone:INSERT=5 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 13:01:48.225605 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d60e1e8-1a65-4516-82b6-237511dbe311 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:01:48.227643 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d60e1e8-1a65-4516-82b6-237511dbe311 admin admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:01:48.312037 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 1130/2249] 10.222.0.30 () {60 vars in 1152 bytes} [Mon May 23 13:01:48 2022] POST /identity/v3/projects => generated 375 bytes in 161 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) May 23 13:01:48.316092 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9324c439-3912-44ed-b8dd-d511ef55a882 None None] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:01:48.327074 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9324c439-3912-44ed-b8dd-d511ef55a882 None None] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:01:48.385121 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07baaaf6-0cac-4c40-b68f-f2aee47418ba admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:01:48.385791 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07baaaf6-0cac-4c40-b68f-f2aee47418ba admin admin] REQUEST_METHOD: `POST` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:01:48.386268 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07baaaf6-0cac-4c40-b68f-f2aee47418ba admin admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:01:48.386545 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07baaaf6-0cac-4c40-b68f-f2aee47418ba admin admin] PATH_INFO: `/v3/users` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:01:48.387440 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07baaaf6-0cac-4c40-b68f-f2aee47418ba admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:01:48.389855 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07baaaf6-0cac-4c40-b68f-f2aee47418ba admin admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:01:48.412375 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 13:01:48.500336 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 1120/2250] 10.222.0.30 () {60 vars in 1143 bytes} [Mon May 23 13:01:48 2022] POST /identity/v3/users => generated 288 bytes in 183 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) May 23 13:01:48.508844 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d60e1e8-1a65-4516-82b6-237511dbe311 admin admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:01:48.520452 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d60e1e8-1a65-4516-82b6-237511dbe311 admin admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:01:48.525173 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8635175a-e1ce-4930-9b2a-e2ad659ad660 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:01:48.525828 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8635175a-e1ce-4930-9b2a-e2ad659ad660 admin admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:01:48.526157 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8635175a-e1ce-4930-9b2a-e2ad659ad660 admin admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:01:48.526436 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8635175a-e1ce-4930-9b2a-e2ad659ad660 admin admin] PATH_INFO: `/v3/roles` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:01:48.527166 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8635175a-e1ce-4930-9b2a-e2ad659ad660 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:01:48.529342 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8635175a-e1ce-4930-9b2a-e2ad659ad660 admin admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:01:48.540327 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 1131/2251] 10.222.0.30 () {58 vars in 1121 bytes} [Mon May 23 13:01:48 2022] GET /identity/v3/roles => generated 1420 bytes in 39 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) May 23 13:01:48.545160 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07baaaf6-0cac-4c40-b68f-f2aee47418ba admin admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:01:48.555264 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07baaaf6-0cac-4c40-b68f-f2aee47418ba admin admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:01:48.560779 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f58b59b-27c5-4018-853e-ada0537ad73a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:01:48.564642 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f58b59b-27c5-4018-853e-ada0537ad73a admin admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:01:48.564642 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f58b59b-27c5-4018-853e-ada0537ad73a admin admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:01:48.564642 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f58b59b-27c5-4018-853e-ada0537ad73a admin admin] PATH_INFO: `/v3/roles` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:01:48.565225 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f58b59b-27c5-4018-853e-ada0537ad73a admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:01:48.566900 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f58b59b-27c5-4018-853e-ada0537ad73a admin admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:01:48.577513 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 1121/2252] 10.222.0.30 () {58 vars in 1121 bytes} [Mon May 23 13:01:48 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) May 23 13:01:48.581532 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8635175a-e1ce-4930-9b2a-e2ad659ad660 admin admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:01:48.592526 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8635175a-e1ce-4930-9b2a-e2ad659ad660 admin admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:01:48.597197 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0869f7aa-b52c-4a01-9dda-820e360cef4b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:01:48.598098 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0869f7aa-b52c-4a01-9dda-820e360cef4b admin admin] REQUEST_METHOD: `PUT` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:01:48.598386 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0869f7aa-b52c-4a01-9dda-820e360cef4b admin admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:01:48.598820 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0869f7aa-b52c-4a01-9dda-820e360cef4b admin admin] PATH_INFO: `/v3/projects/1b0d1830da054b8599242c45f207e1ac/users/c0047fb72e0e481981770285e6dacf33/roles/80fb22507c9a4ff09d44d8f12f6fff1a` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:01:48.614361 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0869f7aa-b52c-4a01-9dda-820e360cef4b admin admin] RBAC: Authorizing `identity:create_grant(project_id=1b0d1830da054b8599242c45f207e1ac, user_id=c0047fb72e0e481981770285e6dacf33, role_id=80fb22507c9a4ff09d44d8f12f6fff1a)` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:01:48.617055 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0869f7aa-b52c-4a01-9dda-820e360cef4b admin admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:01:48.701077 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 1132/2253] 10.222.0.30 () {60 vars in 1482 bytes} [Mon May 23 13:01:48 2022] PUT /identity/v3/projects/1b0d1830da054b8599242c45f207e1ac/users/c0047fb72e0e481981770285e6dacf33/roles/80fb22507c9a4ff09d44d8f12f6fff1a => generated 0 bytes in 120 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) May 23 13:01:48.712503 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-863a9336-a8cf-4f9b-9d32-e4c7dd84f159 None None] REQUEST_METHOD: `POST` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:01:48.712794 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-863a9336-a8cf-4f9b-9d32-e4c7dd84f159 None None] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:01:48.713076 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-863a9336-a8cf-4f9b-9d32-e4c7dd84f159 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:01:48.741989 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.auth.core [None req-863a9336-a8cf-4f9b-9d32-e4c7dd84f159 None None] MFA Rules not processed for user `c0047fb72e0e481981770285e6dacf33`. Rule list: `[]` (Enabled: `True`). {{(pid=92056) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 13:01:48.743276 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.fernet_utils [None req-863a9336-a8cf-4f9b-9d32-e4c7dd84f159 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92056) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:01:48.813547 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 1122/2254] 10.222.0.30 () {58 vars in 957 bytes} [Mon May 23 13:01:48 2022] POST /identity/v3/auth/tokens => generated 3242 bytes in 102 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) May 23 13:01:48.818161 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0869f7aa-b52c-4a01-9dda-820e360cef4b admin admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:01:48.854646 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 13:01:48.881082 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0869f7aa-b52c-4a01-9dda-820e360cef4b admin admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:01:48.885799 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d226fc9-8019-43de-a40b-f31963a1d295 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:01:48.886492 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d226fc9-8019-43de-a40b-f31963a1d295 admin admin] REQUEST_METHOD: `POST` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:01:48.886774 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d226fc9-8019-43de-a40b-f31963a1d295 admin admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:01:48.887644 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d226fc9-8019-43de-a40b-f31963a1d295 admin admin] PATH_INFO: `/v3/projects` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:01:48.888314 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d226fc9-8019-43de-a40b-f31963a1d295 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:01:48.890606 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d226fc9-8019-43de-a40b-f31963a1d295 admin admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:01:48.939811 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=88,keystone:INSERT=2,keystone:DELETE=10 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 13:01:48.976143 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 1133/2255] 10.222.0.30 () {60 vars in 1152 bytes} [Mon May 23 13:01:48 2022] POST /identity/v3/projects => generated 377 bytes in 158 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) May 23 13:01:48.980906 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-863a9336-a8cf-4f9b-9d32-e4c7dd84f159 None None] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:01:48.992741 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-863a9336-a8cf-4f9b-9d32-e4c7dd84f159 None None] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:01:49.052793 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb9ced7d-ffd0-489c-8b68-1ed205050498 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:01:49.053486 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb9ced7d-ffd0-489c-8b68-1ed205050498 admin admin] REQUEST_METHOD: `POST` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:01:49.053765 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb9ced7d-ffd0-489c-8b68-1ed205050498 admin admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:01:49.054061 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb9ced7d-ffd0-489c-8b68-1ed205050498 admin admin] PATH_INFO: `/v3/users` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:01:49.054747 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb9ced7d-ffd0-489c-8b68-1ed205050498 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:01:49.056662 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb9ced7d-ffd0-489c-8b68-1ed205050498 admin admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:01:49.158014 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 1123/2256] 10.222.0.30 () {60 vars in 1143 bytes} [Mon May 23 13:01:48 2022] POST /identity/v3/users => generated 295 bytes in 178 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) May 23 13:01:49.161824 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d226fc9-8019-43de-a40b-f31963a1d295 admin admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:01:49.173221 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d226fc9-8019-43de-a40b-f31963a1d295 admin admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:01:49.177973 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cfc48fdc-a50a-4069-abfe-29633eb8dc84 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:01:49.178630 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfc48fdc-a50a-4069-abfe-29633eb8dc84 admin admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:01:49.178912 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfc48fdc-a50a-4069-abfe-29633eb8dc84 admin admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:01:49.179184 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfc48fdc-a50a-4069-abfe-29633eb8dc84 admin admin] PATH_INFO: `/v3/roles` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:01:49.179933 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cfc48fdc-a50a-4069-abfe-29633eb8dc84 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:01:49.181639 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cfc48fdc-a50a-4069-abfe-29633eb8dc84 admin admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:01:49.192933 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 1134/2257] 10.222.0.30 () {58 vars in 1121 bytes} [Mon May 23 13:01:49 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) May 23 13:01:49.197147 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb9ced7d-ffd0-489c-8b68-1ed205050498 admin admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:01:49.208146 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb9ced7d-ffd0-489c-8b68-1ed205050498 admin admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:01:49.214447 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-faf937c5-e753-4ade-9b10-bf4ab76226eb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:01:49.215886 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-faf937c5-e753-4ade-9b10-bf4ab76226eb admin admin] REQUEST_METHOD: `PUT` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:01:49.216213 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-faf937c5-e753-4ade-9b10-bf4ab76226eb admin admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:01:49.216945 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-faf937c5-e753-4ade-9b10-bf4ab76226eb admin admin] PATH_INFO: `/v3/projects/b3c26b67bf144c619b6db450170bbf7d/users/3a0c2e6cd69b49e588b83c437b4a1dbe/roles/bc3c3c9d654940cbaa5177a6be23d1bc` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:01:49.231961 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-faf937c5-e753-4ade-9b10-bf4ab76226eb admin admin] RBAC: Authorizing `identity:create_grant(project_id=b3c26b67bf144c619b6db450170bbf7d, user_id=3a0c2e6cd69b49e588b83c437b4a1dbe, role_id=bc3c3c9d654940cbaa5177a6be23d1bc)` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:01:49.235036 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-faf937c5-e753-4ade-9b10-bf4ab76226eb admin admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:01:49.254452 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=100 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 13:01:49.300151 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 1124/2258] 10.222.0.30 () {60 vars in 1482 bytes} [Mon May 23 13:01:49 2022] PUT /identity/v3/projects/b3c26b67bf144c619b6db450170bbf7d/users/3a0c2e6cd69b49e588b83c437b4a1dbe/roles/bc3c3c9d654940cbaa5177a6be23d1bc => generated 0 bytes in 103 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) May 23 13:01:49.304284 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cfc48fdc-a50a-4069-abfe-29633eb8dc84 admin admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:01:49.319694 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cfc48fdc-a50a-4069-abfe-29633eb8dc84 admin admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:01:49.321824 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=100 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 13:01:49.352143 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=100 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 13:01:49.375008 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=88,keystone:DELETE=11,keystone:INSERT=1 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 13:01:49.384395 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d04f61e-ae45-4451-beaa-0a13719ab9da admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:01:49.385085 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d04f61e-ae45-4451-beaa-0a13719ab9da admin admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:01:49.385389 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d04f61e-ae45-4451-beaa-0a13719ab9da admin admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:01:49.385657 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d04f61e-ae45-4451-beaa-0a13719ab9da admin admin] PATH_INFO: `/v3/roles` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:01:49.386420 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d04f61e-ae45-4451-beaa-0a13719ab9da admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:01:49.389140 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d04f61e-ae45-4451-beaa-0a13719ab9da admin admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:01:49.399239 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 1135/2259] 10.222.0.30 () {58 vars in 1121 bytes} [Mon May 23 13:01:49 2022] GET /identity/v3/roles => generated 1420 bytes in 96 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) May 23 13:01:49.402780 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 13:01:49.404348 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-faf937c5-e753-4ade-9b10-bf4ab76226eb admin admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:01:49.415753 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-faf937c5-e753-4ade-9b10-bf4ab76226eb admin admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:01:49.421020 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab6ec789-a5c9-4555-aef3-bf0347a8207d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:01:49.422025 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab6ec789-a5c9-4555-aef3-bf0347a8207d admin admin] REQUEST_METHOD: `PUT` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:01:49.422306 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab6ec789-a5c9-4555-aef3-bf0347a8207d admin admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:01:49.422579 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab6ec789-a5c9-4555-aef3-bf0347a8207d admin admin] PATH_INFO: `/v3/projects/b3c26b67bf144c619b6db450170bbf7d/users/3a0c2e6cd69b49e588b83c437b4a1dbe/roles/bc3c3c9d654940cbaa5177a6be23d1bc` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:01:49.424584 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab6ec789-a5c9-4555-aef3-bf0347a8207d admin admin] RBAC: Authorizing `identity:create_grant(project_id=b3c26b67bf144c619b6db450170bbf7d, user_id=3a0c2e6cd69b49e588b83c437b4a1dbe, role_id=bc3c3c9d654940cbaa5177a6be23d1bc)` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:01:49.428833 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab6ec789-a5c9-4555-aef3-bf0347a8207d admin admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:01:49.442236 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 1125/2260] 10.222.0.30 () {60 vars in 1482 bytes} [Mon May 23 13:01:49 2022] PUT /identity/v3/projects/b3c26b67bf144c619b6db450170bbf7d/users/3a0c2e6cd69b49e588b83c437b4a1dbe/roles/bc3c3c9d654940cbaa5177a6be23d1bc => generated 0 bytes in 38 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) May 23 13:01:49.453224 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19abbfe3-6711-4a98-9df4-2fcc2ce48649 None None] REQUEST_METHOD: `POST` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:01:49.453528 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19abbfe3-6711-4a98-9df4-2fcc2ce48649 None None] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:01:49.453813 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19abbfe3-6711-4a98-9df4-2fcc2ce48649 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:01:49.486685 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.auth.core [None req-19abbfe3-6711-4a98-9df4-2fcc2ce48649 None None] MFA Rules not processed for user `3a0c2e6cd69b49e588b83c437b4a1dbe`. Rule list: `[]` (Enabled: `True`). {{(pid=92057) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 13:01:49.487898 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.fernet_utils [None req-19abbfe3-6711-4a98-9df4-2fcc2ce48649 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92057) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:01:49.575038 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 1136/2261] 10.222.0.30 () {58 vars in 957 bytes} [Mon May 23 13:01:49 2022] POST /identity/v3/auth/tokens => generated 3311 bytes in 118 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) May 23 13:01:49.578190 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 13:01:49.583560 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab6ec789-a5c9-4555-aef3-bf0347a8207d admin admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:01:49.588111 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.fernet_utils [None req-ab6ec789-a5c9-4555-aef3-bf0347a8207d admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92056) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:01:49.607603 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:DELETE=3,keystone:SELECT=93,keystone:INSERT=4 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 13:01:49.650772 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab6ec789-a5c9-4555-aef3-bf0347a8207d admin admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:01:49.654851 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cff9168e-19bd-413b-bb79-6075bdaeb15b service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1666097f8ed545c6b0e4809c93e4775a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9c389ae65aaa4542bb5a5c4fd0bacf5b', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:01:49.655500 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cff9168e-19bd-413b-bb79-6075bdaeb15b service cinder] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:01:49.656158 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cff9168e-19bd-413b-bb79-6075bdaeb15b service cinder] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:01:49.656432 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cff9168e-19bd-413b-bb79-6075bdaeb15b service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:01:49.720965 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cff9168e-19bd-413b-bb79-6075bdaeb15b service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:01:49.722451 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cff9168e-19bd-413b-bb79-6075bdaeb15b service cinder] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:01:49.732217 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 1126/2262] 10.222.0.30 () {60 vars in 1365 bytes} [Mon May 23 13:01:49 2022] GET /identity/v3/auth/tokens => generated 3244 bytes in 149 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) May 23 13:01:49.757482 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 13:01:49.858756 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=100 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 13:01:50.100190 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=84,keystone:DELETE=14,keystone:INSERT=2 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 13:01:50.201977 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 13:01:50.234614 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ead4a2d-8659-4382-a2f7-4eee6f7289fc None None] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:01:50.234978 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ead4a2d-8659-4382-a2f7-4eee6f7289fc None None] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:01:50.235264 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ead4a2d-8659-4382-a2f7-4eee6f7289fc None None] PATH_INFO: `/` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:01:50.236664 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 1137/2263] 10.222.0.30 () {54 vars in 912 bytes} [Mon May 23 13:01:50 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) May 23 13:01:50.241691 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0dac18f-556b-469f-baff-41a00d00d926 None None] REQUEST_METHOD: `POST` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:01:50.242008 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0dac18f-556b-469f-baff-41a00d00d926 None None] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:01:50.242322 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0dac18f-556b-469f-baff-41a00d00d926 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:01:50.280550 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.auth.core [None req-c0dac18f-556b-469f-baff-41a00d00d926 None None] MFA Rules not processed for user `2eb83ca38e79449095f3a2f947bbb06a`. Rule list: `[]` (Enabled: `True`). {{(pid=92056) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 13:01:50.282086 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.fernet_utils [None req-c0dac18f-556b-469f-baff-41a00d00d926 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92056) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:01:50.337207 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 1127/2264] 10.222.0.30 () {58 vars in 1009 bytes} [Mon May 23 13:01:50 2022] POST /identity/v3/auth/tokens => generated 3111 bytes in 96 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) May 23 13:01:50.349220 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ead4a2d-8659-4382-a2f7-4eee6f7289fc None None] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:01:50.351032 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.fernet_utils [None req-7ead4a2d-8659-4382-a2f7-4eee6f7289fc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92057) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:01:50.366269 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ead4a2d-8659-4382-a2f7-4eee6f7289fc None None] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:01:50.369696 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-889d096d-712a-4e91-b9c9-8aceb5912132 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2eb83ca38e79449095f3a2f947bbb06a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9c389ae65aaa4542bb5a5c4fd0bacf5b', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:01:50.370651 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-889d096d-712a-4e91-b9c9-8aceb5912132 service neutron] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:01:50.371031 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-889d096d-712a-4e91-b9c9-8aceb5912132 service neutron] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:01:50.371487 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-889d096d-712a-4e91-b9c9-8aceb5912132 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:01:50.380539 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-889d096d-712a-4e91-b9c9-8aceb5912132 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:01:50.382574 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-889d096d-712a-4e91-b9c9-8aceb5912132 service neutron] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:01:50.393446 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 1138/2265] 10.222.0.30 () {60 vars in 1365 bytes} [Mon May 23 13:01:50 2022] GET /identity/v3/auth/tokens => generated 3111 bytes in 45 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) May 23 13:01:50.670041 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=87,keystone:INSERT=6,keystone:DELETE=7 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 13:01:50.709354 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:INSERT=4,keystone:SELECT=96 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 13:01:51.251821 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:DELETE=13,keystone:SELECT=86,keystone:INSERT=1 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 13:01:51.412084 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 13:01:51.523509 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 13:01:51.675192 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 13:01:59.203072 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0dac18f-556b-469f-baff-41a00d00d926 None None] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:01:59.214718 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0dac18f-556b-469f-baff-41a00d00d926 None None] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:01:59.220021 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6305a53d-a3ff-41ad-9747-58a6f69dda1b service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1666097f8ed545c6b0e4809c93e4775a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9c389ae65aaa4542bb5a5c4fd0bacf5b', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:01:59.220751 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6305a53d-a3ff-41ad-9747-58a6f69dda1b service cinder] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:01:59.221120 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6305a53d-a3ff-41ad-9747-58a6f69dda1b service cinder] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:01:59.221475 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6305a53d-a3ff-41ad-9747-58a6f69dda1b service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:01:59.290617 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6305a53d-a3ff-41ad-9747-58a6f69dda1b service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:01:59.292272 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6305a53d-a3ff-41ad-9747-58a6f69dda1b service cinder] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:01:59.302762 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 1128/2266] 10.222.0.30 () {60 vars in 1365 bytes} [Mon May 23 13:01:59 2022] GET /identity/v3/auth/tokens => generated 3242 bytes in 100 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) May 23 13:02:00.922485 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-889d096d-712a-4e91-b9c9-8aceb5912132 service neutron] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:02:00.938253 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-889d096d-712a-4e91-b9c9-8aceb5912132 service neutron] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:02:00.942517 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f21d6dbe-fe39-4c06-8815-6578b0ea4e2f service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1666097f8ed545c6b0e4809c93e4775a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9c389ae65aaa4542bb5a5c4fd0bacf5b', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:02:00.943196 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f21d6dbe-fe39-4c06-8815-6578b0ea4e2f service cinder] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:02:00.943466 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f21d6dbe-fe39-4c06-8815-6578b0ea4e2f service cinder] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:02:00.943898 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f21d6dbe-fe39-4c06-8815-6578b0ea4e2f service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:02:00.955806 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f21d6dbe-fe39-4c06-8815-6578b0ea4e2f service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:02:00.957280 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f21d6dbe-fe39-4c06-8815-6578b0ea4e2f service cinder] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:02:00.978326 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 1139/2267] 10.222.0.30 () {60 vars in 1365 bytes} [Mon May 23 13:02:00 2022] GET /identity/v3/auth/tokens => generated 3311 bytes in 56 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) May 23 13:02:03.584522 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35d66cbf-166e-4e3b-a3c8-773354bef45d None None] REQUEST_METHOD: `POST` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:02:03.585056 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35d66cbf-166e-4e3b-a3c8-773354bef45d None None] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:02:03.585161 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35d66cbf-166e-4e3b-a3c8-773354bef45d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:02:03.602840 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.auth.core [None req-35d66cbf-166e-4e3b-a3c8-773354bef45d None None] MFA Rules not processed for user `7f5e411f4d8e4f13b662cf2846720cd3`. Rule list: `[]` (Enabled: `True`). {{(pid=92056) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 13:02:03.604044 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.fernet_utils [None req-35d66cbf-166e-4e3b-a3c8-773354bef45d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92056) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:02:03.669375 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 1129/2268] 10.222.0.30 () {58 vars in 957 bytes} [Mon May 23 13:02:03 2022] POST /identity/v3/auth/tokens => generated 3238 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) May 23 13:02:04.196313 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f21d6dbe-fe39-4c06-8815-6578b0ea4e2f service cinder] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:02:04.199368 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.fernet_utils [None req-f21d6dbe-fe39-4c06-8815-6578b0ea4e2f service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92057) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:02:04.246194 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:INSERT=1,keystone:SELECT=99 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 13:02:04.274602 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f21d6dbe-fe39-4c06-8815-6578b0ea4e2f service cinder] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:02:04.282439 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a72c32d-417d-43cf-9c56-864ee5d6c18e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:02:04.283857 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a72c32d-417d-43cf-9c56-864ee5d6c18e admin admin] REQUEST_METHOD: `DELETE` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:02:04.284083 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a72c32d-417d-43cf-9c56-864ee5d6c18e admin admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:02:04.284355 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a72c32d-417d-43cf-9c56-864ee5d6c18e admin admin] PATH_INFO: `/v3/users/064e2f8a197a492eb3225733fab4adda` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:02:04.285951 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a72c32d-417d-43cf-9c56-864ee5d6c18e admin admin] RBAC: Authorizing `identity:delete_user(user_id=064e2f8a197a492eb3225733fab4adda)` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:02:04.288096 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a72c32d-417d-43cf-9c56-864ee5d6c18e admin admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:02:04.541866 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [None req-3a72c32d-417d-43cf-9c56-864ee5d6c18e admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '064e2f8a197a492eb3225733fab4adda'} {{(pid=92057) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:02:04.637648 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [None req-3a72c32d-417d-43cf-9c56-864ee5d6c18e admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '064e2f8a197a492eb3225733fab4adda'} {{(pid=92057) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:02:04.645718 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [None req-3a72c32d-417d-43cf-9c56-864ee5d6c18e admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '064e2f8a197a492eb3225733fab4adda'} {{(pid=92057) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:02:04.646412 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [None req-3a72c32d-417d-43cf-9c56-864ee5d6c18e admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '064e2f8a197a492eb3225733fab4adda'} {{(pid=92057) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:02:04.660282 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [None req-3a72c32d-417d-43cf-9c56-864ee5d6c18e admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '064e2f8a197a492eb3225733fab4adda'} {{(pid=92057) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:02:04.666513 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 1140/2269] 10.222.0.30 () {58 vars in 1223 bytes} [Mon May 23 13:02:04 2022] DELETE /identity/v3/users/064e2f8a197a492eb3225733fab4adda => generated 0 bytes in 471 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) May 23 13:02:04.672236 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-814d79db-05f9-4ab7-88b2-ef99b5605328 None None] REQUEST_METHOD: `POST` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:02:04.672731 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-814d79db-05f9-4ab7-88b2-ef99b5605328 None None] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:02:04.672822 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-814d79db-05f9-4ab7-88b2-ef99b5605328 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:02:04.693445 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.auth.core [None req-814d79db-05f9-4ab7-88b2-ef99b5605328 None None] MFA Rules not processed for user `0f7119f8fa5c4445977671140ded91a4`. Rule list: `[]` (Enabled: `True`). {{(pid=92056) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 13:02:04.694938 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.fernet_utils [None req-814d79db-05f9-4ab7-88b2-ef99b5605328 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92056) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:02:04.763896 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 1130/2270] 10.222.0.30 () {58 vars in 957 bytes} [Mon May 23 13:02:04 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 93 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) May 23 13:02:04.774322 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a72c32d-417d-43cf-9c56-864ee5d6c18e admin admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:02:04.775572 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.fernet_utils [None req-3a72c32d-417d-43cf-9c56-864ee5d6c18e admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92057) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:02:04.837910 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a72c32d-417d-43cf-9c56-864ee5d6c18e admin admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:02:04.843105 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b00ae2f-159c-4f97-9c58-6bb7c745e1a5 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2eb83ca38e79449095f3a2f947bbb06a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9c389ae65aaa4542bb5a5c4fd0bacf5b', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:02:04.843732 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b00ae2f-159c-4f97-9c58-6bb7c745e1a5 service neutron] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:02:04.844013 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b00ae2f-159c-4f97-9c58-6bb7c745e1a5 service neutron] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:02:04.844317 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b00ae2f-159c-4f97-9c58-6bb7c745e1a5 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:02:04.853990 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b00ae2f-159c-4f97-9c58-6bb7c745e1a5 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:02:04.855548 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b00ae2f-159c-4f97-9c58-6bb7c745e1a5 service neutron] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:02:04.866381 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 1141/2271] 10.222.0.30 () {60 vars in 1365 bytes} [Mon May 23 13:02:04 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 93 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) May 23 13:02:05.709794 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-814d79db-05f9-4ab7-88b2-ef99b5605328 None None] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:02:05.711799 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.fernet_utils [None req-814d79db-05f9-4ab7-88b2-ef99b5605328 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92056) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:02:05.740451 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-814d79db-05f9-4ab7-88b2-ef99b5605328 None None] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:02:05.764097 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30420365-7aaa-4145-983a-e72fb994d49f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:02:05.764626 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30420365-7aaa-4145-983a-e72fb994d49f admin admin] REQUEST_METHOD: `DELETE` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:02:05.764912 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30420365-7aaa-4145-983a-e72fb994d49f admin admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:02:05.766842 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30420365-7aaa-4145-983a-e72fb994d49f admin admin] PATH_INFO: `/v3/projects/761edc78b23c4faba7aa88f558d3564b` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:02:05.771210 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-30420365-7aaa-4145-983a-e72fb994d49f admin admin] RBAC: Authorizing `identity:delete_project(project_id=761edc78b23c4faba7aa88f558d3564b)` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:02:05.778398 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-30420365-7aaa-4145-983a-e72fb994d49f admin admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:02:05.935676 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=98,keystone:DELETE=2 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 13:02:05.949226 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-30420365-7aaa-4145-983a-e72fb994d49f admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '761edc78b23c4faba7aa88f558d3564b', 'request_id': 'req-30420365-7aaa-4145-983a-e72fb994d49f'} {{(pid=92056) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:02:06.040846 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-30420365-7aaa-4145-983a-e72fb994d49f admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '761edc78b23c4faba7aa88f558d3564b', 'request_id': 'req-30420365-7aaa-4145-983a-e72fb994d49f'} {{(pid=92056) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:02:06.048828 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-30420365-7aaa-4145-983a-e72fb994d49f admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '761edc78b23c4faba7aa88f558d3564b', 'request_id': 'req-30420365-7aaa-4145-983a-e72fb994d49f'} {{(pid=92056) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:02:06.061473 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-30420365-7aaa-4145-983a-e72fb994d49f admin admin] The token cache is being invalidate because project 761edc78b23c4faba7aa88f558d3564b was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=92056) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} May 23 13:02:06.061682 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-30420365-7aaa-4145-983a-e72fb994d49f admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=92056) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:02:06.062484 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-30420365-7aaa-4145-983a-e72fb994d49f admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=92056) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:02:06.066946 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 1131/2272] 10.222.0.30 () {58 vars in 1232 bytes} [Mon May 23 13:02:05 2022] DELETE /identity/v3/projects/761edc78b23c4faba7aa88f558d3564b => generated 0 bytes in 355 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) May 23 13:02:06.072947 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b00ae2f-159c-4f97-9c58-6bb7c745e1a5 service neutron] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:02:06.081292 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b00ae2f-159c-4f97-9c58-6bb7c745e1a5 service neutron] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:02:06.188421 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c57bbb3-3bff-439b-aa58-6f12db9ee545 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:02:06.189630 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c57bbb3-3bff-439b-aa58-6f12db9ee545 admin admin] REQUEST_METHOD: `DELETE` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:02:06.190058 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c57bbb3-3bff-439b-aa58-6f12db9ee545 admin admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:02:06.190528 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c57bbb3-3bff-439b-aa58-6f12db9ee545 admin admin] PATH_INFO: `/v3/users/a008c5d341bc40ada155ef0049fc94a1` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:02:06.192453 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c57bbb3-3bff-439b-aa58-6f12db9ee545 admin admin] RBAC: Authorizing `identity:delete_user(user_id=a008c5d341bc40ada155ef0049fc94a1)` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:02:06.195244 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c57bbb3-3bff-439b-aa58-6f12db9ee545 admin admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:02:06.203159 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 13:02:06.393733 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [None req-2c57bbb3-3bff-439b-aa58-6f12db9ee545 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'a008c5d341bc40ada155ef0049fc94a1'} {{(pid=92057) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:02:06.454759 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [None req-2c57bbb3-3bff-439b-aa58-6f12db9ee545 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'a008c5d341bc40ada155ef0049fc94a1'} {{(pid=92057) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:02:06.463849 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [None req-2c57bbb3-3bff-439b-aa58-6f12db9ee545 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'a008c5d341bc40ada155ef0049fc94a1'} {{(pid=92057) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:02:06.464680 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [None req-2c57bbb3-3bff-439b-aa58-6f12db9ee545 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'a008c5d341bc40ada155ef0049fc94a1'} {{(pid=92057) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:02:06.483132 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [None req-2c57bbb3-3bff-439b-aa58-6f12db9ee545 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'a008c5d341bc40ada155ef0049fc94a1'} {{(pid=92057) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:02:06.500390 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 1142/2273] 10.222.0.30 () {58 vars in 1223 bytes} [Mon May 23 13:02:06 2022] DELETE /identity/v3/users/a008c5d341bc40ada155ef0049fc94a1 => generated 0 bytes in 433 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) May 23 13:02:07.018832 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30420365-7aaa-4145-983a-e72fb994d49f admin admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:02:07.020574 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.fernet_utils [None req-30420365-7aaa-4145-983a-e72fb994d49f admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92056) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:02:07.040221 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30420365-7aaa-4145-983a-e72fb994d49f admin admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:02:07.113577 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9bb02abf-a474-4b63-b31c-50a9d542b4ce admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:02:07.114609 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bb02abf-a474-4b63-b31c-50a9d542b4ce admin admin] REQUEST_METHOD: `DELETE` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:02:07.114948 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bb02abf-a474-4b63-b31c-50a9d542b4ce admin admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:02:07.115464 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bb02abf-a474-4b63-b31c-50a9d542b4ce admin admin] PATH_INFO: `/v3/projects/a31b7b2c9fc54259a9e8d676b1dfc52b` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:02:07.116928 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9bb02abf-a474-4b63-b31c-50a9d542b4ce admin admin] RBAC: Authorizing `identity:delete_project(project_id=a31b7b2c9fc54259a9e8d676b1dfc52b)` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:02:07.119046 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9bb02abf-a474-4b63-b31c-50a9d542b4ce admin admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:02:07.295943 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-9bb02abf-a474-4b63-b31c-50a9d542b4ce admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'a31b7b2c9fc54259a9e8d676b1dfc52b', 'request_id': 'req-9bb02abf-a474-4b63-b31c-50a9d542b4ce'} {{(pid=92056) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:02:07.368784 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-9bb02abf-a474-4b63-b31c-50a9d542b4ce admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'a31b7b2c9fc54259a9e8d676b1dfc52b', 'request_id': 'req-9bb02abf-a474-4b63-b31c-50a9d542b4ce'} {{(pid=92056) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:02:07.375503 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-9bb02abf-a474-4b63-b31c-50a9d542b4ce admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'a31b7b2c9fc54259a9e8d676b1dfc52b', 'request_id': 'req-9bb02abf-a474-4b63-b31c-50a9d542b4ce'} {{(pid=92056) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:02:07.385651 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-9bb02abf-a474-4b63-b31c-50a9d542b4ce admin admin] The token cache is being invalidate because project a31b7b2c9fc54259a9e8d676b1dfc52b was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=92056) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} May 23 13:02:07.385948 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-9bb02abf-a474-4b63-b31c-50a9d542b4ce admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=92056) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:02:07.386576 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-9bb02abf-a474-4b63-b31c-50a9d542b4ce admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=92056) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:02:07.388348 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 1132/2274] 10.222.0.30 () {58 vars in 1232 bytes} [Mon May 23 13:02:07 2022] DELETE /identity/v3/projects/a31b7b2c9fc54259a9e8d676b1dfc52b => generated 0 bytes in 370 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) May 23 13:02:07.393439 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fb7ee3c-ed5a-45f3-bcd5-ab61eb568b71 None None] REQUEST_METHOD: `POST` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:02:07.393940 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fb7ee3c-ed5a-45f3-bcd5-ab61eb568b71 None None] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:02:07.394048 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fb7ee3c-ed5a-45f3-bcd5-ab61eb568b71 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:02:07.412163 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.auth.core [None req-9fb7ee3c-ed5a-45f3-bcd5-ab61eb568b71 None None] MFA Rules not processed for user `0f7119f8fa5c4445977671140ded91a4`. Rule list: `[]` (Enabled: `True`). {{(pid=92057) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 13:02:07.414003 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.fernet_utils [None req-9fb7ee3c-ed5a-45f3-bcd5-ab61eb568b71 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92057) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:02:07.475412 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 1143/2275] 10.222.0.30 () {58 vars in 957 bytes} [Mon May 23 13:02:07 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 83 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) May 23 13:02:07.483314 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81ae902b-3063-4939-91ee-c11b27a17733 None None] REQUEST_METHOD: `POST` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:02:07.483654 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81ae902b-3063-4939-91ee-c11b27a17733 None None] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:02:07.483935 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81ae902b-3063-4939-91ee-c11b27a17733 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:02:07.506636 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.auth.core [None req-81ae902b-3063-4939-91ee-c11b27a17733 None None] MFA Rules not processed for user `0f7119f8fa5c4445977671140ded91a4`. Rule list: `[]` (Enabled: `True`). {{(pid=92056) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 13:02:07.508338 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.fernet_utils [None req-81ae902b-3063-4939-91ee-c11b27a17733 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92056) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:02:07.521214 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 1133/2276] 10.222.0.30 () {58 vars in 957 bytes} [Mon May 23 13:02:07 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 36 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) May 23 13:02:07.522319 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9fb7ee3c-ed5a-45f3-bcd5-ab61eb568b71 None None] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:02:07.534916 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9fb7ee3c-ed5a-45f3-bcd5-ab61eb568b71 None None] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:02:07.540657 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c71ee28d-5f54-4edf-8282-2f0593886c57 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:02:07.541394 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c71ee28d-5f54-4edf-8282-2f0593886c57 admin admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:02:07.541677 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c71ee28d-5f54-4edf-8282-2f0593886c57 admin admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:02:07.542023 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c71ee28d-5f54-4edf-8282-2f0593886c57 admin admin] PATH_INFO: `/v3/domains` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:02:07.542911 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c71ee28d-5f54-4edf-8282-2f0593886c57 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=92057) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} May 23 13:02:07.543218 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c71ee28d-5f54-4edf-8282-2f0593886c57 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:02:07.545038 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c71ee28d-5f54-4edf-8282-2f0593886c57 admin admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:02:07.564583 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 1144/2277] 10.222.0.30 () {58 vars in 1152 bytes} [Mon May 23 13:02:07 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 43 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) May 23 13:02:07.568784 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81ae902b-3063-4939-91ee-c11b27a17733 None None] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:02:07.578566 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81ae902b-3063-4939-91ee-c11b27a17733 None None] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:02:07.583345 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-930a28d3-f6b5-42c2-89b8-9bbbc551a4f8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:02:07.583969 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-930a28d3-f6b5-42c2-89b8-9bbbc551a4f8 admin admin] REQUEST_METHOD: `POST` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:02:07.586357 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-930a28d3-f6b5-42c2-89b8-9bbbc551a4f8 admin admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:02:07.586357 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-930a28d3-f6b5-42c2-89b8-9bbbc551a4f8 admin admin] PATH_INFO: `/v3/projects` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:02:07.586357 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-930a28d3-f6b5-42c2-89b8-9bbbc551a4f8 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:02:07.587275 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-930a28d3-f6b5-42c2-89b8-9bbbc551a4f8 admin admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:02:07.696556 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 1134/2278] 10.222.0.30 () {60 vars in 1152 bytes} [Mon May 23 13:02:07 2022] POST /identity/v3/projects => generated 373 bytes in 128 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) May 23 13:02:07.701872 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c71ee28d-5f54-4edf-8282-2f0593886c57 admin admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:02:07.711779 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c71ee28d-5f54-4edf-8282-2f0593886c57 admin admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:02:07.770437 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b7cf8f7-bb70-46fb-8cb2-1705ab2e3fab admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:02:07.771100 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b7cf8f7-bb70-46fb-8cb2-1705ab2e3fab admin admin] REQUEST_METHOD: `POST` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:02:07.771417 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b7cf8f7-bb70-46fb-8cb2-1705ab2e3fab admin admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:02:07.771719 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b7cf8f7-bb70-46fb-8cb2-1705ab2e3fab admin admin] PATH_INFO: `/v3/users` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:02:07.772454 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b7cf8f7-bb70-46fb-8cb2-1705ab2e3fab admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:02:07.774395 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b7cf8f7-bb70-46fb-8cb2-1705ab2e3fab admin admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:02:07.785654 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=86,keystone:DELETE=10,keystone:INSERT=4 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 13:02:07.879008 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 1145/2279] 10.222.0.30 () {60 vars in 1143 bytes} [Mon May 23 13:02:07 2022] POST /identity/v3/users => generated 287 bytes in 178 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) May 23 13:02:07.883019 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-930a28d3-f6b5-42c2-89b8-9bbbc551a4f8 admin admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:02:07.894993 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-930a28d3-f6b5-42c2-89b8-9bbbc551a4f8 admin admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:02:07.899836 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13f28b95-a946-4a4a-9358-fe7e987d8675 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:02:07.900512 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13f28b95-a946-4a4a-9358-fe7e987d8675 admin admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:02:07.900791 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13f28b95-a946-4a4a-9358-fe7e987d8675 admin admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:02:07.901065 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13f28b95-a946-4a4a-9358-fe7e987d8675 admin admin] PATH_INFO: `/v3/roles` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:02:07.901787 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13f28b95-a946-4a4a-9358-fe7e987d8675 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:02:07.903789 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13f28b95-a946-4a4a-9358-fe7e987d8675 admin admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:02:07.914595 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 1135/2280] 10.222.0.30 () {58 vars in 1121 bytes} [Mon May 23 13:02:07 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) May 23 13:02:07.918561 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b7cf8f7-bb70-46fb-8cb2-1705ab2e3fab admin admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:02:07.928923 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b7cf8f7-bb70-46fb-8cb2-1705ab2e3fab admin admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:02:07.933637 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35a471d7-7453-4de1-b940-bb23a289b721 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:02:07.938037 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35a471d7-7453-4de1-b940-bb23a289b721 admin admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:02:07.938037 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35a471d7-7453-4de1-b940-bb23a289b721 admin admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:02:07.938037 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35a471d7-7453-4de1-b940-bb23a289b721 admin admin] PATH_INFO: `/v3/roles` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:02:07.938037 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35a471d7-7453-4de1-b940-bb23a289b721 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:02:07.938037 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35a471d7-7453-4de1-b940-bb23a289b721 admin admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:02:07.948425 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 1146/2281] 10.222.0.30 () {58 vars in 1121 bytes} [Mon May 23 13:02:07 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) May 23 13:02:07.953237 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13f28b95-a946-4a4a-9358-fe7e987d8675 admin admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:02:07.963198 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13f28b95-a946-4a4a-9358-fe7e987d8675 admin admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:02:07.967998 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6455db44-9044-49d1-a0b9-60b3fa7db96b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:02:07.968906 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6455db44-9044-49d1-a0b9-60b3fa7db96b admin admin] REQUEST_METHOD: `PUT` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:02:07.969208 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6455db44-9044-49d1-a0b9-60b3fa7db96b admin admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:02:07.969479 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6455db44-9044-49d1-a0b9-60b3fa7db96b admin admin] PATH_INFO: `/v3/projects/ed17195d8a3d4c5292bfd6099a9df47d/users/ff1f2649136b42a48f953553c309cd42/roles/80fb22507c9a4ff09d44d8f12f6fff1a` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:02:07.972556 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=85,keystone:DELETE=12,keystone:INSERT=3 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 13:02:07.984347 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6455db44-9044-49d1-a0b9-60b3fa7db96b admin admin] RBAC: Authorizing `identity:create_grant(project_id=ed17195d8a3d4c5292bfd6099a9df47d, user_id=ff1f2649136b42a48f953553c309cd42, role_id=80fb22507c9a4ff09d44d8f12f6fff1a)` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:02:07.987033 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6455db44-9044-49d1-a0b9-60b3fa7db96b admin admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:02:08.106159 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 1136/2282] 10.222.0.30 () {60 vars in 1482 bytes} [Mon May 23 13:02:07 2022] PUT /identity/v3/projects/ed17195d8a3d4c5292bfd6099a9df47d/users/ff1f2649136b42a48f953553c309cd42/roles/80fb22507c9a4ff09d44d8f12f6fff1a => generated 0 bytes in 153 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) May 23 13:02:08.111448 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a0bf08c-647c-4df0-ab44-3b98140de5d9 None None] REQUEST_METHOD: `POST` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:02:08.111743 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a0bf08c-647c-4df0-ab44-3b98140de5d9 None None] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:02:08.112026 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a0bf08c-647c-4df0-ab44-3b98140de5d9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:02:08.129971 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.auth.core [None req-2a0bf08c-647c-4df0-ab44-3b98140de5d9 None None] MFA Rules not processed for user `0f7119f8fa5c4445977671140ded91a4`. Rule list: `[]` (Enabled: `True`). {{(pid=92057) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 13:02:08.131090 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.fernet_utils [None req-2a0bf08c-647c-4df0-ab44-3b98140de5d9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92057) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:02:08.193391 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 1147/2283] 10.222.0.30 () {58 vars in 957 bytes} [Mon May 23 13:02:08 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 83 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) May 23 13:02:08.203716 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6455db44-9044-49d1-a0b9-60b3fa7db96b admin admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:02:08.205403 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.fernet_utils [None req-6455db44-9044-49d1-a0b9-60b3fa7db96b admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92056) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:02:08.272420 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6455db44-9044-49d1-a0b9-60b3fa7db96b admin admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:02:08.276084 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78d9fefa-1e5c-4cec-84ca-d230cfcfd0bf service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2eb83ca38e79449095f3a2f947bbb06a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9c389ae65aaa4542bb5a5c4fd0bacf5b', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:02:08.276733 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78d9fefa-1e5c-4cec-84ca-d230cfcfd0bf service neutron] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:02:08.277016 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78d9fefa-1e5c-4cec-84ca-d230cfcfd0bf service neutron] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:02:08.277292 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78d9fefa-1e5c-4cec-84ca-d230cfcfd0bf service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:02:08.288633 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-78d9fefa-1e5c-4cec-84ca-d230cfcfd0bf service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:02:08.290115 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-78d9fefa-1e5c-4cec-84ca-d230cfcfd0bf service neutron] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:02:08.300924 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 1137/2284] 10.222.0.30 () {60 vars in 1365 bytes} [Mon May 23 13:02:08 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 98 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) May 23 13:02:09.327501 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a0bf08c-647c-4df0-ab44-3b98140de5d9 None None] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:02:09.328947 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.fernet_utils [None req-2a0bf08c-647c-4df0-ab44-3b98140de5d9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92057) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:02:09.341394 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a0bf08c-647c-4df0-ab44-3b98140de5d9 None None] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:02:09.346059 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a3c444a-9342-4cd8-80b9-9ab6bb603384 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:02:09.346926 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a3c444a-9342-4cd8-80b9-9ab6bb603384 admin admin] REQUEST_METHOD: `DELETE` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:02:09.347208 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a3c444a-9342-4cd8-80b9-9ab6bb603384 admin admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:02:09.347485 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a3c444a-9342-4cd8-80b9-9ab6bb603384 admin admin] PATH_INFO: `/v3/users/7ce98e71dfe94015be955ec759a9e312` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:02:09.348725 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a3c444a-9342-4cd8-80b9-9ab6bb603384 admin admin] RBAC: Authorizing `identity:delete_user(user_id=7ce98e71dfe94015be955ec759a9e312)` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:02:09.350679 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a3c444a-9342-4cd8-80b9-9ab6bb603384 admin admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:02:09.558482 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [None req-9a3c444a-9342-4cd8-80b9-9ab6bb603384 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '7ce98e71dfe94015be955ec759a9e312'} {{(pid=92057) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:02:09.629234 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [None req-9a3c444a-9342-4cd8-80b9-9ab6bb603384 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '7ce98e71dfe94015be955ec759a9e312'} {{(pid=92057) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:02:09.638989 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [None req-9a3c444a-9342-4cd8-80b9-9ab6bb603384 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '7ce98e71dfe94015be955ec759a9e312'} {{(pid=92057) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:02:09.639850 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [None req-9a3c444a-9342-4cd8-80b9-9ab6bb603384 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '7ce98e71dfe94015be955ec759a9e312'} {{(pid=92057) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:02:09.654993 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [None req-9a3c444a-9342-4cd8-80b9-9ab6bb603384 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '7ce98e71dfe94015be955ec759a9e312'} {{(pid=92057) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:02:09.667940 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 1148/2285] 10.222.0.30 () {58 vars in 1223 bytes} [Mon May 23 13:02:09 2022] DELETE /identity/v3/users/7ce98e71dfe94015be955ec759a9e312 => generated 0 bytes in 336 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) May 23 13:02:09.677729 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffa7b6c4-13ec-4a39-bba1-f84fead61dd5 None None] REQUEST_METHOD: `POST` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:02:09.678308 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffa7b6c4-13ec-4a39-bba1-f84fead61dd5 None None] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:02:09.678308 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffa7b6c4-13ec-4a39-bba1-f84fead61dd5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:02:09.684760 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad80699b-079f-4782-85ea-de9f3549b08a None None] REQUEST_METHOD: `POST` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:02:09.684760 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad80699b-079f-4782-85ea-de9f3549b08a None None] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:02:09.684760 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad80699b-079f-4782-85ea-de9f3549b08a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:02:09.708197 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.auth.core [None req-ffa7b6c4-13ec-4a39-bba1-f84fead61dd5 None None] MFA Rules not processed for user `0f7119f8fa5c4445977671140ded91a4`. Rule list: `[]` (Enabled: `True`). {{(pid=92056) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 13:02:09.709857 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.fernet_utils [None req-ffa7b6c4-13ec-4a39-bba1-f84fead61dd5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92056) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:02:09.755720 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.auth.core [None req-ad80699b-079f-4782-85ea-de9f3549b08a None None] MFA Rules not processed for user `ff1f2649136b42a48f953553c309cd42`. Rule list: `[]` (Enabled: `True`). {{(pid=92057) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 13:02:09.761633 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.fernet_utils [None req-ad80699b-079f-4782-85ea-de9f3549b08a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92057) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:02:09.800602 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 1138/2286] 10.222.0.30 () {58 vars in 957 bytes} [Mon May 23 13:02:09 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 124 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) May 23 13:02:09.811291 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ffa7b6c4-13ec-4a39-bba1-f84fead61dd5 None None] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:02:09.813736 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.fernet_utils [None req-ffa7b6c4-13ec-4a39-bba1-f84fead61dd5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92056) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:02:09.840101 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 13:02:09.860250 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 1149/2287] 10.222.0.30 () {58 vars in 957 bytes} [Mon May 23 13:02:09 2022] POST /identity/v3/auth/tokens => generated 3240 bytes in 183 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) May 23 13:02:09.864653 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad80699b-079f-4782-85ea-de9f3549b08a None None] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:02:09.865994 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.fernet_utils [None req-ad80699b-079f-4782-85ea-de9f3549b08a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92057) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:02:09.875189 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ffa7b6c4-13ec-4a39-bba1-f84fead61dd5 None None] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:02:09.879051 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87691de3-5be6-4d24-9e60-2fc0effe1c9b service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2eb83ca38e79449095f3a2f947bbb06a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9c389ae65aaa4542bb5a5c4fd0bacf5b', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:02:09.879695 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87691de3-5be6-4d24-9e60-2fc0effe1c9b service neutron] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:02:09.879975 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87691de3-5be6-4d24-9e60-2fc0effe1c9b service neutron] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:02:09.880251 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87691de3-5be6-4d24-9e60-2fc0effe1c9b service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:02:09.881642 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad80699b-079f-4782-85ea-de9f3549b08a None None] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:02:09.886347 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc59fc10-9b81-48df-bd00-9eeb33097a84 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:02:09.893660 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc59fc10-9b81-48df-bd00-9eeb33097a84 admin admin] REQUEST_METHOD: `POST` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:02:09.893660 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc59fc10-9b81-48df-bd00-9eeb33097a84 admin admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:02:09.893660 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc59fc10-9b81-48df-bd00-9eeb33097a84 admin admin] PATH_INFO: `/v3/projects` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:02:09.893660 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc59fc10-9b81-48df-bd00-9eeb33097a84 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:02:09.894113 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87691de3-5be6-4d24-9e60-2fc0effe1c9b service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:02:09.894207 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc59fc10-9b81-48df-bd00-9eeb33097a84 admin admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:02:09.894301 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87691de3-5be6-4d24-9e60-2fc0effe1c9b service neutron] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:02:09.901972 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 1139/2288] 10.222.0.30 () {60 vars in 1365 bytes} [Mon May 23 13:02:09 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 91 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) May 23 13:02:10.022778 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 1150/2289] 10.222.0.30 () {60 vars in 1152 bytes} [Mon May 23 13:02:09 2022] POST /identity/v3/projects => generated 373 bytes in 158 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) May 23 13:02:10.026757 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87691de3-5be6-4d24-9e60-2fc0effe1c9b service neutron] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:02:10.036556 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87691de3-5be6-4d24-9e60-2fc0effe1c9b service neutron] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:02:10.067133 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 13:02:10.094425 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-add1f380-015e-46e2-a8ad-2726f57673ad admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:02:10.095086 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-add1f380-015e-46e2-a8ad-2726f57673ad admin admin] REQUEST_METHOD: `POST` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:02:10.095367 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-add1f380-015e-46e2-a8ad-2726f57673ad admin admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:02:10.095639 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-add1f380-015e-46e2-a8ad-2726f57673ad admin admin] PATH_INFO: `/v3/users` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:02:10.096432 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-add1f380-015e-46e2-a8ad-2726f57673ad admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:02:10.098370 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-add1f380-015e-46e2-a8ad-2726f57673ad admin admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:02:10.184221 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 1140/2290] 10.222.0.30 () {60 vars in 1143 bytes} [Mon May 23 13:02:10 2022] POST /identity/v3/users => generated 293 bytes in 158 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) May 23 13:02:10.188758 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc59fc10-9b81-48df-bd00-9eeb33097a84 admin admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:02:10.199994 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc59fc10-9b81-48df-bd00-9eeb33097a84 admin admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:02:10.205535 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7819b17d-359a-48e2-a078-7061e14b306e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:02:10.206310 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7819b17d-359a-48e2-a078-7061e14b306e admin admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:02:10.206643 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7819b17d-359a-48e2-a078-7061e14b306e admin admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:02:10.206986 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7819b17d-359a-48e2-a078-7061e14b306e admin admin] PATH_INFO: `/v3/roles` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:02:10.207715 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7819b17d-359a-48e2-a078-7061e14b306e admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:02:10.209392 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7819b17d-359a-48e2-a078-7061e14b306e admin admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:02:10.220749 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 1151/2291] 10.222.0.30 () {58 vars in 1121 bytes} [Mon May 23 13:02:10 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) May 23 13:02:10.226290 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-add1f380-015e-46e2-a8ad-2726f57673ad admin admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:02:10.237614 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-add1f380-015e-46e2-a8ad-2726f57673ad admin admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:02:10.242424 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef7b8910-3364-4d29-9f19-a41fe6f5a481 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:02:10.243309 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef7b8910-3364-4d29-9f19-a41fe6f5a481 admin admin] REQUEST_METHOD: `PUT` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:02:10.243715 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef7b8910-3364-4d29-9f19-a41fe6f5a481 admin admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:02:10.243998 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef7b8910-3364-4d29-9f19-a41fe6f5a481 admin admin] PATH_INFO: `/v3/projects/6f0b7e44a32f4307a7893046f807fbfa/users/aa81faad75874e9e8139724b9f4ab6d0/roles/bc3c3c9d654940cbaa5177a6be23d1bc` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:02:10.261065 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef7b8910-3364-4d29-9f19-a41fe6f5a481 admin admin] RBAC: Authorizing `identity:create_grant(project_id=6f0b7e44a32f4307a7893046f807fbfa, user_id=aa81faad75874e9e8139724b9f4ab6d0, role_id=bc3c3c9d654940cbaa5177a6be23d1bc)` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:02:10.273504 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef7b8910-3364-4d29-9f19-a41fe6f5a481 admin admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:02:10.359324 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7819b17d-359a-48e2-a078-7061e14b306e admin admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:02:10.361299 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.fernet_utils [None req-7819b17d-359a-48e2-a078-7061e14b306e admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92057) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:02:10.375411 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7819b17d-359a-48e2-a078-7061e14b306e admin admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:02:10.381975 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3bc24228-23eb-4ea4-8227-0b3954049f59 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:02:10.383113 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bc24228-23eb-4ea4-8227-0b3954049f59 admin admin] REQUEST_METHOD: `DELETE` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:02:10.383533 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bc24228-23eb-4ea4-8227-0b3954049f59 admin admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:02:10.383900 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bc24228-23eb-4ea4-8227-0b3954049f59 admin admin] PATH_INFO: `/v3/projects/d177483c7cf24ca7aadd852a66c2e427` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:02:10.385341 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3bc24228-23eb-4ea4-8227-0b3954049f59 admin admin] RBAC: Authorizing `identity:delete_project(project_id=d177483c7cf24ca7aadd852a66c2e427)` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:02:10.386171 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 1141/2292] 10.222.0.30 () {60 vars in 1482 bytes} [Mon May 23 13:02:10 2022] PUT /identity/v3/projects/6f0b7e44a32f4307a7893046f807fbfa/users/aa81faad75874e9e8139724b9f4ab6d0/roles/bc3c3c9d654940cbaa5177a6be23d1bc => generated 0 bytes in 160 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) May 23 13:02:10.387842 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3bc24228-23eb-4ea4-8227-0b3954049f59 admin admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:02:10.398402 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef7b8910-3364-4d29-9f19-a41fe6f5a481 admin admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:02:10.468022 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef7b8910-3364-4d29-9f19-a41fe6f5a481 admin admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:02:10.472717 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dea73a58-149f-4806-858a-c6e405a9e913 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:02:10.473375 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dea73a58-149f-4806-858a-c6e405a9e913 admin admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:02:10.473658 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dea73a58-149f-4806-858a-c6e405a9e913 admin admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:02:10.473970 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dea73a58-149f-4806-858a-c6e405a9e913 admin admin] PATH_INFO: `/v3/roles` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:02:10.474698 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dea73a58-149f-4806-858a-c6e405a9e913 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:02:10.476360 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dea73a58-149f-4806-858a-c6e405a9e913 admin admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:02:10.486342 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 1142/2293] 10.222.0.30 () {58 vars in 1121 bytes} [Mon May 23 13:02:10 2022] GET /identity/v3/roles => generated 1420 bytes in 97 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) May 23 13:02:10.490276 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dea73a58-149f-4806-858a-c6e405a9e913 admin admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:02:10.499683 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dea73a58-149f-4806-858a-c6e405a9e913 admin admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:02:10.504460 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69d5f3cf-ab2b-4c30-bb06-601d5ea34a05 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:02:10.505323 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69d5f3cf-ab2b-4c30-bb06-601d5ea34a05 admin admin] REQUEST_METHOD: `PUT` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:02:10.505604 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69d5f3cf-ab2b-4c30-bb06-601d5ea34a05 admin admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:02:10.505905 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69d5f3cf-ab2b-4c30-bb06-601d5ea34a05 admin admin] PATH_INFO: `/v3/projects/6f0b7e44a32f4307a7893046f807fbfa/users/aa81faad75874e9e8139724b9f4ab6d0/roles/bc3c3c9d654940cbaa5177a6be23d1bc` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:02:10.508434 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69d5f3cf-ab2b-4c30-bb06-601d5ea34a05 admin admin] RBAC: Authorizing `identity:create_grant(project_id=6f0b7e44a32f4307a7893046f807fbfa, user_id=aa81faad75874e9e8139724b9f4ab6d0, role_id=bc3c3c9d654940cbaa5177a6be23d1bc)` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:02:10.511205 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69d5f3cf-ab2b-4c30-bb06-601d5ea34a05 admin admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:02:10.516085 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [None req-3bc24228-23eb-4ea4-8227-0b3954049f59 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'd177483c7cf24ca7aadd852a66c2e427', 'request_id': 'req-3bc24228-23eb-4ea4-8227-0b3954049f59'} {{(pid=92057) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:02:10.521662 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [None req-3bc24228-23eb-4ea4-8227-0b3954049f59 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'd177483c7cf24ca7aadd852a66c2e427', 'request_id': 'req-3bc24228-23eb-4ea4-8227-0b3954049f59'} {{(pid=92057) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:02:10.523066 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 1143/2294] 10.222.0.30 () {60 vars in 1482 bytes} [Mon May 23 13:02:10 2022] PUT /identity/v3/projects/6f0b7e44a32f4307a7893046f807fbfa/users/aa81faad75874e9e8139724b9f4ab6d0/roles/bc3c3c9d654940cbaa5177a6be23d1bc => generated 0 bytes in 33 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) May 23 13:02:10.600491 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [None req-3bc24228-23eb-4ea4-8227-0b3954049f59 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'd177483c7cf24ca7aadd852a66c2e427', 'request_id': 'req-3bc24228-23eb-4ea4-8227-0b3954049f59'} {{(pid=92057) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:02:10.612684 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [None req-3bc24228-23eb-4ea4-8227-0b3954049f59 admin admin] The token cache is being invalidate because project d177483c7cf24ca7aadd852a66c2e427 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=92057) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} May 23 13:02:10.612970 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [None req-3bc24228-23eb-4ea4-8227-0b3954049f59 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=92057) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:02:10.613714 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [None req-3bc24228-23eb-4ea4-8227-0b3954049f59 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=92057) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:02:10.615380 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 1152/2295] 10.222.0.30 () {58 vars in 1232 bytes} [Mon May 23 13:02:10 2022] DELETE /identity/v3/projects/d177483c7cf24ca7aadd852a66c2e427 => generated 0 bytes in 257 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) May 23 13:02:10.619419 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69d5f3cf-ab2b-4c30-bb06-601d5ea34a05 admin admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:02:10.621166 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.fernet_utils [None req-69d5f3cf-ab2b-4c30-bb06-601d5ea34a05 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92056) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:02:10.720661 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69d5f3cf-ab2b-4c30-bb06-601d5ea34a05 admin admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:02:10.726555 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae08c359-8013-4ecf-83d7-89d52006ee63 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:02:10.727774 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae08c359-8013-4ecf-83d7-89d52006ee63 admin admin] REQUEST_METHOD: `DELETE` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:02:10.727882 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae08c359-8013-4ecf-83d7-89d52006ee63 admin admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:02:10.728416 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae08c359-8013-4ecf-83d7-89d52006ee63 admin admin] PATH_INFO: `/v3/users/c0047fb72e0e481981770285e6dacf33` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:02:10.729436 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae08c359-8013-4ecf-83d7-89d52006ee63 admin admin] RBAC: Authorizing `identity:delete_user(user_id=c0047fb72e0e481981770285e6dacf33)` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:02:10.732160 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae08c359-8013-4ecf-83d7-89d52006ee63 admin admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:02:10.758759 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=92,keystone:INSERT=5,keystone:DELETE=3 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 13:02:10.968424 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-ae08c359-8013-4ecf-83d7-89d52006ee63 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'c0047fb72e0e481981770285e6dacf33'} {{(pid=92056) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:02:10.970921 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-ae08c359-8013-4ecf-83d7-89d52006ee63 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'c0047fb72e0e481981770285e6dacf33'} {{(pid=92056) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:02:11.046554 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-ae08c359-8013-4ecf-83d7-89d52006ee63 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'c0047fb72e0e481981770285e6dacf33'} {{(pid=92056) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:02:11.059505 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-ae08c359-8013-4ecf-83d7-89d52006ee63 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'c0047fb72e0e481981770285e6dacf33'} {{(pid=92056) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:02:11.066680 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-ae08c359-8013-4ecf-83d7-89d52006ee63 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'c0047fb72e0e481981770285e6dacf33'} {{(pid=92056) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:02:11.072847 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 1144/2296] 10.222.0.30 () {58 vars in 1223 bytes} [Mon May 23 13:02:10 2022] DELETE /identity/v3/users/c0047fb72e0e481981770285e6dacf33 => generated 0 bytes in 454 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) May 23 13:02:11.541822 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3bc24228-23eb-4ea4-8227-0b3954049f59 admin admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:02:11.611069 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3bc24228-23eb-4ea4-8227-0b3954049f59 admin admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:02:11.655448 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=91,keystone:INSERT=2,keystone:DELETE=7 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 13:02:11.679428 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2bab8ccc-2259-4abb-8b2b-ba5b17f5ff29 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:02:11.679428 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bab8ccc-2259-4abb-8b2b-ba5b17f5ff29 admin admin] REQUEST_METHOD: `DELETE` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:02:11.679428 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bab8ccc-2259-4abb-8b2b-ba5b17f5ff29 admin admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:02:11.679428 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bab8ccc-2259-4abb-8b2b-ba5b17f5ff29 admin admin] PATH_INFO: `/v3/projects/1b0d1830da054b8599242c45f207e1ac` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:02:11.679835 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2bab8ccc-2259-4abb-8b2b-ba5b17f5ff29 admin admin] RBAC: Authorizing `identity:delete_project(project_id=1b0d1830da054b8599242c45f207e1ac)` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:02:11.679835 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2bab8ccc-2259-4abb-8b2b-ba5b17f5ff29 admin admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:02:11.803002 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [None req-2bab8ccc-2259-4abb-8b2b-ba5b17f5ff29 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '1b0d1830da054b8599242c45f207e1ac', 'request_id': 'req-2bab8ccc-2259-4abb-8b2b-ba5b17f5ff29'} {{(pid=92057) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:02:11.808512 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [None req-2bab8ccc-2259-4abb-8b2b-ba5b17f5ff29 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '1b0d1830da054b8599242c45f207e1ac', 'request_id': 'req-2bab8ccc-2259-4abb-8b2b-ba5b17f5ff29'} {{(pid=92057) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:02:11.870893 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [None req-2bab8ccc-2259-4abb-8b2b-ba5b17f5ff29 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '1b0d1830da054b8599242c45f207e1ac', 'request_id': 'req-2bab8ccc-2259-4abb-8b2b-ba5b17f5ff29'} {{(pid=92057) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:02:11.898243 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [None req-2bab8ccc-2259-4abb-8b2b-ba5b17f5ff29 admin admin] The token cache is being invalidate because project 1b0d1830da054b8599242c45f207e1ac was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=92057) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} May 23 13:02:11.901046 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [None req-2bab8ccc-2259-4abb-8b2b-ba5b17f5ff29 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=92057) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:02:11.901749 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [None req-2bab8ccc-2259-4abb-8b2b-ba5b17f5ff29 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=92057) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:02:11.903549 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 1153/2297] 10.222.0.30 () {58 vars in 1232 bytes} [Mon May 23 13:02:11 2022] DELETE /identity/v3/projects/1b0d1830da054b8599242c45f207e1ac => generated 0 bytes in 362 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) May 23 13:02:11.907667 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae08c359-8013-4ecf-83d7-89d52006ee63 admin admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:02:11.909599 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.fernet_utils [None req-ae08c359-8013-4ecf-83d7-89d52006ee63 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92056) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:02:11.937830 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae08c359-8013-4ecf-83d7-89d52006ee63 admin admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:02:12.029840 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74f1acff-1dcb-4f52-8b55-65cb65d03d6f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:02:12.031437 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74f1acff-1dcb-4f52-8b55-65cb65d03d6f admin admin] REQUEST_METHOD: `DELETE` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:02:12.031718 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74f1acff-1dcb-4f52-8b55-65cb65d03d6f admin admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:02:12.031990 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74f1acff-1dcb-4f52-8b55-65cb65d03d6f admin admin] PATH_INFO: `/v3/users/3a0c2e6cd69b49e588b83c437b4a1dbe` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:02:12.033250 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74f1acff-1dcb-4f52-8b55-65cb65d03d6f admin admin] RBAC: Authorizing `identity:delete_user(user_id=3a0c2e6cd69b49e588b83c437b4a1dbe)` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:02:12.035962 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74f1acff-1dcb-4f52-8b55-65cb65d03d6f admin admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:02:12.049399 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ac6becd-1b5b-4867-9885-39682dc6277f None None] REQUEST_METHOD: `POST` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:02:12.049399 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ac6becd-1b5b-4867-9885-39682dc6277f None None] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:02:12.049676 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ac6becd-1b5b-4867-9885-39682dc6277f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:02:12.091930 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.auth.core [None req-4ac6becd-1b5b-4867-9885-39682dc6277f None None] MFA Rules not processed for user `aa81faad75874e9e8139724b9f4ab6d0`. Rule list: `[]` (Enabled: `True`). {{(pid=92057) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 13:02:12.093234 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.fernet_utils [None req-4ac6becd-1b5b-4867-9885-39682dc6277f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92057) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:02:12.204399 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 1154/2298] 10.222.0.30 () {58 vars in 957 bytes} [Mon May 23 13:02:12 2022] POST /identity/v3/auth/tokens => generated 3307 bytes in 154 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) May 23 13:02:12.214050 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ac6becd-1b5b-4867-9885-39682dc6277f None None] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:02:12.216060 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.fernet_utils [None req-4ac6becd-1b5b-4867-9885-39682dc6277f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92057) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:02:12.274752 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-74f1acff-1dcb-4f52-8b55-65cb65d03d6f admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '3a0c2e6cd69b49e588b83c437b4a1dbe'} {{(pid=92056) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:02:12.275451 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-74f1acff-1dcb-4f52-8b55-65cb65d03d6f admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '3a0c2e6cd69b49e588b83c437b4a1dbe'} {{(pid=92056) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:02:12.292976 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 13:02:12.307664 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ac6becd-1b5b-4867-9885-39682dc6277f None None] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:02:12.313978 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae42c3d1-4609-42e8-be32-53db13341881 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1666097f8ed545c6b0e4809c93e4775a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9c389ae65aaa4542bb5a5c4fd0bacf5b', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:02:12.318970 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae42c3d1-4609-42e8-be32-53db13341881 service cinder] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:02:12.318970 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae42c3d1-4609-42e8-be32-53db13341881 service cinder] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:02:12.318970 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae42c3d1-4609-42e8-be32-53db13341881 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:02:12.318970 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.fernet_utils [None req-ae42c3d1-4609-42e8-be32-53db13341881 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92057) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:02:12.387962 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae42c3d1-4609-42e8-be32-53db13341881 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:02:12.389642 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae42c3d1-4609-42e8-be32-53db13341881 service cinder] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:02:12.395374 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-74f1acff-1dcb-4f52-8b55-65cb65d03d6f admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '3a0c2e6cd69b49e588b83c437b4a1dbe'} {{(pid=92056) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:02:12.403613 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 1155/2299] 10.222.0.30 () {60 vars in 1365 bytes} [Mon May 23 13:02:12 2022] GET /identity/v3/auth/tokens => generated 3240 bytes in 191 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) May 23 13:02:12.412184 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-74f1acff-1dcb-4f52-8b55-65cb65d03d6f admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '3a0c2e6cd69b49e588b83c437b4a1dbe'} {{(pid=92056) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:02:12.431351 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-74f1acff-1dcb-4f52-8b55-65cb65d03d6f admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '3a0c2e6cd69b49e588b83c437b4a1dbe'} {{(pid=92056) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:02:12.439783 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 1145/2300] 10.222.0.30 () {58 vars in 1223 bytes} [Mon May 23 13:02:11 2022] DELETE /identity/v3/users/3a0c2e6cd69b49e588b83c437b4a1dbe => generated 0 bytes in 532 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) May 23 13:02:12.860209 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae42c3d1-4609-42e8-be32-53db13341881 service cinder] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:02:12.861516 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.fernet_utils [None req-ae42c3d1-4609-42e8-be32-53db13341881 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92057) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:02:12.896434 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae42c3d1-4609-42e8-be32-53db13341881 service cinder] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:02:12.902056 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bda8347b-fa31-481f-86a2-f88f840f216b None None] REQUEST_METHOD: `POST` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:02:12.902399 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bda8347b-fa31-481f-86a2-f88f840f216b None None] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:02:12.910463 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bda8347b-fa31-481f-86a2-f88f840f216b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:02:12.933834 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.auth.core [None req-bda8347b-fa31-481f-86a2-f88f840f216b None None] MFA Rules not processed for user `c6a6096760734ac49ce2f26c6065b701`. Rule list: `[]` (Enabled: `True`). {{(pid=92056) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 13:02:12.935731 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.fernet_utils [None req-bda8347b-fa31-481f-86a2-f88f840f216b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92056) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:02:12.967861 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:DELETE=17,keystone:SELECT=81,keystone:INSERT=2 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 13:02:12.976423 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b753cc38-bcef-45ad-aa04-48f05dc09aac admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:02:12.977252 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b753cc38-bcef-45ad-aa04-48f05dc09aac admin admin] REQUEST_METHOD: `DELETE` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:02:12.977537 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b753cc38-bcef-45ad-aa04-48f05dc09aac admin admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:02:12.977813 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b753cc38-bcef-45ad-aa04-48f05dc09aac admin admin] PATH_INFO: `/v3/projects/b3c26b67bf144c619b6db450170bbf7d` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:02:12.979151 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b753cc38-bcef-45ad-aa04-48f05dc09aac admin admin] RBAC: Authorizing `identity:delete_project(project_id=b3c26b67bf144c619b6db450170bbf7d)` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:02:12.981635 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b753cc38-bcef-45ad-aa04-48f05dc09aac admin admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:02:13.050362 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 1146/2301] 10.222.0.30 () {58 vars in 957 bytes} [Mon May 23 13:02:12 2022] POST /identity/v3/auth/tokens => generated 3246 bytes in 150 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) May 23 13:02:13.102346 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [None req-b753cc38-bcef-45ad-aa04-48f05dc09aac admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'b3c26b67bf144c619b6db450170bbf7d', 'request_id': 'req-b753cc38-bcef-45ad-aa04-48f05dc09aac'} {{(pid=92057) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:02:13.110368 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [None req-b753cc38-bcef-45ad-aa04-48f05dc09aac admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'b3c26b67bf144c619b6db450170bbf7d', 'request_id': 'req-b753cc38-bcef-45ad-aa04-48f05dc09aac'} {{(pid=92057) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:02:13.223576 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [None req-b753cc38-bcef-45ad-aa04-48f05dc09aac admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'b3c26b67bf144c619b6db450170bbf7d', 'request_id': 'req-b753cc38-bcef-45ad-aa04-48f05dc09aac'} {{(pid=92057) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:02:13.240791 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [None req-b753cc38-bcef-45ad-aa04-48f05dc09aac admin admin] The token cache is being invalidate because project b3c26b67bf144c619b6db450170bbf7d was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=92057) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} May 23 13:02:13.241169 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [None req-b753cc38-bcef-45ad-aa04-48f05dc09aac admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=92057) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:02:13.242039 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [None req-b753cc38-bcef-45ad-aa04-48f05dc09aac admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=92057) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:02:13.247834 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 1156/2302] 10.222.0.30 () {58 vars in 1232 bytes} [Mon May 23 13:02:12 2022] DELETE /identity/v3/projects/b3c26b67bf144c619b6db450170bbf7d => generated 0 bytes in 387 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) May 23 13:02:13.251277 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b32ddd39-3247-470c-a954-cc9dc0863ac2 None None] REQUEST_METHOD: `POST` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:02:13.251277 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b32ddd39-3247-470c-a954-cc9dc0863ac2 None None] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:02:13.251277 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b32ddd39-3247-470c-a954-cc9dc0863ac2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:02:13.274455 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.auth.core [None req-b32ddd39-3247-470c-a954-cc9dc0863ac2 None None] MFA Rules not processed for user `0f7119f8fa5c4445977671140ded91a4`. Rule list: `[]` (Enabled: `True`). {{(pid=92056) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 13:02:13.274455 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.fernet_utils [None req-b32ddd39-3247-470c-a954-cc9dc0863ac2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92056) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:02:13.401911 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 1147/2303] 10.222.0.30 () {58 vars in 957 bytes} [Mon May 23 13:02:13 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 152 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) May 23 13:02:13.410548 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76b1df30-c82a-4cb8-9aa2-1de445ca2fd2 None None] REQUEST_METHOD: `POST` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:02:13.410865 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76b1df30-c82a-4cb8-9aa2-1de445ca2fd2 None None] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:02:13.411264 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76b1df30-c82a-4cb8-9aa2-1de445ca2fd2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:02:13.431401 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.auth.core [None req-76b1df30-c82a-4cb8-9aa2-1de445ca2fd2 None None] MFA Rules not processed for user `0f7119f8fa5c4445977671140ded91a4`. Rule list: `[]` (Enabled: `True`). {{(pid=92057) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 13:02:13.432629 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.fernet_utils [None req-76b1df30-c82a-4cb8-9aa2-1de445ca2fd2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92057) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:02:13.442162 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 1157/2304] 10.222.0.30 () {58 vars in 957 bytes} [Mon May 23 13:02:13 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 33 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) May 23 13:02:13.446678 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b32ddd39-3247-470c-a954-cc9dc0863ac2 None None] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:02:13.457331 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b32ddd39-3247-470c-a954-cc9dc0863ac2 None None] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:02:13.461935 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-099a9fee-eefc-49cb-b85f-80f2ca2390b6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:02:13.462610 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-099a9fee-eefc-49cb-b85f-80f2ca2390b6 admin admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:02:13.463288 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-099a9fee-eefc-49cb-b85f-80f2ca2390b6 admin admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:02:13.463288 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-099a9fee-eefc-49cb-b85f-80f2ca2390b6 admin admin] PATH_INFO: `/v3/domains` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:02:13.463875 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-099a9fee-eefc-49cb-b85f-80f2ca2390b6 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=92056) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} May 23 13:02:13.464185 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-099a9fee-eefc-49cb-b85f-80f2ca2390b6 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:02:13.466616 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-099a9fee-eefc-49cb-b85f-80f2ca2390b6 admin admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:02:13.478918 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 1148/2305] 10.222.0.30 () {58 vars in 1152 bytes} [Mon May 23 13:02:13 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) May 23 13:02:13.484888 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76b1df30-c82a-4cb8-9aa2-1de445ca2fd2 None None] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:02:13.493750 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76b1df30-c82a-4cb8-9aa2-1de445ca2fd2 None None] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:02:13.500031 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ff0fc6a-6839-4e7c-b07c-154a57efd626 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:02:13.500725 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ff0fc6a-6839-4e7c-b07c-154a57efd626 admin admin] REQUEST_METHOD: `POST` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:02:13.501020 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ff0fc6a-6839-4e7c-b07c-154a57efd626 admin admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:02:13.501291 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ff0fc6a-6839-4e7c-b07c-154a57efd626 admin admin] PATH_INFO: `/v3/projects` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:02:13.502214 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ff0fc6a-6839-4e7c-b07c-154a57efd626 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:02:13.504159 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ff0fc6a-6839-4e7c-b07c-154a57efd626 admin admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:02:13.526453 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=91,keystone:DELETE=7,keystone:INSERT=2 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 13:02:13.607091 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 1158/2306] 10.222.0.30 () {60 vars in 1152 bytes} [Mon May 23 13:02:13 2022] POST /identity/v3/projects => generated 381 bytes in 123 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) May 23 13:02:13.611182 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-099a9fee-eefc-49cb-b85f-80f2ca2390b6 admin admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:02:13.621781 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-099a9fee-eefc-49cb-b85f-80f2ca2390b6 admin admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:02:13.700786 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0731ad5-6b6d-4087-b418-2081a341126a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:02:13.702201 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0731ad5-6b6d-4087-b418-2081a341126a admin admin] REQUEST_METHOD: `POST` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:02:13.702730 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0731ad5-6b6d-4087-b418-2081a341126a admin admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:02:13.703150 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0731ad5-6b6d-4087-b418-2081a341126a admin admin] PATH_INFO: `/v3/users` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:02:13.704695 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f0731ad5-6b6d-4087-b418-2081a341126a admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:02:13.706894 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f0731ad5-6b6d-4087-b418-2081a341126a admin admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:02:13.808066 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 1149/2307] 10.222.0.30 () {60 vars in 1143 bytes} [Mon May 23 13:02:13 2022] POST /identity/v3/users => generated 291 bytes in 197 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) May 23 13:02:13.809642 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ff0fc6a-6839-4e7c-b07c-154a57efd626 admin admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:02:13.821733 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ff0fc6a-6839-4e7c-b07c-154a57efd626 admin admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:02:13.827299 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc6f24cd-efdb-4d89-9424-0d05d1b11876 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:02:13.828071 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc6f24cd-efdb-4d89-9424-0d05d1b11876 admin admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:02:13.828434 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc6f24cd-efdb-4d89-9424-0d05d1b11876 admin admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:02:13.828783 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc6f24cd-efdb-4d89-9424-0d05d1b11876 admin admin] PATH_INFO: `/v3/roles` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:02:13.829582 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc6f24cd-efdb-4d89-9424-0d05d1b11876 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:02:13.831871 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc6f24cd-efdb-4d89-9424-0d05d1b11876 admin admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:02:13.842342 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 1159/2308] 10.222.0.30 () {58 vars in 1121 bytes} [Mon May 23 13:02:13 2022] GET /identity/v3/roles => generated 1420 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) May 23 13:02:13.847126 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0731ad5-6b6d-4087-b418-2081a341126a admin admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:02:13.857679 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0731ad5-6b6d-4087-b418-2081a341126a admin admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:02:13.862956 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d42e6120-cc9f-4871-b90a-00912224f68e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:02:13.863623 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d42e6120-cc9f-4871-b90a-00912224f68e admin admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:02:13.863904 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d42e6120-cc9f-4871-b90a-00912224f68e admin admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:02:13.864178 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d42e6120-cc9f-4871-b90a-00912224f68e admin admin] PATH_INFO: `/v3/roles` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:02:13.864906 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d42e6120-cc9f-4871-b90a-00912224f68e admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:02:13.866584 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d42e6120-cc9f-4871-b90a-00912224f68e admin admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:02:13.880318 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 1150/2309] 10.222.0.30 () {58 vars in 1121 bytes} [Mon May 23 13:02:13 2022] GET /identity/v3/roles => generated 1420 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) May 23 13:02:13.884598 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc6f24cd-efdb-4d89-9424-0d05d1b11876 admin admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:02:13.896380 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc6f24cd-efdb-4d89-9424-0d05d1b11876 admin admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:02:13.901659 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b2b457a-7438-4d1f-99bd-0037fefeac90 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:02:13.902705 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b2b457a-7438-4d1f-99bd-0037fefeac90 admin admin] REQUEST_METHOD: `PUT` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:02:13.903166 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b2b457a-7438-4d1f-99bd-0037fefeac90 admin admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:02:13.903474 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b2b457a-7438-4d1f-99bd-0037fefeac90 admin admin] PATH_INFO: `/v3/projects/ccd27933078b44b48323995e98b21318/users/5c6147fed25a4212b6175aacc460943d/roles/80fb22507c9a4ff09d44d8f12f6fff1a` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:02:13.918237 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b2b457a-7438-4d1f-99bd-0037fefeac90 admin admin] RBAC: Authorizing `identity:create_grant(project_id=ccd27933078b44b48323995e98b21318, user_id=5c6147fed25a4212b6175aacc460943d, role_id=80fb22507c9a4ff09d44d8f12f6fff1a)` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:02:13.920921 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b2b457a-7438-4d1f-99bd-0037fefeac90 admin admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:02:14.042481 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 1160/2310] 10.222.0.30 () {60 vars in 1482 bytes} [Mon May 23 13:02:13 2022] PUT /identity/v3/projects/ccd27933078b44b48323995e98b21318/users/5c6147fed25a4212b6175aacc460943d/roles/80fb22507c9a4ff09d44d8f12f6fff1a => generated 0 bytes in 155 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) May 23 13:02:14.044560 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21911d7f-d3a6-41de-9eda-01bbbe0ea27f None None] REQUEST_METHOD: `POST` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:02:14.044985 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21911d7f-d3a6-41de-9eda-01bbbe0ea27f None None] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:02:14.045066 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21911d7f-d3a6-41de-9eda-01bbbe0ea27f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:02:14.076121 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.auth.core [None req-21911d7f-d3a6-41de-9eda-01bbbe0ea27f None None] MFA Rules not processed for user `0f7119f8fa5c4445977671140ded91a4`. Rule list: `[]` (Enabled: `True`). {{(pid=92056) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 13:02:14.077459 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.fernet_utils [None req-21911d7f-d3a6-41de-9eda-01bbbe0ea27f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92056) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:02:14.115199 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 13:02:14.201495 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 1151/2311] 10.222.0.30 () {58 vars in 957 bytes} [Mon May 23 13:02:14 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 159 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) May 23 13:02:14.207879 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b2b457a-7438-4d1f-99bd-0037fefeac90 admin admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:02:14.210458 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.fernet_utils [None req-3b2b457a-7438-4d1f-99bd-0037fefeac90 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92057) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:02:14.291865 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b2b457a-7438-4d1f-99bd-0037fefeac90 admin admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:02:14.295831 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bde81849-1274-44f0-b5d9-cb1bada6f99d service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2eb83ca38e79449095f3a2f947bbb06a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9c389ae65aaa4542bb5a5c4fd0bacf5b', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:02:14.296494 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bde81849-1274-44f0-b5d9-cb1bada6f99d service neutron] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:02:14.296780 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bde81849-1274-44f0-b5d9-cb1bada6f99d service neutron] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:02:14.297061 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bde81849-1274-44f0-b5d9-cb1bada6f99d service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:02:14.307321 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bde81849-1274-44f0-b5d9-cb1bada6f99d service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:02:14.308860 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bde81849-1274-44f0-b5d9-cb1bada6f99d service neutron] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:02:14.319943 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 1161/2312] 10.222.0.30 () {60 vars in 1365 bytes} [Mon May 23 13:02:14 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 113 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) May 23 13:02:15.265551 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21911d7f-d3a6-41de-9eda-01bbbe0ea27f None None] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:02:15.267026 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.fernet_utils [None req-21911d7f-d3a6-41de-9eda-01bbbe0ea27f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92056) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:02:15.358754 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21911d7f-d3a6-41de-9eda-01bbbe0ea27f None None] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:02:15.362844 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3cce4f1-00e4-4fee-8e4c-5aa86637bdd4 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1666097f8ed545c6b0e4809c93e4775a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9c389ae65aaa4542bb5a5c4fd0bacf5b', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:02:15.363610 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3cce4f1-00e4-4fee-8e4c-5aa86637bdd4 service cinder] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:02:15.363859 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3cce4f1-00e4-4fee-8e4c-5aa86637bdd4 service cinder] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:02:15.364563 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3cce4f1-00e4-4fee-8e4c-5aa86637bdd4 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:02:15.366425 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.fernet_utils [None req-e3cce4f1-00e4-4fee-8e4c-5aa86637bdd4 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92056) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:02:15.502342 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3cce4f1-00e4-4fee-8e4c-5aa86637bdd4 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:02:15.503881 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3cce4f1-00e4-4fee-8e4c-5aa86637bdd4 service cinder] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:02:15.519621 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 1152/2313] 10.222.0.30 () {60 vars in 1365 bytes} [Mon May 23 13:02:15 2022] GET /identity/v3/auth/tokens => generated 3307 bytes in 255 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) May 23 13:02:15.855741 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ec7f193-a512-4406-97c8-29a4e6780150 None None] REQUEST_METHOD: `POST` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:02:15.856934 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ec7f193-a512-4406-97c8-29a4e6780150 None None] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:02:15.857018 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ec7f193-a512-4406-97c8-29a4e6780150 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:02:15.886370 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.auth.core [None req-8ec7f193-a512-4406-97c8-29a4e6780150 None None] MFA Rules not processed for user `5c6147fed25a4212b6175aacc460943d`. Rule list: `[]` (Enabled: `True`). {{(pid=92057) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 13:02:15.889000 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.fernet_utils [None req-8ec7f193-a512-4406-97c8-29a4e6780150 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92057) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:02:15.979224 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 1162/2314] 10.222.0.30 () {58 vars in 957 bytes} [Mon May 23 13:02:15 2022] POST /identity/v3/auth/tokens => generated 3248 bytes in 120 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) May 23 13:02:15.991703 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3cce4f1-00e4-4fee-8e4c-5aa86637bdd4 service cinder] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:02:15.993204 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.fernet_utils [None req-e3cce4f1-00e4-4fee-8e4c-5aa86637bdd4 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92056) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:02:16.009067 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3cce4f1-00e4-4fee-8e4c-5aa86637bdd4 service cinder] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:02:16.023323 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48dec4af-fadf-48cf-b7e1-ea5366741cab service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1666097f8ed545c6b0e4809c93e4775a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9c389ae65aaa4542bb5a5c4fd0bacf5b', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:02:16.024048 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48dec4af-fadf-48cf-b7e1-ea5366741cab service cinder] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:02:16.024332 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48dec4af-fadf-48cf-b7e1-ea5366741cab service cinder] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:02:16.024607 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48dec4af-fadf-48cf-b7e1-ea5366741cab service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:02:16.043383 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48dec4af-fadf-48cf-b7e1-ea5366741cab service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:02:16.046158 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48dec4af-fadf-48cf-b7e1-ea5366741cab service cinder] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:02:16.058257 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 1153/2315] 10.222.0.30 () {60 vars in 1365 bytes} [Mon May 23 13:02:15 2022] GET /identity/v3/auth/tokens => generated 3248 bytes in 67 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) May 23 13:02:19.003763 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ce87a1f-fbd9-4264-b760-920f5392897b None None] REQUEST_METHOD: `POST` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:02:19.004246 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ce87a1f-fbd9-4264-b760-920f5392897b None None] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:02:19.004338 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ce87a1f-fbd9-4264-b760-920f5392897b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:02:19.022186 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.auth.core [None req-8ce87a1f-fbd9-4264-b760-920f5392897b None None] MFA Rules not processed for user `5c6147fed25a4212b6175aacc460943d`. Rule list: `[]` (Enabled: `True`). {{(pid=92057) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 13:02:19.023398 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.fernet_utils [None req-8ce87a1f-fbd9-4264-b760-920f5392897b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92057) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:02:19.030686 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 1163/2316] 10.222.0.30 () {58 vars in 957 bytes} [Mon May 23 13:02:19 2022] POST /identity/v3/auth/tokens => generated 3248 bytes in 28 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) May 23 13:02:26.052974 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=70 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 13:02:29.016916 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=78,keystone:INSERT=1 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 13:02:45.420382 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48dec4af-fadf-48cf-b7e1-ea5366741cab service cinder] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:02:45.422121 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.fernet_utils [None req-48dec4af-fadf-48cf-b7e1-ea5366741cab service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92056) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:02:45.439481 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48dec4af-fadf-48cf-b7e1-ea5366741cab service cinder] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:02:45.444951 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29049a4d-024f-43b8-9e66-a53c9d91d561 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:02:45.446005 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29049a4d-024f-43b8-9e66-a53c9d91d561 admin admin] REQUEST_METHOD: `DELETE` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:02:45.446292 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29049a4d-024f-43b8-9e66-a53c9d91d561 admin admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:02:45.446573 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29049a4d-024f-43b8-9e66-a53c9d91d561 admin admin] PATH_INFO: `/v3/users/275524da6b1247638abfdb01f8761d1c` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:02:45.447819 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-29049a4d-024f-43b8-9e66-a53c9d91d561 admin admin] RBAC: Authorizing `identity:delete_user(user_id=275524da6b1247638abfdb01f8761d1c)` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:02:45.449761 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-29049a4d-024f-43b8-9e66-a53c9d91d561 admin admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:02:45.658471 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-29049a4d-024f-43b8-9e66-a53c9d91d561 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '275524da6b1247638abfdb01f8761d1c'} {{(pid=92056) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:02:45.659133 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-29049a4d-024f-43b8-9e66-a53c9d91d561 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '275524da6b1247638abfdb01f8761d1c'} {{(pid=92056) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:02:45.752915 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-29049a4d-024f-43b8-9e66-a53c9d91d561 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '275524da6b1247638abfdb01f8761d1c'} {{(pid=92056) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:02:45.768235 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-29049a4d-024f-43b8-9e66-a53c9d91d561 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '275524da6b1247638abfdb01f8761d1c'} {{(pid=92056) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:02:45.775831 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-29049a4d-024f-43b8-9e66-a53c9d91d561 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '275524da6b1247638abfdb01f8761d1c'} {{(pid=92056) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:02:45.788119 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 1154/2317] 10.222.0.30 () {58 vars in 1223 bytes} [Mon May 23 13:02:45 2022] DELETE /identity/v3/users/275524da6b1247638abfdb01f8761d1c => generated 0 bytes in 365 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) May 23 13:02:45.788522 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30d04bbe-b906-41bd-9931-17d87e51e390 None None] REQUEST_METHOD: `POST` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:02:45.791732 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30d04bbe-b906-41bd-9931-17d87e51e390 None None] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:02:45.791732 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30d04bbe-b906-41bd-9931-17d87e51e390 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:02:45.839494 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.auth.core [None req-30d04bbe-b906-41bd-9931-17d87e51e390 None None] MFA Rules not processed for user `0f7119f8fa5c4445977671140ded91a4`. Rule list: `[]` (Enabled: `True`). {{(pid=92057) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 13:02:45.840871 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.fernet_utils [None req-30d04bbe-b906-41bd-9931-17d87e51e390 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92057) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:02:45.920368 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 1164/2318] 10.222.0.30 () {58 vars in 957 bytes} [Mon May 23 13:02:45 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 132 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) May 23 13:02:45.927963 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29049a4d-024f-43b8-9e66-a53c9d91d561 admin admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:02:45.929521 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.fernet_utils [None req-29049a4d-024f-43b8-9e66-a53c9d91d561 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92056) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:02:46.000765 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29049a4d-024f-43b8-9e66-a53c9d91d561 admin admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:02:46.008411 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea76c548-763f-4839-b4b3-e943f9cc3b8b service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2eb83ca38e79449095f3a2f947bbb06a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9c389ae65aaa4542bb5a5c4fd0bacf5b', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:02:46.009083 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea76c548-763f-4839-b4b3-e943f9cc3b8b service neutron] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:02:46.009367 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea76c548-763f-4839-b4b3-e943f9cc3b8b service neutron] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:02:46.009804 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea76c548-763f-4839-b4b3-e943f9cc3b8b service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:02:46.023861 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea76c548-763f-4839-b4b3-e943f9cc3b8b service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:02:46.025635 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea76c548-763f-4839-b4b3-e943f9cc3b8b service neutron] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:02:46.040629 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 1155/2319] 10.222.0.30 () {60 vars in 1365 bytes} [Mon May 23 13:02:45 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 113 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) May 23 13:02:46.515922 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30d04bbe-b906-41bd-9931-17d87e51e390 None None] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:02:46.517946 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.fernet_utils [None req-30d04bbe-b906-41bd-9931-17d87e51e390 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92057) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:02:46.532046 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30d04bbe-b906-41bd-9931-17d87e51e390 None None] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:02:46.538018 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4bd9d27-a2bf-4df1-ae17-5d624c6cce5a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:02:46.538994 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4bd9d27-a2bf-4df1-ae17-5d624c6cce5a admin admin] REQUEST_METHOD: `DELETE` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:02:46.539352 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4bd9d27-a2bf-4df1-ae17-5d624c6cce5a admin admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:02:46.539699 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4bd9d27-a2bf-4df1-ae17-5d624c6cce5a admin admin] PATH_INFO: `/v3/projects/abdeef898e9b49c18cec057dfba5b3af` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:02:46.541122 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f4bd9d27-a2bf-4df1-ae17-5d624c6cce5a admin admin] RBAC: Authorizing `identity:delete_project(project_id=abdeef898e9b49c18cec057dfba5b3af)` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:02:46.543197 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f4bd9d27-a2bf-4df1-ae17-5d624c6cce5a admin admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:02:46.678243 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [None req-f4bd9d27-a2bf-4df1-ae17-5d624c6cce5a admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'abdeef898e9b49c18cec057dfba5b3af', 'request_id': 'req-f4bd9d27-a2bf-4df1-ae17-5d624c6cce5a'} {{(pid=92057) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:02:46.684784 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [None req-f4bd9d27-a2bf-4df1-ae17-5d624c6cce5a admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'abdeef898e9b49c18cec057dfba5b3af', 'request_id': 'req-f4bd9d27-a2bf-4df1-ae17-5d624c6cce5a'} {{(pid=92057) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:02:46.764181 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [None req-f4bd9d27-a2bf-4df1-ae17-5d624c6cce5a admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'abdeef898e9b49c18cec057dfba5b3af', 'request_id': 'req-f4bd9d27-a2bf-4df1-ae17-5d624c6cce5a'} {{(pid=92057) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:02:46.777519 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [None req-f4bd9d27-a2bf-4df1-ae17-5d624c6cce5a admin admin] The token cache is being invalidate because project abdeef898e9b49c18cec057dfba5b3af was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=92057) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} May 23 13:02:46.777819 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [None req-f4bd9d27-a2bf-4df1-ae17-5d624c6cce5a admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=92057) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:02:46.778571 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [None req-f4bd9d27-a2bf-4df1-ae17-5d624c6cce5a admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=92057) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:02:46.781149 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 1165/2320] 10.222.0.30 () {58 vars in 1232 bytes} [Mon May 23 13:02:46 2022] DELETE /identity/v3/projects/abdeef898e9b49c18cec057dfba5b3af => generated 0 bytes in 266 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) May 23 13:02:46.786111 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c298c0c-1dbc-41b8-9a46-d0339662a3ea None None] REQUEST_METHOD: `POST` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:02:46.786600 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c298c0c-1dbc-41b8-9a46-d0339662a3ea None None] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:02:46.790873 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c298c0c-1dbc-41b8-9a46-d0339662a3ea None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:02:46.805293 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.auth.core [None req-3c298c0c-1dbc-41b8-9a46-d0339662a3ea None None] MFA Rules not processed for user `0f7119f8fa5c4445977671140ded91a4`. Rule list: `[]` (Enabled: `True`). {{(pid=92056) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 13:02:46.806739 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.fernet_utils [None req-3c298c0c-1dbc-41b8-9a46-d0339662a3ea None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92056) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:02:46.891767 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 1156/2321] 10.222.0.30 () {58 vars in 957 bytes} [Mon May 23 13:02:46 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 107 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) May 23 13:02:46.899491 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80339f9d-a86b-4429-b7eb-b6c2a647088f None None] REQUEST_METHOD: `POST` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:02:46.900026 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80339f9d-a86b-4429-b7eb-b6c2a647088f None None] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:02:46.900119 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80339f9d-a86b-4429-b7eb-b6c2a647088f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:02:46.921579 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.auth.core [None req-80339f9d-a86b-4429-b7eb-b6c2a647088f None None] MFA Rules not processed for user `0f7119f8fa5c4445977671140ded91a4`. Rule list: `[]` (Enabled: `True`). {{(pid=92057) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 13:02:46.923318 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.fernet_utils [None req-80339f9d-a86b-4429-b7eb-b6c2a647088f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92057) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:02:46.932363 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 1166/2322] 10.222.0.30 () {58 vars in 957 bytes} [Mon May 23 13:02:46 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 34 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) May 23 13:02:46.937396 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c298c0c-1dbc-41b8-9a46-d0339662a3ea None None] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:02:46.949810 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c298c0c-1dbc-41b8-9a46-d0339662a3ea None None] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:02:46.960208 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c97dfe8-b644-4a3c-95ba-f5e88bb6eac0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:02:46.960899 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c97dfe8-b644-4a3c-95ba-f5e88bb6eac0 admin admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:02:46.961189 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c97dfe8-b644-4a3c-95ba-f5e88bb6eac0 admin admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:02:46.961950 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c97dfe8-b644-4a3c-95ba-f5e88bb6eac0 admin admin] PATH_INFO: `/v3/domains` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:02:46.962965 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c97dfe8-b644-4a3c-95ba-f5e88bb6eac0 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=92056) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} May 23 13:02:46.963440 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c97dfe8-b644-4a3c-95ba-f5e88bb6eac0 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:02:46.967437 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c97dfe8-b644-4a3c-95ba-f5e88bb6eac0 admin admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:02:46.983535 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 1157/2323] 10.222.0.30 () {58 vars in 1152 bytes} [Mon May 23 13:02:46 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 47 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) May 23 13:02:46.988929 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80339f9d-a86b-4429-b7eb-b6c2a647088f None None] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:02:47.014304 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80339f9d-a86b-4429-b7eb-b6c2a647088f None None] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:02:47.020532 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09ff90f2-d1ec-4f0e-82c6-05af066d2b9f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:02:47.021760 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09ff90f2-d1ec-4f0e-82c6-05af066d2b9f admin admin] REQUEST_METHOD: `POST` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:02:47.023535 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09ff90f2-d1ec-4f0e-82c6-05af066d2b9f admin admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:02:47.024477 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09ff90f2-d1ec-4f0e-82c6-05af066d2b9f admin admin] PATH_INFO: `/v3/projects` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:02:47.026323 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09ff90f2-d1ec-4f0e-82c6-05af066d2b9f admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:02:47.028835 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09ff90f2-d1ec-4f0e-82c6-05af066d2b9f admin admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:02:47.038020 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c97dfe8-b644-4a3c-95ba-f5e88bb6eac0 admin admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:02:47.039785 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.fernet_utils [None req-2c97dfe8-b644-4a3c-95ba-f5e88bb6eac0 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92056) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:02:47.059112 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 13:02:47.065542 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c97dfe8-b644-4a3c-95ba-f5e88bb6eac0 admin admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:02:47.070237 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8bd91125-d2ab-4e7d-9112-bdf6130e2bf3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:02:47.071191 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8bd91125-d2ab-4e7d-9112-bdf6130e2bf3 admin admin] REQUEST_METHOD: `DELETE` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:02:47.071484 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8bd91125-d2ab-4e7d-9112-bdf6130e2bf3 admin admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:02:47.071741 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8bd91125-d2ab-4e7d-9112-bdf6130e2bf3 admin admin] PATH_INFO: `/v3/users/ff1f2649136b42a48f953553c309cd42` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:02:47.072926 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8bd91125-d2ab-4e7d-9112-bdf6130e2bf3 admin admin] RBAC: Authorizing `identity:delete_user(user_id=ff1f2649136b42a48f953553c309cd42)` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:02:47.074884 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8bd91125-d2ab-4e7d-9112-bdf6130e2bf3 admin admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:02:47.143131 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 1167/2324] 10.222.0.30 () {60 vars in 1152 bytes} [Mon May 23 13:02:46 2022] POST /identity/v3/projects => generated 385 bytes in 155 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) May 23 13:02:47.147146 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09ff90f2-d1ec-4f0e-82c6-05af066d2b9f admin admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:02:47.205929 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=91,keystone:DELETE=7,keystone:INSERT=2 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 13:02:47.210973 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09ff90f2-d1ec-4f0e-82c6-05af066d2b9f admin admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:02:47.215704 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-255dc32c-7fbd-45b6-8392-b93ec79265ec admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:02:47.217229 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-255dc32c-7fbd-45b6-8392-b93ec79265ec admin admin] REQUEST_METHOD: `POST` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:02:47.217513 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-255dc32c-7fbd-45b6-8392-b93ec79265ec admin admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:02:47.217802 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-255dc32c-7fbd-45b6-8392-b93ec79265ec admin admin] PATH_INFO: `/v3/users` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:02:47.218542 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-255dc32c-7fbd-45b6-8392-b93ec79265ec admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:02:47.220610 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-255dc32c-7fbd-45b6-8392-b93ec79265ec admin admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:02:47.286437 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-8bd91125-d2ab-4e7d-9112-bdf6130e2bf3 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'ff1f2649136b42a48f953553c309cd42'} {{(pid=92056) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:02:47.287012 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-8bd91125-d2ab-4e7d-9112-bdf6130e2bf3 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'ff1f2649136b42a48f953553c309cd42'} {{(pid=92056) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:02:47.306689 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 1168/2325] 10.222.0.30 () {60 vars in 1143 bytes} [Mon May 23 13:02:47 2022] POST /identity/v3/users => generated 293 bytes in 160 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) May 23 13:02:47.310990 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-255dc32c-7fbd-45b6-8392-b93ec79265ec admin admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:02:47.312669 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.fernet_utils [None req-255dc32c-7fbd-45b6-8392-b93ec79265ec admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92057) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:02:47.324843 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-255dc32c-7fbd-45b6-8392-b93ec79265ec admin admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:02:47.330290 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6958437-1bd4-465b-9682-45224d156d36 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:02:47.330928 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6958437-1bd4-465b-9682-45224d156d36 admin admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:02:47.331340 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6958437-1bd4-465b-9682-45224d156d36 admin admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:02:47.331621 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6958437-1bd4-465b-9682-45224d156d36 admin admin] PATH_INFO: `/v3/roles` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:02:47.332327 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6958437-1bd4-465b-9682-45224d156d36 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:02:47.334103 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6958437-1bd4-465b-9682-45224d156d36 admin admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:02:47.345319 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 1169/2326] 10.222.0.30 () {58 vars in 1121 bytes} [Mon May 23 13:02:47 2022] GET /identity/v3/roles => generated 1420 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) May 23 13:02:47.353243 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6958437-1bd4-465b-9682-45224d156d36 admin admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:02:47.362856 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6958437-1bd4-465b-9682-45224d156d36 admin admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:02:47.367370 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48481c91-6b68-471b-82ac-3f3dd66aa84d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:02:47.367973 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48481c91-6b68-471b-82ac-3f3dd66aa84d admin admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:02:47.368329 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48481c91-6b68-471b-82ac-3f3dd66aa84d admin admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:02:47.368604 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48481c91-6b68-471b-82ac-3f3dd66aa84d admin admin] PATH_INFO: `/v3/roles` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:02:47.369292 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48481c91-6b68-471b-82ac-3f3dd66aa84d admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:02:47.370871 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48481c91-6b68-471b-82ac-3f3dd66aa84d admin admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:02:47.380846 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 1170/2327] 10.222.0.30 () {58 vars in 1121 bytes} [Mon May 23 13:02:47 2022] GET /identity/v3/roles => generated 1420 bytes in 28 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) May 23 13:02:47.384640 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48481c91-6b68-471b-82ac-3f3dd66aa84d admin admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:02:47.394935 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48481c91-6b68-471b-82ac-3f3dd66aa84d admin admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:02:47.398290 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-8bd91125-d2ab-4e7d-9112-bdf6130e2bf3 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'ff1f2649136b42a48f953553c309cd42'} {{(pid=92056) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:02:47.400137 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0d85157-d68b-4e1a-85f8-c13f454e028d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:02:47.401126 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0d85157-d68b-4e1a-85f8-c13f454e028d admin admin] REQUEST_METHOD: `PUT` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:02:47.401478 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0d85157-d68b-4e1a-85f8-c13f454e028d admin admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:02:47.401821 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0d85157-d68b-4e1a-85f8-c13f454e028d admin admin] PATH_INFO: `/v3/projects/25f876268bb24dbe8ee677203da9eaef/users/63e9547e372d4b93af6856c9753ebf44/roles/80fb22507c9a4ff09d44d8f12f6fff1a` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:02:47.412404 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-8bd91125-d2ab-4e7d-9112-bdf6130e2bf3 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'ff1f2649136b42a48f953553c309cd42'} {{(pid=92056) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:02:47.417641 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0d85157-d68b-4e1a-85f8-c13f454e028d admin admin] RBAC: Authorizing `identity:create_grant(project_id=25f876268bb24dbe8ee677203da9eaef, user_id=63e9547e372d4b93af6856c9753ebf44, role_id=80fb22507c9a4ff09d44d8f12f6fff1a)` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:02:47.419178 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-8bd91125-d2ab-4e7d-9112-bdf6130e2bf3 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'ff1f2649136b42a48f953553c309cd42'} {{(pid=92056) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:02:47.420500 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0d85157-d68b-4e1a-85f8-c13f454e028d admin admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:02:47.425602 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 1158/2328] 10.222.0.30 () {58 vars in 1223 bytes} [Mon May 23 13:02:47 2022] DELETE /identity/v3/users/ff1f2649136b42a48f953553c309cd42 => generated 0 bytes in 388 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) May 23 13:02:47.484421 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 1171/2329] 10.222.0.30 () {60 vars in 1482 bytes} [Mon May 23 13:02:47 2022] PUT /identity/v3/projects/25f876268bb24dbe8ee677203da9eaef/users/63e9547e372d4b93af6856c9753ebf44/roles/80fb22507c9a4ff09d44d8f12f6fff1a => generated 0 bytes in 100 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) May 23 13:02:47.494592 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75485a98-6c8c-4b5d-8e46-1cb09e410033 None None] REQUEST_METHOD: `POST` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:02:47.494881 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75485a98-6c8c-4b5d-8e46-1cb09e410033 None None] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:02:47.495157 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75485a98-6c8c-4b5d-8e46-1cb09e410033 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:02:47.522668 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.auth.core [None req-75485a98-6c8c-4b5d-8e46-1cb09e410033 None None] MFA Rules not processed for user `63e9547e372d4b93af6856c9753ebf44`. Rule list: `[]` (Enabled: `True`). {{(pid=92056) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 13:02:47.523868 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.fernet_utils [None req-75485a98-6c8c-4b5d-8e46-1cb09e410033 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92056) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:02:47.592136 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 1159/2330] 10.222.0.30 () {58 vars in 957 bytes} [Mon May 23 13:02:47 2022] POST /identity/v3/auth/tokens => generated 3252 bytes in 99 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) May 23 13:02:47.603354 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0d85157-d68b-4e1a-85f8-c13f454e028d admin admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:02:47.605111 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.fernet_utils [None req-b0d85157-d68b-4e1a-85f8-c13f454e028d admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92057) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:02:47.662623 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0d85157-d68b-4e1a-85f8-c13f454e028d admin admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:02:47.666329 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3914e0c-3671-41cf-8d83-148be17d4fb9 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1666097f8ed545c6b0e4809c93e4775a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9c389ae65aaa4542bb5a5c4fd0bacf5b', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:02:47.671038 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3914e0c-3671-41cf-8d83-148be17d4fb9 service cinder] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:02:47.671038 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3914e0c-3671-41cf-8d83-148be17d4fb9 service cinder] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:02:47.671038 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3914e0c-3671-41cf-8d83-148be17d4fb9 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:02:47.675860 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3914e0c-3671-41cf-8d83-148be17d4fb9 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:02:47.677354 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3914e0c-3671-41cf-8d83-148be17d4fb9 service cinder] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:02:47.687429 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 1172/2331] 10.222.0.30 () {60 vars in 1365 bytes} [Mon May 23 13:02:47 2022] GET /identity/v3/auth/tokens => generated 3252 bytes in 85 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) May 23 13:02:47.698022 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75485a98-6c8c-4b5d-8e46-1cb09e410033 None None] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:02:47.698783 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.fernet_utils [None req-75485a98-6c8c-4b5d-8e46-1cb09e410033 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92056) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:02:47.779579 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75485a98-6c8c-4b5d-8e46-1cb09e410033 None None] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:02:47.786015 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7a70077-a2ea-459d-89e6-5d2ee9c059a4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:02:47.790586 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7a70077-a2ea-459d-89e6-5d2ee9c059a4 admin admin] REQUEST_METHOD: `DELETE` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:02:47.790586 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7a70077-a2ea-459d-89e6-5d2ee9c059a4 admin admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:02:47.790586 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7a70077-a2ea-459d-89e6-5d2ee9c059a4 admin admin] PATH_INFO: `/v3/projects/ed17195d8a3d4c5292bfd6099a9df47d` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:02:47.790586 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7a70077-a2ea-459d-89e6-5d2ee9c059a4 admin admin] RBAC: Authorizing `identity:delete_project(project_id=ed17195d8a3d4c5292bfd6099a9df47d)` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:02:47.791567 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7a70077-a2ea-459d-89e6-5d2ee9c059a4 admin admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:02:47.800687 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 13:02:47.925478 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-e7a70077-a2ea-459d-89e6-5d2ee9c059a4 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'ed17195d8a3d4c5292bfd6099a9df47d', 'request_id': 'req-e7a70077-a2ea-459d-89e6-5d2ee9c059a4'} {{(pid=92056) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:02:47.992401 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-e7a70077-a2ea-459d-89e6-5d2ee9c059a4 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'ed17195d8a3d4c5292bfd6099a9df47d', 'request_id': 'req-e7a70077-a2ea-459d-89e6-5d2ee9c059a4'} {{(pid=92056) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:02:48.001659 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-e7a70077-a2ea-459d-89e6-5d2ee9c059a4 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'ed17195d8a3d4c5292bfd6099a9df47d', 'request_id': 'req-e7a70077-a2ea-459d-89e6-5d2ee9c059a4'} {{(pid=92056) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:02:48.010405 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-e7a70077-a2ea-459d-89e6-5d2ee9c059a4 admin admin] The token cache is being invalidate because project ed17195d8a3d4c5292bfd6099a9df47d was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=92056) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} May 23 13:02:48.010693 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-e7a70077-a2ea-459d-89e6-5d2ee9c059a4 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=92056) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:02:48.011350 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-e7a70077-a2ea-459d-89e6-5d2ee9c059a4 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=92056) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:02:48.013007 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 1160/2332] 10.222.0.30 () {58 vars in 1232 bytes} [Mon May 23 13:02:47 2022] DELETE /identity/v3/projects/ed17195d8a3d4c5292bfd6099a9df47d => generated 0 bytes in 316 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) May 23 13:02:48.017134 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3914e0c-3671-41cf-8d83-148be17d4fb9 service cinder] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:02:48.028207 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3914e0c-3671-41cf-8d83-148be17d4fb9 service cinder] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:02:48.088008 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1ad63e6-0a3e-4844-b6b4-f7109f834783 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:02:48.088879 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1ad63e6-0a3e-4844-b6b4-f7109f834783 admin admin] REQUEST_METHOD: `DELETE` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:02:48.089166 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1ad63e6-0a3e-4844-b6b4-f7109f834783 admin admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:02:48.089445 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1ad63e6-0a3e-4844-b6b4-f7109f834783 admin admin] PATH_INFO: `/v3/users/aa81faad75874e9e8139724b9f4ab6d0` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:02:48.090696 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1ad63e6-0a3e-4844-b6b4-f7109f834783 admin admin] RBAC: Authorizing `identity:delete_user(user_id=aa81faad75874e9e8139724b9f4ab6d0)` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:02:48.092795 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1ad63e6-0a3e-4844-b6b4-f7109f834783 admin admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:02:48.311878 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=90,keystone:INSERT=4,keystone:DELETE=6 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 13:02:48.314921 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [None req-c1ad63e6-0a3e-4844-b6b4-f7109f834783 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'aa81faad75874e9e8139724b9f4ab6d0'} {{(pid=92057) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:02:48.384181 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [None req-c1ad63e6-0a3e-4844-b6b4-f7109f834783 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'aa81faad75874e9e8139724b9f4ab6d0'} {{(pid=92057) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:02:48.393456 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [None req-c1ad63e6-0a3e-4844-b6b4-f7109f834783 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'aa81faad75874e9e8139724b9f4ab6d0'} {{(pid=92057) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:02:48.394159 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [None req-c1ad63e6-0a3e-4844-b6b4-f7109f834783 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'aa81faad75874e9e8139724b9f4ab6d0'} {{(pid=92057) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:02:48.406363 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [None req-c1ad63e6-0a3e-4844-b6b4-f7109f834783 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'aa81faad75874e9e8139724b9f4ab6d0'} {{(pid=92057) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:02:48.412554 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 1173/2333] 10.222.0.30 () {58 vars in 1223 bytes} [Mon May 23 13:02:48 2022] DELETE /identity/v3/users/aa81faad75874e9e8139724b9f4ab6d0 => generated 0 bytes in 396 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) May 23 13:02:48.679839 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7a70077-a2ea-459d-89e6-5d2ee9c059a4 admin admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:02:48.681529 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.fernet_utils [None req-e7a70077-a2ea-459d-89e6-5d2ee9c059a4 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92056) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:02:48.695407 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7a70077-a2ea-459d-89e6-5d2ee9c059a4 admin admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:02:48.754256 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-045f7531-0140-4c63-bf2b-644fede4f45d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:02:48.755116 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-045f7531-0140-4c63-bf2b-644fede4f45d admin admin] REQUEST_METHOD: `DELETE` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:02:48.755401 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-045f7531-0140-4c63-bf2b-644fede4f45d admin admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:02:48.755677 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-045f7531-0140-4c63-bf2b-644fede4f45d admin admin] PATH_INFO: `/v3/projects/6f0b7e44a32f4307a7893046f807fbfa` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:02:48.756854 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-045f7531-0140-4c63-bf2b-644fede4f45d admin admin] RBAC: Authorizing `identity:delete_project(project_id=6f0b7e44a32f4307a7893046f807fbfa)` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:02:48.758806 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-045f7531-0140-4c63-bf2b-644fede4f45d admin admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:02:48.884130 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-045f7531-0140-4c63-bf2b-644fede4f45d admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '6f0b7e44a32f4307a7893046f807fbfa', 'request_id': 'req-045f7531-0140-4c63-bf2b-644fede4f45d'} {{(pid=92056) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:02:48.949473 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-045f7531-0140-4c63-bf2b-644fede4f45d admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '6f0b7e44a32f4307a7893046f807fbfa', 'request_id': 'req-045f7531-0140-4c63-bf2b-644fede4f45d'} {{(pid=92056) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:02:48.956527 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-045f7531-0140-4c63-bf2b-644fede4f45d admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '6f0b7e44a32f4307a7893046f807fbfa', 'request_id': 'req-045f7531-0140-4c63-bf2b-644fede4f45d'} {{(pid=92056) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:02:48.967950 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-045f7531-0140-4c63-bf2b-644fede4f45d admin admin] The token cache is being invalidate because project 6f0b7e44a32f4307a7893046f807fbfa was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=92056) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} May 23 13:02:48.968235 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-045f7531-0140-4c63-bf2b-644fede4f45d admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=92056) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:02:48.968879 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-045f7531-0140-4c63-bf2b-644fede4f45d admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=92056) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:02:48.970616 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 1161/2334] 10.222.0.30 () {58 vars in 1232 bytes} [Mon May 23 13:02:48 2022] DELETE /identity/v3/projects/6f0b7e44a32f4307a7893046f807fbfa => generated 0 bytes in 291 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) May 23 13:02:48.975436 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87496abb-b598-4d45-8f24-4ed77143ee09 None None] REQUEST_METHOD: `POST` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:02:48.975726 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87496abb-b598-4d45-8f24-4ed77143ee09 None None] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:02:48.976003 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87496abb-b598-4d45-8f24-4ed77143ee09 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:02:48.995256 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.auth.core [None req-87496abb-b598-4d45-8f24-4ed77143ee09 None None] MFA Rules not processed for user `0f7119f8fa5c4445977671140ded91a4`. Rule list: `[]` (Enabled: `True`). {{(pid=92057) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 13:02:48.996441 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.fernet_utils [None req-87496abb-b598-4d45-8f24-4ed77143ee09 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92057) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:02:49.079509 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 1174/2335] 10.222.0.30 () {58 vars in 957 bytes} [Mon May 23 13:02:48 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 105 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) May 23 13:02:49.087596 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff2c5fae-d1b6-4a91-b4e2-cae64f0c26eb None None] REQUEST_METHOD: `POST` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:02:49.087596 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff2c5fae-d1b6-4a91-b4e2-cae64f0c26eb None None] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:02:49.087993 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff2c5fae-d1b6-4a91-b4e2-cae64f0c26eb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:02:49.108500 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.auth.core [None req-ff2c5fae-d1b6-4a91-b4e2-cae64f0c26eb None None] MFA Rules not processed for user `0f7119f8fa5c4445977671140ded91a4`. Rule list: `[]` (Enabled: `True`). {{(pid=92056) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 13:02:49.109741 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.fernet_utils [None req-ff2c5fae-d1b6-4a91-b4e2-cae64f0c26eb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92056) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:02:49.117956 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 1162/2336] 10.222.0.30 () {58 vars in 957 bytes} [Mon May 23 13:02:49 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) May 23 13:02:49.122237 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87496abb-b598-4d45-8f24-4ed77143ee09 None None] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:02:49.132381 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87496abb-b598-4d45-8f24-4ed77143ee09 None None] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:02:49.137125 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc8a63f0-05f9-4748-885c-c750064e02af admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:02:49.137788 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc8a63f0-05f9-4748-885c-c750064e02af admin admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:02:49.138995 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc8a63f0-05f9-4748-885c-c750064e02af admin admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:02:49.139752 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc8a63f0-05f9-4748-885c-c750064e02af admin admin] PATH_INFO: `/v3/domains` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:02:49.140614 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc8a63f0-05f9-4748-885c-c750064e02af admin admin] RBAC: Adding query filter params (name=Default) {{(pid=92057) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} May 23 13:02:49.140933 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc8a63f0-05f9-4748-885c-c750064e02af admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:02:49.142895 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc8a63f0-05f9-4748-885c-c750064e02af admin admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:02:49.155795 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 1175/2337] 10.222.0.30 () {58 vars in 1152 bytes} [Mon May 23 13:02:49 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) May 23 13:02:49.160097 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff2c5fae-d1b6-4a91-b4e2-cae64f0c26eb None None] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:02:49.169227 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff2c5fae-d1b6-4a91-b4e2-cae64f0c26eb None None] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:02:49.174630 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-caf85078-65b5-4986-90f7-ddba1c87e2da admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:02:49.175294 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-caf85078-65b5-4986-90f7-ddba1c87e2da admin admin] REQUEST_METHOD: `POST` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:02:49.175663 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-caf85078-65b5-4986-90f7-ddba1c87e2da admin admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:02:49.175838 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-caf85078-65b5-4986-90f7-ddba1c87e2da admin admin] PATH_INFO: `/v3/projects` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:02:49.176541 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-caf85078-65b5-4986-90f7-ddba1c87e2da admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:02:49.178513 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-caf85078-65b5-4986-90f7-ddba1c87e2da admin admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:02:49.193973 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=83,keystone:DELETE=14,keystone:INSERT=3 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 13:02:49.283653 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 1163/2338] 10.222.0.30 () {60 vars in 1152 bytes} [Mon May 23 13:02:49 2022] POST /identity/v3/projects => generated 375 bytes in 124 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) May 23 13:02:49.287873 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc8a63f0-05f9-4748-885c-c750064e02af admin admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:02:49.298366 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc8a63f0-05f9-4748-885c-c750064e02af admin admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:02:49.355994 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66263b0a-6c21-4b2e-9219-545959d4be43 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:02:49.356748 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66263b0a-6c21-4b2e-9219-545959d4be43 admin admin] REQUEST_METHOD: `POST` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:02:49.357033 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66263b0a-6c21-4b2e-9219-545959d4be43 admin admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:02:49.357306 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66263b0a-6c21-4b2e-9219-545959d4be43 admin admin] PATH_INFO: `/v3/users` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:02:49.358033 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66263b0a-6c21-4b2e-9219-545959d4be43 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:02:49.359955 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66263b0a-6c21-4b2e-9219-545959d4be43 admin admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:02:49.461331 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 1176/2339] 10.222.0.30 () {60 vars in 1143 bytes} [Mon May 23 13:02:49 2022] POST /identity/v3/users => generated 288 bytes in 174 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) May 23 13:02:49.467709 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-caf85078-65b5-4986-90f7-ddba1c87e2da admin admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:02:49.477747 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-caf85078-65b5-4986-90f7-ddba1c87e2da admin admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:02:49.482847 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64726155-9d67-48a6-b6fc-65327e667022 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:02:49.483492 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64726155-9d67-48a6-b6fc-65327e667022 admin admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:02:49.483769 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64726155-9d67-48a6-b6fc-65327e667022 admin admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:02:49.484042 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64726155-9d67-48a6-b6fc-65327e667022 admin admin] PATH_INFO: `/v3/roles` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:02:49.484799 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-64726155-9d67-48a6-b6fc-65327e667022 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:02:49.486432 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-64726155-9d67-48a6-b6fc-65327e667022 admin admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:02:49.498365 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 1164/2340] 10.222.0.30 () {58 vars in 1121 bytes} [Mon May 23 13:02:49 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) May 23 13:02:49.502327 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66263b0a-6c21-4b2e-9219-545959d4be43 admin admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:02:49.512473 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66263b0a-6c21-4b2e-9219-545959d4be43 admin admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:02:49.517649 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04577c78-0a30-407b-8121-9d794325d536 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:02:49.518357 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04577c78-0a30-407b-8121-9d794325d536 admin admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:02:49.518742 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04577c78-0a30-407b-8121-9d794325d536 admin admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:02:49.519030 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04577c78-0a30-407b-8121-9d794325d536 admin admin] PATH_INFO: `/v3/roles` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:02:49.519764 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04577c78-0a30-407b-8121-9d794325d536 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:02:49.521375 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04577c78-0a30-407b-8121-9d794325d536 admin admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:02:49.531569 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 1177/2341] 10.222.0.30 () {58 vars in 1121 bytes} [Mon May 23 13:02:49 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) May 23 13:02:49.535379 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64726155-9d67-48a6-b6fc-65327e667022 admin admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:02:49.545575 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64726155-9d67-48a6-b6fc-65327e667022 admin admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:02:49.550295 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46858833-9610-4512-a343-63a73f7a94b6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:02:49.551154 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46858833-9610-4512-a343-63a73f7a94b6 admin admin] REQUEST_METHOD: `PUT` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:02:49.551437 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46858833-9610-4512-a343-63a73f7a94b6 admin admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:02:49.551711 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46858833-9610-4512-a343-63a73f7a94b6 admin admin] PATH_INFO: `/v3/projects/590ed5c70dca4aaf8dbef73ce1f00954/users/6b8c4a63ea3746e28dccfff48bc58e96/roles/80fb22507c9a4ff09d44d8f12f6fff1a` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:02:49.567146 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46858833-9610-4512-a343-63a73f7a94b6 admin admin] RBAC: Authorizing `identity:create_grant(project_id=590ed5c70dca4aaf8dbef73ce1f00954, user_id=6b8c4a63ea3746e28dccfff48bc58e96, role_id=80fb22507c9a4ff09d44d8f12f6fff1a)` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:02:49.569810 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46858833-9610-4512-a343-63a73f7a94b6 admin admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:02:49.655241 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 1165/2342] 10.222.0.30 () {60 vars in 1482 bytes} [Mon May 23 13:02:49 2022] PUT /identity/v3/projects/590ed5c70dca4aaf8dbef73ce1f00954/users/6b8c4a63ea3746e28dccfff48bc58e96/roles/80fb22507c9a4ff09d44d8f12f6fff1a => generated 0 bytes in 120 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) May 23 13:02:49.666367 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0b00e28-2d18-4a17-9b13-afb064504503 None None] REQUEST_METHOD: `POST` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:02:49.666660 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0b00e28-2d18-4a17-9b13-afb064504503 None None] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:02:49.666943 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0b00e28-2d18-4a17-9b13-afb064504503 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:02:49.694841 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.auth.core [None req-d0b00e28-2d18-4a17-9b13-afb064504503 None None] MFA Rules not processed for user `6b8c4a63ea3746e28dccfff48bc58e96`. Rule list: `[]` (Enabled: `True`). {{(pid=92057) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 13:02:49.696029 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.fernet_utils [None req-d0b00e28-2d18-4a17-9b13-afb064504503 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92057) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:02:49.708408 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:DELETE=4,keystone:SELECT=92,keystone:INSERT=4 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 13:02:49.760176 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 1178/2343] 10.222.0.30 () {58 vars in 957 bytes} [Mon May 23 13:02:49 2022] POST /identity/v3/auth/tokens => generated 3242 bytes in 95 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) May 23 13:02:49.771457 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46858833-9610-4512-a343-63a73f7a94b6 admin admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:02:49.776421 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.fernet_utils [None req-46858833-9610-4512-a343-63a73f7a94b6 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92056) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:02:49.832203 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46858833-9610-4512-a343-63a73f7a94b6 admin admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:02:49.835830 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-648f1431-8536-42d1-b266-19bab2417cc8 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1666097f8ed545c6b0e4809c93e4775a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9c389ae65aaa4542bb5a5c4fd0bacf5b', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:02:49.836469 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-648f1431-8536-42d1-b266-19bab2417cc8 service cinder] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:02:49.836751 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-648f1431-8536-42d1-b266-19bab2417cc8 service cinder] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:02:49.837022 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-648f1431-8536-42d1-b266-19bab2417cc8 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:02:49.845772 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-648f1431-8536-42d1-b266-19bab2417cc8 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:02:49.847228 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-648f1431-8536-42d1-b266-19bab2417cc8 service cinder] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:02:49.855961 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 1166/2344] 10.222.0.30 () {60 vars in 1365 bytes} [Mon May 23 13:02:49 2022] GET /identity/v3/auth/tokens => generated 3242 bytes in 85 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) May 23 13:02:53.826405 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e47267c-437d-4cc5-94c7-531ecd62e9ea None None] REQUEST_METHOD: `POST` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:02:53.827300 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e47267c-437d-4cc5-94c7-531ecd62e9ea None None] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:02:53.827300 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e47267c-437d-4cc5-94c7-531ecd62e9ea None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:02:53.874249 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.auth.core [None req-2e47267c-437d-4cc5-94c7-531ecd62e9ea None None] MFA Rules not processed for user `6d21a56126fd4ccd837c33ccc08ab609`. Rule list: `[]` (Enabled: `True`). {{(pid=92057) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 13:02:53.875552 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.fernet_utils [None req-2e47267c-437d-4cc5-94c7-531ecd62e9ea None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92057) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:02:53.948228 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 1179/2345] 10.222.0.30 () {58 vars in 1009 bytes} [Mon May 23 13:02:53 2022] POST /identity/v3/auth/tokens => generated 3185 bytes in 124 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) May 23 13:02:54.007425 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-648f1431-8536-42d1-b266-19bab2417cc8 service cinder] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:02:54.009075 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.fernet_utils [None req-648f1431-8536-42d1-b266-19bab2417cc8 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92056) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:02:54.121326 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-648f1431-8536-42d1-b266-19bab2417cc8 service cinder] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:02:54.127589 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-117244b5-6a2a-49a3-a783-cb8c1bcc2ee3 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '138d91edb99f450e846765f9eef00d62', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9c389ae65aaa4542bb5a5c4fd0bacf5b', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'service', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:02:54.128279 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-117244b5-6a2a-49a3-a783-cb8c1bcc2ee3 service swift] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:02:54.128584 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-117244b5-6a2a-49a3-a783-cb8c1bcc2ee3 service swift] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:02:54.128844 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-117244b5-6a2a-49a3-a783-cb8c1bcc2ee3 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:02:54.137810 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-117244b5-6a2a-49a3-a783-cb8c1bcc2ee3 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:02:54.141000 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-117244b5-6a2a-49a3-a783-cb8c1bcc2ee3 service swift] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:02:54.149926 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 1167/2346] 10.222.0.30 () {60 vars in 1384 bytes} [Mon May 23 13:02:54 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 143 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) May 23 13:02:55.118280 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e47267c-437d-4cc5-94c7-531ecd62e9ea None None] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:02:55.120392 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.fernet_utils [None req-2e47267c-437d-4cc5-94c7-531ecd62e9ea None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92057) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:02:55.217356 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e47267c-437d-4cc5-94c7-531ecd62e9ea None None] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:02:55.231447 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5086c4c9-5ce3-495a-b417-cc56337f659c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:02:55.234078 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5086c4c9-5ce3-495a-b417-cc56337f659c admin admin] REQUEST_METHOD: `DELETE` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:02:55.234078 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5086c4c9-5ce3-495a-b417-cc56337f659c admin admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:02:55.234078 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5086c4c9-5ce3-495a-b417-cc56337f659c admin admin] PATH_INFO: `/v3/users/63e9547e372d4b93af6856c9753ebf44` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:02:55.235490 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5086c4c9-5ce3-495a-b417-cc56337f659c admin admin] RBAC: Authorizing `identity:delete_user(user_id=63e9547e372d4b93af6856c9753ebf44)` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:02:55.238095 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5086c4c9-5ce3-495a-b417-cc56337f659c admin admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:02:55.316636 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=96,keystone:DELETE=4 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 13:02:55.525570 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [None req-5086c4c9-5ce3-495a-b417-cc56337f659c admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '63e9547e372d4b93af6856c9753ebf44'} {{(pid=92057) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:02:55.620445 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [None req-5086c4c9-5ce3-495a-b417-cc56337f659c admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '63e9547e372d4b93af6856c9753ebf44'} {{(pid=92057) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:02:55.636202 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [None req-5086c4c9-5ce3-495a-b417-cc56337f659c admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '63e9547e372d4b93af6856c9753ebf44'} {{(pid=92057) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:02:55.637750 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [None req-5086c4c9-5ce3-495a-b417-cc56337f659c admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '63e9547e372d4b93af6856c9753ebf44'} {{(pid=92057) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:02:55.660508 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [None req-5086c4c9-5ce3-495a-b417-cc56337f659c admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '63e9547e372d4b93af6856c9753ebf44'} {{(pid=92057) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:02:55.668579 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 1180/2347] 10.222.0.30 () {58 vars in 1223 bytes} [Mon May 23 13:02:55 2022] DELETE /identity/v3/users/63e9547e372d4b93af6856c9753ebf44 => generated 0 bytes in 550 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) May 23 13:02:55.672707 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6cdb1c7-53bc-4544-8446-d9a05e9c5784 None None] REQUEST_METHOD: `POST` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:02:55.673268 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6cdb1c7-53bc-4544-8446-d9a05e9c5784 None None] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:02:55.673463 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6cdb1c7-53bc-4544-8446-d9a05e9c5784 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:02:55.694877 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.auth.core [None req-a6cdb1c7-53bc-4544-8446-d9a05e9c5784 None None] MFA Rules not processed for user `0f7119f8fa5c4445977671140ded91a4`. Rule list: `[]` (Enabled: `True`). {{(pid=92056) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 13:02:55.696426 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.fernet_utils [None req-a6cdb1c7-53bc-4544-8446-d9a05e9c5784 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92056) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:02:55.732940 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 13:02:55.771296 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 1168/2348] 10.222.0.30 () {58 vars in 957 bytes} [Mon May 23 13:02:55 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 100 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) May 23 13:02:55.781983 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5086c4c9-5ce3-495a-b417-cc56337f659c admin admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:02:55.783367 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.fernet_utils [None req-5086c4c9-5ce3-495a-b417-cc56337f659c admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92057) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:02:55.841246 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5086c4c9-5ce3-495a-b417-cc56337f659c admin admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:02:55.845311 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c542376-d997-4f8c-ad31-78fa76f8f723 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2eb83ca38e79449095f3a2f947bbb06a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9c389ae65aaa4542bb5a5c4fd0bacf5b', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:02:55.848599 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c542376-d997-4f8c-ad31-78fa76f8f723 service neutron] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:02:55.848599 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c542376-d997-4f8c-ad31-78fa76f8f723 service neutron] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:02:55.848599 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c542376-d997-4f8c-ad31-78fa76f8f723 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:02:55.857204 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c542376-d997-4f8c-ad31-78fa76f8f723 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:02:55.858981 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c542376-d997-4f8c-ad31-78fa76f8f723 service neutron] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:02:55.873739 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 1181/2349] 10.222.0.30 () {60 vars in 1365 bytes} [Mon May 23 13:02:55 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 92 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) May 23 13:02:56.503672 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6cdb1c7-53bc-4544-8446-d9a05e9c5784 None None] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:02:56.505337 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.fernet_utils [None req-a6cdb1c7-53bc-4544-8446-d9a05e9c5784 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92056) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:02:56.539176 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6cdb1c7-53bc-4544-8446-d9a05e9c5784 None None] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:02:56.548770 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-258a5c24-3e64-46d5-b016-95708abe270c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:02:56.550062 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-258a5c24-3e64-46d5-b016-95708abe270c admin admin] REQUEST_METHOD: `DELETE` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:02:56.551152 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-258a5c24-3e64-46d5-b016-95708abe270c admin admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:02:56.551672 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-258a5c24-3e64-46d5-b016-95708abe270c admin admin] PATH_INFO: `/v3/projects/25f876268bb24dbe8ee677203da9eaef` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:02:56.553208 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-258a5c24-3e64-46d5-b016-95708abe270c admin admin] RBAC: Authorizing `identity:delete_project(project_id=25f876268bb24dbe8ee677203da9eaef)` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:02:56.556300 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-258a5c24-3e64-46d5-b016-95708abe270c admin admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:02:56.707983 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-258a5c24-3e64-46d5-b016-95708abe270c admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '25f876268bb24dbe8ee677203da9eaef', 'request_id': 'req-258a5c24-3e64-46d5-b016-95708abe270c'} {{(pid=92056) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:02:56.775848 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-258a5c24-3e64-46d5-b016-95708abe270c admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '25f876268bb24dbe8ee677203da9eaef', 'request_id': 'req-258a5c24-3e64-46d5-b016-95708abe270c'} {{(pid=92056) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:02:56.791670 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-258a5c24-3e64-46d5-b016-95708abe270c admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '25f876268bb24dbe8ee677203da9eaef', 'request_id': 'req-258a5c24-3e64-46d5-b016-95708abe270c'} {{(pid=92056) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:02:56.805490 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-258a5c24-3e64-46d5-b016-95708abe270c admin admin] The token cache is being invalidate because project 25f876268bb24dbe8ee677203da9eaef was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=92056) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} May 23 13:02:56.805791 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-258a5c24-3e64-46d5-b016-95708abe270c admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=92056) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:02:56.806975 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-258a5c24-3e64-46d5-b016-95708abe270c admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=92056) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:02:56.809718 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 1169/2350] 10.222.0.30 () {58 vars in 1232 bytes} [Mon May 23 13:02:56 2022] DELETE /identity/v3/projects/25f876268bb24dbe8ee677203da9eaef => generated 0 bytes in 307 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) May 23 13:02:56.827617 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e2353d7-e422-45ce-b589-ce17ee926d75 None None] REQUEST_METHOD: `POST` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:02:56.828746 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e2353d7-e422-45ce-b589-ce17ee926d75 None None] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:02:56.829034 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e2353d7-e422-45ce-b589-ce17ee926d75 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:02:56.915823 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.auth.core [None req-5e2353d7-e422-45ce-b589-ce17ee926d75 None None] MFA Rules not processed for user `0f7119f8fa5c4445977671140ded91a4`. Rule list: `[]` (Enabled: `True`). {{(pid=92057) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 13:02:56.918101 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.fernet_utils [None req-5e2353d7-e422-45ce-b589-ce17ee926d75 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92057) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:02:57.040624 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 1182/2351] 10.222.0.30 () {58 vars in 957 bytes} [Mon May 23 13:02:56 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 226 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) May 23 13:02:57.047901 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f49fd5d6-04dc-4ff7-900c-4d9c660a3956 None None] REQUEST_METHOD: `POST` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:02:57.048417 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f49fd5d6-04dc-4ff7-900c-4d9c660a3956 None None] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:02:57.049437 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f49fd5d6-04dc-4ff7-900c-4d9c660a3956 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:02:57.087718 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.auth.core [None req-f49fd5d6-04dc-4ff7-900c-4d9c660a3956 None None] MFA Rules not processed for user `0f7119f8fa5c4445977671140ded91a4`. Rule list: `[]` (Enabled: `True`). {{(pid=92056) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 13:02:57.090749 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.fernet_utils [None req-f49fd5d6-04dc-4ff7-900c-4d9c660a3956 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92056) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:02:57.105750 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 1170/2352] 10.222.0.30 () {58 vars in 957 bytes} [Mon May 23 13:02:57 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 59 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) May 23 13:02:57.110805 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e2353d7-e422-45ce-b589-ce17ee926d75 None None] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:02:57.131868 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e2353d7-e422-45ce-b589-ce17ee926d75 None None] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:02:57.147830 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-402fa6f8-4e81-475f-bbce-f2a7a325479a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:02:57.148787 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-402fa6f8-4e81-475f-bbce-f2a7a325479a admin admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:02:57.149269 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-402fa6f8-4e81-475f-bbce-f2a7a325479a admin admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:02:57.149781 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-402fa6f8-4e81-475f-bbce-f2a7a325479a admin admin] PATH_INFO: `/v3/domains` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:02:57.151169 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-402fa6f8-4e81-475f-bbce-f2a7a325479a admin admin] RBAC: Adding query filter params (name=Default) {{(pid=92057) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} May 23 13:02:57.151879 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-402fa6f8-4e81-475f-bbce-f2a7a325479a admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:02:57.153937 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-402fa6f8-4e81-475f-bbce-f2a7a325479a admin admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:02:57.174553 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 1183/2353] 10.222.0.30 () {58 vars in 1152 bytes} [Mon May 23 13:02:57 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 64 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) May 23 13:02:57.175917 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f49fd5d6-04dc-4ff7-900c-4d9c660a3956 None None] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:02:57.207375 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f49fd5d6-04dc-4ff7-900c-4d9c660a3956 None None] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:02:57.227915 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38c86b51-acba-472b-b094-d238f0750d89 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:02:57.229895 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38c86b51-acba-472b-b094-d238f0750d89 admin admin] REQUEST_METHOD: `POST` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:02:57.230556 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38c86b51-acba-472b-b094-d238f0750d89 admin admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:02:57.230904 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38c86b51-acba-472b-b094-d238f0750d89 admin admin] PATH_INFO: `/v3/projects` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:02:57.234411 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38c86b51-acba-472b-b094-d238f0750d89 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:02:57.236301 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38c86b51-acba-472b-b094-d238f0750d89 admin admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:02:57.336468 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 1171/2354] 10.222.0.30 () {60 vars in 1152 bytes} [Mon May 23 13:02:57 2022] POST /identity/v3/projects => generated 399 bytes in 162 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) May 23 13:02:57.340127 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-402fa6f8-4e81-475f-bbce-f2a7a325479a admin admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:02:57.351701 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-402fa6f8-4e81-475f-bbce-f2a7a325479a admin admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:02:57.434250 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=93,keystone:DELETE=6,keystone:INSERT=1 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 13:02:57.494923 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43467507-f401-48b2-b60d-31c2cd03cf32 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:02:57.496682 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43467507-f401-48b2-b60d-31c2cd03cf32 admin admin] REQUEST_METHOD: `POST` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:02:57.497045 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43467507-f401-48b2-b60d-31c2cd03cf32 admin admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:02:57.497432 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43467507-f401-48b2-b60d-31c2cd03cf32 admin admin] PATH_INFO: `/v3/users` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:02:57.498499 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43467507-f401-48b2-b60d-31c2cd03cf32 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:02:57.504170 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43467507-f401-48b2-b60d-31c2cd03cf32 admin admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:02:57.541159 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38c86b51-acba-472b-b094-d238f0750d89 admin admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:02:57.543096 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.fernet_utils [None req-38c86b51-acba-472b-b094-d238f0750d89 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92056) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:02:57.571143 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38c86b51-acba-472b-b094-d238f0750d89 admin admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:02:57.578783 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-778afba5-8c9e-434a-9fc6-418e2a49467a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:02:57.579685 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-778afba5-8c9e-434a-9fc6-418e2a49467a admin admin] REQUEST_METHOD: `DELETE` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:02:57.579976 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-778afba5-8c9e-434a-9fc6-418e2a49467a admin admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:02:57.580254 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-778afba5-8c9e-434a-9fc6-418e2a49467a admin admin] PATH_INFO: `/v3/users/6b8c4a63ea3746e28dccfff48bc58e96` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:02:57.581512 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-778afba5-8c9e-434a-9fc6-418e2a49467a admin admin] RBAC: Authorizing `identity:delete_user(user_id=6b8c4a63ea3746e28dccfff48bc58e96)` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:02:57.583682 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-778afba5-8c9e-434a-9fc6-418e2a49467a admin admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:02:57.617170 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 1184/2355] 10.222.0.30 () {60 vars in 1143 bytes} [Mon May 23 13:02:57 2022] POST /identity/v3/users => generated 300 bytes in 276 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) May 23 13:02:57.617170 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43467507-f401-48b2-b60d-31c2cd03cf32 admin admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:02:57.630283 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43467507-f401-48b2-b60d-31c2cd03cf32 admin admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:02:57.639319 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81e814b5-a41a-49bd-ab2b-e78b360a810b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:02:57.642774 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81e814b5-a41a-49bd-ab2b-e78b360a810b admin admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:02:57.642774 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81e814b5-a41a-49bd-ab2b-e78b360a810b admin admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:02:57.642774 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81e814b5-a41a-49bd-ab2b-e78b360a810b admin admin] PATH_INFO: `/v3/roles` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:02:57.644166 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81e814b5-a41a-49bd-ab2b-e78b360a810b admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:02:57.646204 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81e814b5-a41a-49bd-ab2b-e78b360a810b admin admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:02:57.665593 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 1185/2356] 10.222.0.30 () {58 vars in 1121 bytes} [Mon May 23 13:02:57 2022] GET /identity/v3/roles => generated 1420 bytes in 49 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) May 23 13:02:57.667644 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81e814b5-a41a-49bd-ab2b-e78b360a810b admin admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:02:57.687520 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81e814b5-a41a-49bd-ab2b-e78b360a810b admin admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:02:57.703030 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c2b326d-6dcd-4a9e-a31a-997a1303c6c8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:02:57.703818 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c2b326d-6dcd-4a9e-a31a-997a1303c6c8 admin admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:02:57.704209 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c2b326d-6dcd-4a9e-a31a-997a1303c6c8 admin admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:02:57.704582 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c2b326d-6dcd-4a9e-a31a-997a1303c6c8 admin admin] PATH_INFO: `/v3/roles` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:02:57.705579 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c2b326d-6dcd-4a9e-a31a-997a1303c6c8 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:02:57.707735 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c2b326d-6dcd-4a9e-a31a-997a1303c6c8 admin admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:02:57.726746 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 1186/2357] 10.222.0.30 () {58 vars in 1121 bytes} [Mon May 23 13:02:57 2022] GET /identity/v3/roles => generated 1420 bytes in 60 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) May 23 13:02:57.727789 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c2b326d-6dcd-4a9e-a31a-997a1303c6c8 admin admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:02:57.738810 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c2b326d-6dcd-4a9e-a31a-997a1303c6c8 admin admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:02:57.745153 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc0930a3-0989-4cac-b08b-ba7a10923c00 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:02:57.746113 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc0930a3-0989-4cac-b08b-ba7a10923c00 admin admin] REQUEST_METHOD: `PUT` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:02:57.746493 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc0930a3-0989-4cac-b08b-ba7a10923c00 admin admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:02:57.746750 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc0930a3-0989-4cac-b08b-ba7a10923c00 admin admin] PATH_INFO: `/v3/projects/de7f8540cf7d466086d5b5ed1b0d0cc6/users/81b2c470bfc74086be43e5a33873377b/roles/80fb22507c9a4ff09d44d8f12f6fff1a` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:02:57.766648 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc0930a3-0989-4cac-b08b-ba7a10923c00 admin admin] RBAC: Authorizing `identity:create_grant(project_id=de7f8540cf7d466086d5b5ed1b0d0cc6, user_id=81b2c470bfc74086be43e5a33873377b, role_id=80fb22507c9a4ff09d44d8f12f6fff1a)` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:02:57.769946 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc0930a3-0989-4cac-b08b-ba7a10923c00 admin admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:02:57.818647 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-778afba5-8c9e-434a-9fc6-418e2a49467a admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '6b8c4a63ea3746e28dccfff48bc58e96'} {{(pid=92056) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:02:57.819674 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-778afba5-8c9e-434a-9fc6-418e2a49467a admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '6b8c4a63ea3746e28dccfff48bc58e96'} {{(pid=92056) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:02:57.912693 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 1187/2358] 10.222.0.30 () {60 vars in 1482 bytes} [Mon May 23 13:02:57 2022] PUT /identity/v3/projects/de7f8540cf7d466086d5b5ed1b0d0cc6/users/81b2c470bfc74086be43e5a33873377b/roles/80fb22507c9a4ff09d44d8f12f6fff1a => generated 0 bytes in 185 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) May 23 13:02:57.927538 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5369c0a8-ec17-4268-ae00-1463a868b284 None None] REQUEST_METHOD: `POST` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:02:57.927836 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5369c0a8-ec17-4268-ae00-1463a868b284 None None] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:02:57.928122 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5369c0a8-ec17-4268-ae00-1463a868b284 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:02:57.971344 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-778afba5-8c9e-434a-9fc6-418e2a49467a admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '6b8c4a63ea3746e28dccfff48bc58e96'} {{(pid=92056) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:02:57.984128 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.auth.core [None req-5369c0a8-ec17-4268-ae00-1463a868b284 None None] MFA Rules not processed for user `81b2c470bfc74086be43e5a33873377b`. Rule list: `[]` (Enabled: `True`). {{(pid=92057) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 13:02:57.986893 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.fernet_utils [None req-5369c0a8-ec17-4268-ae00-1463a868b284 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92057) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:02:58.002138 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-778afba5-8c9e-434a-9fc6-418e2a49467a admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '6b8c4a63ea3746e28dccfff48bc58e96'} {{(pid=92056) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:02:58.050298 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-778afba5-8c9e-434a-9fc6-418e2a49467a admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '6b8c4a63ea3746e28dccfff48bc58e96'} {{(pid=92056) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:02:58.055965 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=80,keystone:DELETE=17,keystone:INSERT=3 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 13:02:58.062140 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 1172/2359] 10.222.0.30 () {58 vars in 1223 bytes} [Mon May 23 13:02:57 2022] DELETE /identity/v3/users/6b8c4a63ea3746e28dccfff48bc58e96 => generated 0 bytes in 521 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) May 23 13:02:58.068545 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c35c0bfa-3386-4cf3-9877-99ebe1aafbb4 None None] REQUEST_METHOD: `POST` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:02:58.068838 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c35c0bfa-3386-4cf3-9877-99ebe1aafbb4 None None] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:02:58.069346 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c35c0bfa-3386-4cf3-9877-99ebe1aafbb4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:02:58.091588 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.auth.core [None req-c35c0bfa-3386-4cf3-9877-99ebe1aafbb4 None None] MFA Rules not processed for user `0f7119f8fa5c4445977671140ded91a4`. Rule list: `[]` (Enabled: `True`). {{(pid=92056) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 13:02:58.094538 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.fernet_utils [None req-c35c0bfa-3386-4cf3-9877-99ebe1aafbb4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92056) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:02:58.118542 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 1188/2360] 10.222.0.30 () {58 vars in 957 bytes} [Mon May 23 13:02:57 2022] POST /identity/v3/auth/tokens => generated 3266 bytes in 192 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) May 23 13:02:58.132508 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5369c0a8-ec17-4268-ae00-1463a868b284 None None] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:02:58.134314 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.fernet_utils [None req-5369c0a8-ec17-4268-ae00-1463a868b284 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92057) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:02:58.231731 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 1173/2361] 10.222.0.30 () {58 vars in 957 bytes} [Mon May 23 13:02:58 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 164 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) May 23 13:02:58.242109 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c35c0bfa-3386-4cf3-9877-99ebe1aafbb4 None None] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:02:58.243777 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.fernet_utils [None req-c35c0bfa-3386-4cf3-9877-99ebe1aafbb4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92056) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:02:58.283356 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5369c0a8-ec17-4268-ae00-1463a868b284 None None] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:02:58.315282 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 13:02:58.320429 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c35c0bfa-3386-4cf3-9877-99ebe1aafbb4 None None] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:02:58.331854 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc076945-522e-4e4d-9f51-bdf5de2e02f5 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2eb83ca38e79449095f3a2f947bbb06a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9c389ae65aaa4542bb5a5c4fd0bacf5b', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:02:58.334162 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc076945-522e-4e4d-9f51-bdf5de2e02f5 service neutron] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:02:58.335156 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc076945-522e-4e4d-9f51-bdf5de2e02f5 service neutron] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:02:58.335561 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc076945-522e-4e4d-9f51-bdf5de2e02f5 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:02:58.371948 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc076945-522e-4e4d-9f51-bdf5de2e02f5 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:02:58.374523 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2fef55f6-c158-48fd-b855-9ab29c5f9fe3 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1666097f8ed545c6b0e4809c93e4775a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9c389ae65aaa4542bb5a5c4fd0bacf5b', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:02:58.375396 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fef55f6-c158-48fd-b855-9ab29c5f9fe3 service cinder] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:02:58.378274 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc076945-522e-4e4d-9f51-bdf5de2e02f5 service neutron] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:02:58.378534 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fef55f6-c158-48fd-b855-9ab29c5f9fe3 service cinder] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:02:58.378534 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fef55f6-c158-48fd-b855-9ab29c5f9fe3 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:02:58.405418 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 1174/2362] 10.222.0.30 () {60 vars in 1365 bytes} [Mon May 23 13:02:58 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 164 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) May 23 13:02:58.460043 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2fef55f6-c158-48fd-b855-9ab29c5f9fe3 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:02:58.462404 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2fef55f6-c158-48fd-b855-9ab29c5f9fe3 service cinder] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:02:58.478256 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 1189/2363] 10.222.0.30 () {60 vars in 1365 bytes} [Mon May 23 13:02:58 2022] GET /identity/v3/auth/tokens => generated 3266 bytes in 346 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) May 23 13:02:59.111379 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc076945-522e-4e4d-9f51-bdf5de2e02f5 service neutron] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:02:59.118030 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.fernet_utils [None req-cc076945-522e-4e4d-9f51-bdf5de2e02f5 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92056) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:02:59.145601 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc076945-522e-4e4d-9f51-bdf5de2e02f5 service neutron] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:02:59.156958 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2f25f51-e4c8-4268-a32a-3ad8c5dd1d3c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:02:59.158876 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2f25f51-e4c8-4268-a32a-3ad8c5dd1d3c admin admin] REQUEST_METHOD: `DELETE` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:02:59.159466 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2f25f51-e4c8-4268-a32a-3ad8c5dd1d3c admin admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:02:59.159919 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2f25f51-e4c8-4268-a32a-3ad8c5dd1d3c admin admin] PATH_INFO: `/v3/projects/590ed5c70dca4aaf8dbef73ce1f00954` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:02:59.161845 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2f25f51-e4c8-4268-a32a-3ad8c5dd1d3c admin admin] RBAC: Authorizing `identity:delete_project(project_id=590ed5c70dca4aaf8dbef73ce1f00954)` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:02:59.164697 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2f25f51-e4c8-4268-a32a-3ad8c5dd1d3c admin admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:02:59.307170 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-d2f25f51-e4c8-4268-a32a-3ad8c5dd1d3c admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '590ed5c70dca4aaf8dbef73ce1f00954', 'request_id': 'req-d2f25f51-e4c8-4268-a32a-3ad8c5dd1d3c'} {{(pid=92056) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:02:59.422193 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-d2f25f51-e4c8-4268-a32a-3ad8c5dd1d3c admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '590ed5c70dca4aaf8dbef73ce1f00954', 'request_id': 'req-d2f25f51-e4c8-4268-a32a-3ad8c5dd1d3c'} {{(pid=92056) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:02:59.431278 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-d2f25f51-e4c8-4268-a32a-3ad8c5dd1d3c admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '590ed5c70dca4aaf8dbef73ce1f00954', 'request_id': 'req-d2f25f51-e4c8-4268-a32a-3ad8c5dd1d3c'} {{(pid=92056) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:02:59.448400 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-d2f25f51-e4c8-4268-a32a-3ad8c5dd1d3c admin admin] The token cache is being invalidate because project 590ed5c70dca4aaf8dbef73ce1f00954 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=92056) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} May 23 13:02:59.448959 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-d2f25f51-e4c8-4268-a32a-3ad8c5dd1d3c admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=92056) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:02:59.449658 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-d2f25f51-e4c8-4268-a32a-3ad8c5dd1d3c admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=92056) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:02:59.451749 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 1175/2364] 10.222.0.30 () {58 vars in 1232 bytes} [Mon May 23 13:02:59 2022] DELETE /identity/v3/projects/590ed5c70dca4aaf8dbef73ce1f00954 => generated 0 bytes in 341 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) May 23 13:02:59.457125 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9f1df5d-5033-4a6c-a876-cf14045fed74 None None] REQUEST_METHOD: `POST` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:02:59.457869 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9f1df5d-5033-4a6c-a876-cf14045fed74 None None] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:02:59.458283 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9f1df5d-5033-4a6c-a876-cf14045fed74 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:02:59.492292 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.auth.core [None req-f9f1df5d-5033-4a6c-a876-cf14045fed74 None None] MFA Rules not processed for user `0f7119f8fa5c4445977671140ded91a4`. Rule list: `[]` (Enabled: `True`). {{(pid=92057) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 13:02:59.498429 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.fernet_utils [None req-f9f1df5d-5033-4a6c-a876-cf14045fed74 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92057) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:02:59.630406 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 1190/2365] 10.222.0.30 () {58 vars in 957 bytes} [Mon May 23 13:02:59 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 175 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) May 23 13:02:59.649869 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-861d70ea-52c7-413c-ab16-97eb1ff5df67 None None] REQUEST_METHOD: `POST` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:02:59.650512 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-861d70ea-52c7-413c-ab16-97eb1ff5df67 None None] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:02:59.651128 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-861d70ea-52c7-413c-ab16-97eb1ff5df67 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:02:59.718796 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.auth.core [None req-861d70ea-52c7-413c-ab16-97eb1ff5df67 None None] MFA Rules not processed for user `0f7119f8fa5c4445977671140ded91a4`. Rule list: `[]` (Enabled: `True`). {{(pid=92056) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 13:02:59.720764 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.fernet_utils [None req-861d70ea-52c7-413c-ab16-97eb1ff5df67 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92056) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:02:59.744791 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 1176/2366] 10.222.0.30 () {58 vars in 957 bytes} [Mon May 23 13:02:59 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 94 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) May 23 13:02:59.745986 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9f1df5d-5033-4a6c-a876-cf14045fed74 None None] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:02:59.765750 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9f1df5d-5033-4a6c-a876-cf14045fed74 None None] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:02:59.772302 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8d54d0a-dad8-432e-a4c2-3d298467826d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:02:59.773160 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8d54d0a-dad8-432e-a4c2-3d298467826d admin admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:02:59.773687 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8d54d0a-dad8-432e-a4c2-3d298467826d admin admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:02:59.774131 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8d54d0a-dad8-432e-a4c2-3d298467826d admin admin] PATH_INFO: `/v3/domains` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:02:59.774875 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8d54d0a-dad8-432e-a4c2-3d298467826d admin admin] RBAC: Adding query filter params (name=Default) {{(pid=92057) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} May 23 13:02:59.775174 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8d54d0a-dad8-432e-a4c2-3d298467826d admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:02:59.777507 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8d54d0a-dad8-432e-a4c2-3d298467826d admin admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:02:59.805177 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 1191/2367] 10.222.0.30 () {58 vars in 1152 bytes} [Mon May 23 13:02:59 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 56 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) May 23 13:02:59.808378 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-861d70ea-52c7-413c-ab16-97eb1ff5df67 None None] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:02:59.861044 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-861d70ea-52c7-413c-ab16-97eb1ff5df67 None None] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:02:59.888836 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30e759e3-dd8a-4aef-bafa-607748229831 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:02:59.890876 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30e759e3-dd8a-4aef-bafa-607748229831 admin admin] REQUEST_METHOD: `POST` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:02:59.891391 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30e759e3-dd8a-4aef-bafa-607748229831 admin admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:02:59.892800 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30e759e3-dd8a-4aef-bafa-607748229831 admin admin] PATH_INFO: `/v3/projects` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:02:59.895895 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-30e759e3-dd8a-4aef-bafa-607748229831 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:02:59.901652 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-30e759e3-dd8a-4aef-bafa-607748229831 admin admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:03:00.039118 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=91,keystone:DELETE=7,keystone:INSERT=2 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 13:03:00.048857 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 1177/2368] 10.222.0.30 () {60 vars in 1152 bytes} [Mon May 23 13:02:59 2022] POST /identity/v3/projects => generated 375 bytes in 243 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) May 23 13:03:00.052909 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8d54d0a-dad8-432e-a4c2-3d298467826d admin admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:03:00.063380 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8d54d0a-dad8-432e-a4c2-3d298467826d admin admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:03:00.127309 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f31257e0-b071-46be-a3d1-88fcca0a4017 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:03:00.127979 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f31257e0-b071-46be-a3d1-88fcca0a4017 admin admin] REQUEST_METHOD: `POST` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:03:00.128271 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f31257e0-b071-46be-a3d1-88fcca0a4017 admin admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:03:00.128552 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f31257e0-b071-46be-a3d1-88fcca0a4017 admin admin] PATH_INFO: `/v3/users` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:03:00.129284 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f31257e0-b071-46be-a3d1-88fcca0a4017 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:03:00.131226 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f31257e0-b071-46be-a3d1-88fcca0a4017 admin admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:03:00.142686 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 13:03:00.245801 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 1192/2369] 10.222.0.30 () {60 vars in 1143 bytes} [Mon May 23 13:03:00 2022] POST /identity/v3/users => generated 288 bytes in 192 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) May 23 13:03:00.248354 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30e759e3-dd8a-4aef-bafa-607748229831 admin admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:03:00.258905 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30e759e3-dd8a-4aef-bafa-607748229831 admin admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:03:00.264357 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be335caf-6775-4143-a3a4-fd8edf8c1774 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:03:00.265043 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be335caf-6775-4143-a3a4-fd8edf8c1774 admin admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:03:00.267317 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be335caf-6775-4143-a3a4-fd8edf8c1774 admin admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:03:00.267317 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be335caf-6775-4143-a3a4-fd8edf8c1774 admin admin] PATH_INFO: `/v3/roles` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:03:00.267317 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be335caf-6775-4143-a3a4-fd8edf8c1774 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:03:00.268170 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be335caf-6775-4143-a3a4-fd8edf8c1774 admin admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:03:00.280863 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 1178/2370] 10.222.0.30 () {58 vars in 1121 bytes} [Mon May 23 13:03:00 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) May 23 13:03:00.285264 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f31257e0-b071-46be-a3d1-88fcca0a4017 admin admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:03:00.295633 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f31257e0-b071-46be-a3d1-88fcca0a4017 admin admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:03:00.301084 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01b617ea-6a89-468f-8256-15cb1c66c6e0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:03:00.301851 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01b617ea-6a89-468f-8256-15cb1c66c6e0 admin admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:03:00.302162 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01b617ea-6a89-468f-8256-15cb1c66c6e0 admin admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:03:00.302449 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01b617ea-6a89-468f-8256-15cb1c66c6e0 admin admin] PATH_INFO: `/v3/roles` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:03:00.303177 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01b617ea-6a89-468f-8256-15cb1c66c6e0 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:03:00.304861 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01b617ea-6a89-468f-8256-15cb1c66c6e0 admin admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:03:00.325155 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 1193/2371] 10.222.0.30 () {58 vars in 1121 bytes} [Mon May 23 13:03:00 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) May 23 13:03:00.329095 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be335caf-6775-4143-a3a4-fd8edf8c1774 admin admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:03:00.349015 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be335caf-6775-4143-a3a4-fd8edf8c1774 admin admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:03:00.354534 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-883ed165-32c7-43ae-a20e-27e21340f956 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:03:00.355976 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-883ed165-32c7-43ae-a20e-27e21340f956 admin admin] REQUEST_METHOD: `PUT` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:03:00.356258 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-883ed165-32c7-43ae-a20e-27e21340f956 admin admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:03:00.356590 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-883ed165-32c7-43ae-a20e-27e21340f956 admin admin] PATH_INFO: `/v3/projects/5c80d6715868431ba6e380161a197bf2/users/092afb685f72474fa09531e3b4d117ae/roles/80fb22507c9a4ff09d44d8f12f6fff1a` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:03:00.398664 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-883ed165-32c7-43ae-a20e-27e21340f956 admin admin] RBAC: Authorizing `identity:create_grant(project_id=5c80d6715868431ba6e380161a197bf2, user_id=092afb685f72474fa09531e3b4d117ae, role_id=80fb22507c9a4ff09d44d8f12f6fff1a)` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:03:00.408021 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-883ed165-32c7-43ae-a20e-27e21340f956 admin admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:03:00.486599 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 1179/2372] 10.222.0.30 () {60 vars in 1482 bytes} [Mon May 23 13:03:00 2022] PUT /identity/v3/projects/5c80d6715868431ba6e380161a197bf2/users/092afb685f72474fa09531e3b4d117ae/roles/80fb22507c9a4ff09d44d8f12f6fff1a => generated 0 bytes in 158 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) May 23 13:03:00.498657 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-572a80cb-57c5-4743-adfd-f7b075e3cb32 None None] REQUEST_METHOD: `POST` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:03:00.499199 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-572a80cb-57c5-4743-adfd-f7b075e3cb32 None None] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:03:00.499199 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-572a80cb-57c5-4743-adfd-f7b075e3cb32 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:03:00.539494 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.auth.core [None req-572a80cb-57c5-4743-adfd-f7b075e3cb32 None None] MFA Rules not processed for user `092afb685f72474fa09531e3b4d117ae`. Rule list: `[]` (Enabled: `True`). {{(pid=92057) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 13:03:00.541103 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.fernet_utils [None req-572a80cb-57c5-4743-adfd-f7b075e3cb32 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92057) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:03:00.646634 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 1194/2373] 10.222.0.30 () {58 vars in 957 bytes} [Mon May 23 13:03:00 2022] POST /identity/v3/auth/tokens => generated 3242 bytes in 149 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) May 23 13:03:00.662880 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-883ed165-32c7-43ae-a20e-27e21340f956 admin admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:03:00.665162 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.fernet_utils [None req-883ed165-32c7-43ae-a20e-27e21340f956 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92056) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:03:00.769417 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-883ed165-32c7-43ae-a20e-27e21340f956 admin admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:03:00.774358 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19c1ba59-c248-4411-891d-1d11241d3f99 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1666097f8ed545c6b0e4809c93e4775a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9c389ae65aaa4542bb5a5c4fd0bacf5b', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:03:00.775231 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19c1ba59-c248-4411-891d-1d11241d3f99 service cinder] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:03:00.775682 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19c1ba59-c248-4411-891d-1d11241d3f99 service cinder] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:03:00.776138 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19c1ba59-c248-4411-891d-1d11241d3f99 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:03:00.788169 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19c1ba59-c248-4411-891d-1d11241d3f99 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:03:00.789797 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19c1ba59-c248-4411-891d-1d11241d3f99 service cinder] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:03:00.801954 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 1180/2374] 10.222.0.30 () {60 vars in 1365 bytes} [Mon May 23 13:03:00 2022] GET /identity/v3/auth/tokens => generated 3242 bytes in 139 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) May 23 13:03:01.192981 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb684ec9-891d-4673-a012-401c2e9399a1 None None] REQUEST_METHOD: `POST` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:03:01.193277 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb684ec9-891d-4673-a012-401c2e9399a1 None None] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:03:01.193609 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb684ec9-891d-4673-a012-401c2e9399a1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:03:01.218604 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.auth.core [None req-cb684ec9-891d-4673-a012-401c2e9399a1 None None] MFA Rules not processed for user `b18e5f1b24634096995068e41d94cad6`. Rule list: `[]` (Enabled: `True`). {{(pid=92057) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 13:03:01.220507 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.fernet_utils [None req-cb684ec9-891d-4673-a012-401c2e9399a1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92057) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:03:01.294335 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 1195/2375] 10.222.0.30 () {58 vars in 957 bytes} [Mon May 23 13:03:01 2022] POST /identity/v3/auth/tokens => generated 3258 bytes in 102 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) May 23 13:03:03.386776 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19c1ba59-c248-4411-891d-1d11241d3f99 service cinder] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:03:03.389406 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.fernet_utils [None req-19c1ba59-c248-4411-891d-1d11241d3f99 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92056) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:03:03.491769 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19c1ba59-c248-4411-891d-1d11241d3f99 service cinder] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:03:03.495608 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32af12a6-c9b2-433c-84a2-44249d52c170 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2eb83ca38e79449095f3a2f947bbb06a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9c389ae65aaa4542bb5a5c4fd0bacf5b', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:03:03.496274 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32af12a6-c9b2-433c-84a2-44249d52c170 service neutron] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:03:03.496561 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32af12a6-c9b2-433c-84a2-44249d52c170 service neutron] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:03:03.496835 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32af12a6-c9b2-433c-84a2-44249d52c170 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:03:03.499004 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.fernet_utils [None req-32af12a6-c9b2-433c-84a2-44249d52c170 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92056) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:03:03.567190 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32af12a6-c9b2-433c-84a2-44249d52c170 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:03:03.568629 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32af12a6-c9b2-433c-84a2-44249d52c170 service neutron] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:03:03.581170 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 1181/2376] 10.222.0.30 () {60 vars in 1365 bytes} [Mon May 23 13:03:03 2022] GET /identity/v3/auth/tokens => generated 3311 bytes in 194 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) May 23 13:03:11.285741 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:INSERT=2,keystone:SELECT=64 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 13:03:12.103351 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb684ec9-891d-4673-a012-401c2e9399a1 None None] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:03:12.105162 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.fernet_utils [None req-cb684ec9-891d-4673-a012-401c2e9399a1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92057) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:03:12.187688 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb684ec9-891d-4673-a012-401c2e9399a1 None None] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:03:12.192452 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29bb9a27-6f3c-4fd8-a60a-a87e0fe52f1a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:03:12.193365 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29bb9a27-6f3c-4fd8-a60a-a87e0fe52f1a admin admin] REQUEST_METHOD: `DELETE` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:03:12.193721 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29bb9a27-6f3c-4fd8-a60a-a87e0fe52f1a admin admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:03:12.194044 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29bb9a27-6f3c-4fd8-a60a-a87e0fe52f1a admin admin] PATH_INFO: `/v3/users/81b2c470bfc74086be43e5a33873377b` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:03:12.195341 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-29bb9a27-6f3c-4fd8-a60a-a87e0fe52f1a admin admin] RBAC: Authorizing `identity:delete_user(user_id=81b2c470bfc74086be43e5a33873377b)` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:03:12.197373 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-29bb9a27-6f3c-4fd8-a60a-a87e0fe52f1a admin admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:03:12.430542 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [None req-29bb9a27-6f3c-4fd8-a60a-a87e0fe52f1a admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '81b2c470bfc74086be43e5a33873377b'} {{(pid=92057) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:03:12.492743 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [None req-29bb9a27-6f3c-4fd8-a60a-a87e0fe52f1a admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '81b2c470bfc74086be43e5a33873377b'} {{(pid=92057) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:03:12.501823 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [None req-29bb9a27-6f3c-4fd8-a60a-a87e0fe52f1a admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '81b2c470bfc74086be43e5a33873377b'} {{(pid=92057) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:03:12.502569 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [None req-29bb9a27-6f3c-4fd8-a60a-a87e0fe52f1a admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '81b2c470bfc74086be43e5a33873377b'} {{(pid=92057) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:03:12.515323 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [None req-29bb9a27-6f3c-4fd8-a60a-a87e0fe52f1a admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '81b2c470bfc74086be43e5a33873377b'} {{(pid=92057) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:03:12.521698 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 1196/2377] 10.222.0.30 () {58 vars in 1223 bytes} [Mon May 23 13:03:12 2022] DELETE /identity/v3/users/81b2c470bfc74086be43e5a33873377b => generated 0 bytes in 419 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) May 23 13:03:12.526539 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4acd8ef9-bdd6-479e-8642-a0f877345602 None None] REQUEST_METHOD: `POST` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:03:12.526946 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4acd8ef9-bdd6-479e-8642-a0f877345602 None None] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:03:12.527102 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4acd8ef9-bdd6-479e-8642-a0f877345602 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:03:12.548001 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.auth.core [None req-4acd8ef9-bdd6-479e-8642-a0f877345602 None None] MFA Rules not processed for user `0f7119f8fa5c4445977671140ded91a4`. Rule list: `[]` (Enabled: `True`). {{(pid=92056) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 13:03:12.549248 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.fernet_utils [None req-4acd8ef9-bdd6-479e-8642-a0f877345602 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92056) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:03:12.560771 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 13:03:12.617030 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 1182/2378] 10.222.0.30 () {58 vars in 957 bytes} [Mon May 23 13:03:12 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 91 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) May 23 13:03:12.627890 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29bb9a27-6f3c-4fd8-a60a-a87e0fe52f1a admin admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:03:12.630183 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.fernet_utils [None req-29bb9a27-6f3c-4fd8-a60a-a87e0fe52f1a admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92057) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:03:12.713348 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29bb9a27-6f3c-4fd8-a60a-a87e0fe52f1a admin admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:03:12.718707 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5a3423a-4f08-43b7-a569-43e094f47a3f service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2eb83ca38e79449095f3a2f947bbb06a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9c389ae65aaa4542bb5a5c4fd0bacf5b', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:03:12.719451 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5a3423a-4f08-43b7-a569-43e094f47a3f service neutron] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:03:12.719901 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5a3423a-4f08-43b7-a569-43e094f47a3f service neutron] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:03:12.720251 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5a3423a-4f08-43b7-a569-43e094f47a3f service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:03:12.734917 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5a3423a-4f08-43b7-a569-43e094f47a3f service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:03:12.736428 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5a3423a-4f08-43b7-a569-43e094f47a3f service neutron] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:03:12.748871 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 1197/2379] 10.222.0.30 () {60 vars in 1365 bytes} [Mon May 23 13:03:12 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 121 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) May 23 13:03:13.192109 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4acd8ef9-bdd6-479e-8642-a0f877345602 None None] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:03:13.193661 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.fernet_utils [None req-4acd8ef9-bdd6-479e-8642-a0f877345602 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92056) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:03:13.208666 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4acd8ef9-bdd6-479e-8642-a0f877345602 None None] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:03:13.213683 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82a44629-fbaa-4495-ac8d-2a7fe05bc1c8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:03:13.214659 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82a44629-fbaa-4495-ac8d-2a7fe05bc1c8 admin admin] REQUEST_METHOD: `DELETE` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:03:13.229413 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82a44629-fbaa-4495-ac8d-2a7fe05bc1c8 admin admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:03:13.229413 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82a44629-fbaa-4495-ac8d-2a7fe05bc1c8 admin admin] PATH_INFO: `/v3/projects/de7f8540cf7d466086d5b5ed1b0d0cc6` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:03:13.229413 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82a44629-fbaa-4495-ac8d-2a7fe05bc1c8 admin admin] RBAC: Authorizing `identity:delete_project(project_id=de7f8540cf7d466086d5b5ed1b0d0cc6)` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:03:13.229413 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82a44629-fbaa-4495-ac8d-2a7fe05bc1c8 admin admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:03:13.343633 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-82a44629-fbaa-4495-ac8d-2a7fe05bc1c8 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'de7f8540cf7d466086d5b5ed1b0d0cc6', 'request_id': 'req-82a44629-fbaa-4495-ac8d-2a7fe05bc1c8'} {{(pid=92056) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:03:13.401155 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-82a44629-fbaa-4495-ac8d-2a7fe05bc1c8 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'de7f8540cf7d466086d5b5ed1b0d0cc6', 'request_id': 'req-82a44629-fbaa-4495-ac8d-2a7fe05bc1c8'} {{(pid=92056) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:03:13.409822 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-82a44629-fbaa-4495-ac8d-2a7fe05bc1c8 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'de7f8540cf7d466086d5b5ed1b0d0cc6', 'request_id': 'req-82a44629-fbaa-4495-ac8d-2a7fe05bc1c8'} {{(pid=92056) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:03:13.421330 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-82a44629-fbaa-4495-ac8d-2a7fe05bc1c8 admin admin] The token cache is being invalidate because project de7f8540cf7d466086d5b5ed1b0d0cc6 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=92056) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} May 23 13:03:13.421630 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-82a44629-fbaa-4495-ac8d-2a7fe05bc1c8 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=92056) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:03:13.422323 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-82a44629-fbaa-4495-ac8d-2a7fe05bc1c8 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=92056) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:03:13.423973 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 1183/2380] 10.222.0.30 () {58 vars in 1232 bytes} [Mon May 23 13:03:13 2022] DELETE /identity/v3/projects/de7f8540cf7d466086d5b5ed1b0d0cc6 => generated 0 bytes in 232 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) May 23 13:03:21.385257 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5a3423a-4f08-43b7-a569-43e094f47a3f service neutron] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:03:21.386717 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.fernet_utils [None req-f5a3423a-4f08-43b7-a569-43e094f47a3f service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92057) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:03:21.400664 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5a3423a-4f08-43b7-a569-43e094f47a3f service neutron] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:03:21.421530 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 13:03:21.475798 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4b07902-4ed5-48a6-8000-1248f1f66dab admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:03:21.476674 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4b07902-4ed5-48a6-8000-1248f1f66dab admin admin] REQUEST_METHOD: `DELETE` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:03:21.476927 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4b07902-4ed5-48a6-8000-1248f1f66dab admin admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:03:21.477239 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4b07902-4ed5-48a6-8000-1248f1f66dab admin admin] PATH_INFO: `/v3/users/092afb685f72474fa09531e3b4d117ae` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:03:21.478524 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c4b07902-4ed5-48a6-8000-1248f1f66dab admin admin] RBAC: Authorizing `identity:delete_user(user_id=092afb685f72474fa09531e3b4d117ae)` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:03:21.480605 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c4b07902-4ed5-48a6-8000-1248f1f66dab admin admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:03:21.710335 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [None req-c4b07902-4ed5-48a6-8000-1248f1f66dab admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '092afb685f72474fa09531e3b4d117ae'} {{(pid=92057) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:03:21.807182 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [None req-c4b07902-4ed5-48a6-8000-1248f1f66dab admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '092afb685f72474fa09531e3b4d117ae'} {{(pid=92057) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:03:21.815735 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [None req-c4b07902-4ed5-48a6-8000-1248f1f66dab admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '092afb685f72474fa09531e3b4d117ae'} {{(pid=92057) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:03:21.816538 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [None req-c4b07902-4ed5-48a6-8000-1248f1f66dab admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '092afb685f72474fa09531e3b4d117ae'} {{(pid=92057) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:03:21.831209 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [None req-c4b07902-4ed5-48a6-8000-1248f1f66dab admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '092afb685f72474fa09531e3b4d117ae'} {{(pid=92057) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:03:21.838384 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 1198/2381] 10.222.0.30 () {58 vars in 1223 bytes} [Mon May 23 13:03:21 2022] DELETE /identity/v3/users/092afb685f72474fa09531e3b4d117ae => generated 0 bytes in 453 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) May 23 13:03:21.843556 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cfb64b8-48ab-4d29-a497-110f4f0bb188 None None] REQUEST_METHOD: `POST` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:03:21.844446 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cfb64b8-48ab-4d29-a497-110f4f0bb188 None None] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:03:21.844739 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cfb64b8-48ab-4d29-a497-110f4f0bb188 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:03:21.863237 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.auth.core [None req-2cfb64b8-48ab-4d29-a497-110f4f0bb188 None None] MFA Rules not processed for user `0f7119f8fa5c4445977671140ded91a4`. Rule list: `[]` (Enabled: `True`). {{(pid=92056) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 13:03:21.864506 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.fernet_utils [None req-2cfb64b8-48ab-4d29-a497-110f4f0bb188 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92056) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:03:21.929790 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 1184/2382] 10.222.0.30 () {58 vars in 957 bytes} [Mon May 23 13:03:21 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 87 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) May 23 13:03:21.939745 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4b07902-4ed5-48a6-8000-1248f1f66dab admin admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:03:21.941247 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.fernet_utils [None req-c4b07902-4ed5-48a6-8000-1248f1f66dab admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92057) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:03:22.008908 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4b07902-4ed5-48a6-8000-1248f1f66dab admin admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:03:22.013768 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3dc6471c-7cfc-42d8-86e4-d7e393eb329b service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2eb83ca38e79449095f3a2f947bbb06a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9c389ae65aaa4542bb5a5c4fd0bacf5b', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:03:22.014318 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3dc6471c-7cfc-42d8-86e4-d7e393eb329b service neutron] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:03:22.014657 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3dc6471c-7cfc-42d8-86e4-d7e393eb329b service neutron] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:03:22.015002 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3dc6471c-7cfc-42d8-86e4-d7e393eb329b service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:03:22.026666 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3dc6471c-7cfc-42d8-86e4-d7e393eb329b service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:03:22.028129 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3dc6471c-7cfc-42d8-86e4-d7e393eb329b service neutron] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:03:22.039375 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 1199/2383] 10.222.0.30 () {60 vars in 1365 bytes} [Mon May 23 13:03:21 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 100 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) May 23 13:03:22.520674 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2cfb64b8-48ab-4d29-a497-110f4f0bb188 None None] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:03:22.522249 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.fernet_utils [None req-2cfb64b8-48ab-4d29-a497-110f4f0bb188 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92056) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:03:22.539543 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2cfb64b8-48ab-4d29-a497-110f4f0bb188 None None] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:03:22.545168 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a102f42-0cd5-4589-a228-bb1a42e68bcb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:03:22.546080 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a102f42-0cd5-4589-a228-bb1a42e68bcb admin admin] REQUEST_METHOD: `DELETE` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:03:22.546551 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a102f42-0cd5-4589-a228-bb1a42e68bcb admin admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:03:22.546832 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a102f42-0cd5-4589-a228-bb1a42e68bcb admin admin] PATH_INFO: `/v3/projects/5c80d6715868431ba6e380161a197bf2` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:03:22.548223 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a102f42-0cd5-4589-a228-bb1a42e68bcb admin admin] RBAC: Authorizing `identity:delete_project(project_id=5c80d6715868431ba6e380161a197bf2)` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:03:22.550496 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a102f42-0cd5-4589-a228-bb1a42e68bcb admin admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:03:22.648711 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=90,keystone:DELETE=9,keystone:INSERT=1 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 13:03:22.664186 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-7a102f42-0cd5-4589-a228-bb1a42e68bcb admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '5c80d6715868431ba6e380161a197bf2', 'request_id': 'req-7a102f42-0cd5-4589-a228-bb1a42e68bcb'} {{(pid=92056) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:03:22.729821 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-7a102f42-0cd5-4589-a228-bb1a42e68bcb admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '5c80d6715868431ba6e380161a197bf2', 'request_id': 'req-7a102f42-0cd5-4589-a228-bb1a42e68bcb'} {{(pid=92056) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:03:22.739044 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-7a102f42-0cd5-4589-a228-bb1a42e68bcb admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '5c80d6715868431ba6e380161a197bf2', 'request_id': 'req-7a102f42-0cd5-4589-a228-bb1a42e68bcb'} {{(pid=92056) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:03:22.752384 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-7a102f42-0cd5-4589-a228-bb1a42e68bcb admin admin] The token cache is being invalidate because project 5c80d6715868431ba6e380161a197bf2 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=92056) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} May 23 13:03:22.752384 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-7a102f42-0cd5-4589-a228-bb1a42e68bcb admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=92056) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:03:22.752969 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-7a102f42-0cd5-4589-a228-bb1a42e68bcb admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=92056) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:03:22.754845 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 1185/2384] 10.222.0.30 () {58 vars in 1232 bytes} [Mon May 23 13:03:22 2022] DELETE /identity/v3/projects/5c80d6715868431ba6e380161a197bf2 => generated 0 bytes in 234 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) May 23 13:03:22.759803 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-355cb63f-fc6e-4e99-a106-f20775ae52b2 None None] REQUEST_METHOD: `POST` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:03:22.760563 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-355cb63f-fc6e-4e99-a106-f20775ae52b2 None None] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:03:22.760563 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-355cb63f-fc6e-4e99-a106-f20775ae52b2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:03:22.781022 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.auth.core [None req-355cb63f-fc6e-4e99-a106-f20775ae52b2 None None] MFA Rules not processed for user `0f7119f8fa5c4445977671140ded91a4`. Rule list: `[]` (Enabled: `True`). {{(pid=92057) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 13:03:22.782469 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.fernet_utils [None req-355cb63f-fc6e-4e99-a106-f20775ae52b2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92057) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:03:22.837731 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 13:03:22.867765 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 1200/2385] 10.222.0.30 () {58 vars in 957 bytes} [Mon May 23 13:03:22 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 109 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) May 23 13:03:22.869600 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa0905d3-2482-482e-84fa-e62f7c0df7c7 None None] REQUEST_METHOD: `POST` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:03:22.871620 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa0905d3-2482-482e-84fa-e62f7c0df7c7 None None] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:03:22.873418 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa0905d3-2482-482e-84fa-e62f7c0df7c7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:03:22.898303 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.auth.core [None req-aa0905d3-2482-482e-84fa-e62f7c0df7c7 None None] MFA Rules not processed for user `0f7119f8fa5c4445977671140ded91a4`. Rule list: `[]` (Enabled: `True`). {{(pid=92056) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 13:03:22.899809 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.fernet_utils [None req-aa0905d3-2482-482e-84fa-e62f7c0df7c7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92056) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:03:22.912927 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 1186/2386] 10.222.0.30 () {58 vars in 957 bytes} [Mon May 23 13:03:22 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 44 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) May 23 13:03:22.919287 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-355cb63f-fc6e-4e99-a106-f20775ae52b2 None None] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:03:22.943680 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-355cb63f-fc6e-4e99-a106-f20775ae52b2 None None] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:03:22.990427 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b769b23c-e501-4b2e-8e77-9f4cd5a38ba7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:03:22.995090 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b769b23c-e501-4b2e-8e77-9f4cd5a38ba7 admin admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:03:22.995513 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b769b23c-e501-4b2e-8e77-9f4cd5a38ba7 admin admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:03:22.995800 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b769b23c-e501-4b2e-8e77-9f4cd5a38ba7 admin admin] PATH_INFO: `/v3/domains` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:03:22.996616 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b769b23c-e501-4b2e-8e77-9f4cd5a38ba7 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=92057) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} May 23 13:03:22.996934 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b769b23c-e501-4b2e-8e77-9f4cd5a38ba7 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:03:23.000797 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b769b23c-e501-4b2e-8e77-9f4cd5a38ba7 admin admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:03:23.049923 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 1201/2387] 10.222.0.30 () {58 vars in 1152 bytes} [Mon May 23 13:03:22 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 130 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) May 23 13:03:23.053014 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa0905d3-2482-482e-84fa-e62f7c0df7c7 None None] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:03:23.063992 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa0905d3-2482-482e-84fa-e62f7c0df7c7 None None] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:03:23.077015 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d82467d2-12ba-4068-bc1d-8c91f4e7f277 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:03:23.077015 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d82467d2-12ba-4068-bc1d-8c91f4e7f277 admin admin] REQUEST_METHOD: `POST` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:03:23.077015 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d82467d2-12ba-4068-bc1d-8c91f4e7f277 admin admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:03:23.077015 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d82467d2-12ba-4068-bc1d-8c91f4e7f277 admin admin] PATH_INFO: `/v3/projects` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:03:23.077720 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d82467d2-12ba-4068-bc1d-8c91f4e7f277 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:03:23.080513 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d82467d2-12ba-4068-bc1d-8c91f4e7f277 admin admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:03:23.229179 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 1187/2388] 10.222.0.30 () {60 vars in 1152 bytes} [Mon May 23 13:03:23 2022] POST /identity/v3/projects => generated 389 bytes in 177 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) May 23 13:03:23.234059 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b769b23c-e501-4b2e-8e77-9f4cd5a38ba7 admin admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:03:23.244112 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b769b23c-e501-4b2e-8e77-9f4cd5a38ba7 admin admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:03:23.321812 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b11da675-bbe9-4a68-99f6-c60204892433 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:03:23.322946 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b11da675-bbe9-4a68-99f6-c60204892433 admin admin] REQUEST_METHOD: `POST` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:03:23.323433 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b11da675-bbe9-4a68-99f6-c60204892433 admin admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:03:23.323926 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b11da675-bbe9-4a68-99f6-c60204892433 admin admin] PATH_INFO: `/v3/users` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:03:23.324978 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b11da675-bbe9-4a68-99f6-c60204892433 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:03:23.328098 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b11da675-bbe9-4a68-99f6-c60204892433 admin admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:03:23.452811 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 1202/2389] 10.222.0.30 () {60 vars in 1143 bytes} [Mon May 23 13:03:23 2022] POST /identity/v3/users => generated 295 bytes in 216 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) May 23 13:03:23.454750 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d82467d2-12ba-4068-bc1d-8c91f4e7f277 admin admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:03:23.471859 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d82467d2-12ba-4068-bc1d-8c91f4e7f277 admin admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:03:23.486081 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11989b5c-00db-41d3-b381-db1490e04834 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:03:23.487026 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11989b5c-00db-41d3-b381-db1490e04834 admin admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:03:23.487508 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11989b5c-00db-41d3-b381-db1490e04834 admin admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:03:23.488512 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11989b5c-00db-41d3-b381-db1490e04834 admin admin] PATH_INFO: `/v3/roles` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:03:23.490010 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11989b5c-00db-41d3-b381-db1490e04834 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:03:23.492286 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11989b5c-00db-41d3-b381-db1490e04834 admin admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:03:23.506395 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 1188/2390] 10.222.0.30 () {58 vars in 1121 bytes} [Mon May 23 13:03:23 2022] GET /identity/v3/roles => generated 1420 bytes in 52 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) May 23 13:03:23.510120 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b11da675-bbe9-4a68-99f6-c60204892433 admin admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:03:23.521124 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b11da675-bbe9-4a68-99f6-c60204892433 admin admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:03:23.525582 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c7eb898-7fb4-457c-9cd7-22b39e7b4d84 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:03:23.526299 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c7eb898-7fb4-457c-9cd7-22b39e7b4d84 admin admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:03:23.526582 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c7eb898-7fb4-457c-9cd7-22b39e7b4d84 admin admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:03:23.526857 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c7eb898-7fb4-457c-9cd7-22b39e7b4d84 admin admin] PATH_INFO: `/v3/roles` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:03:23.527576 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c7eb898-7fb4-457c-9cd7-22b39e7b4d84 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:03:23.529937 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c7eb898-7fb4-457c-9cd7-22b39e7b4d84 admin admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:03:23.540148 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 1203/2391] 10.222.0.30 () {58 vars in 1121 bytes} [Mon May 23 13:03:23 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) May 23 13:03:23.544194 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11989b5c-00db-41d3-b381-db1490e04834 admin admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:03:23.554027 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11989b5c-00db-41d3-b381-db1490e04834 admin admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:03:23.559086 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47002a49-4531-41ba-b0fa-3f4c1c132c81 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:03:23.560500 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47002a49-4531-41ba-b0fa-3f4c1c132c81 admin admin] REQUEST_METHOD: `PUT` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:03:23.560903 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47002a49-4531-41ba-b0fa-3f4c1c132c81 admin admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:03:23.561183 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47002a49-4531-41ba-b0fa-3f4c1c132c81 admin admin] PATH_INFO: `/v3/projects/ef486840486f408bab70af5c07f3d683/users/68c12740b64549fdab49b511d23ad01f/roles/80fb22507c9a4ff09d44d8f12f6fff1a` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:03:23.587589 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47002a49-4531-41ba-b0fa-3f4c1c132c81 admin admin] RBAC: Authorizing `identity:create_grant(project_id=ef486840486f408bab70af5c07f3d683, user_id=68c12740b64549fdab49b511d23ad01f, role_id=80fb22507c9a4ff09d44d8f12f6fff1a)` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:03:23.590487 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47002a49-4531-41ba-b0fa-3f4c1c132c81 admin admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:03:23.699968 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 1189/2392] 10.222.0.30 () {60 vars in 1482 bytes} [Mon May 23 13:03:23 2022] PUT /identity/v3/projects/ef486840486f408bab70af5c07f3d683/users/68c12740b64549fdab49b511d23ad01f/roles/80fb22507c9a4ff09d44d8f12f6fff1a => generated 0 bytes in 156 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) May 23 13:03:23.710687 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f2e6324-f1ed-496c-bee5-2b2a198068dd None None] REQUEST_METHOD: `POST` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:03:23.711001 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f2e6324-f1ed-496c-bee5-2b2a198068dd None None] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:03:23.711260 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f2e6324-f1ed-496c-bee5-2b2a198068dd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:03:23.749321 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.auth.core [None req-8f2e6324-f1ed-496c-bee5-2b2a198068dd None None] MFA Rules not processed for user `68c12740b64549fdab49b511d23ad01f`. Rule list: `[]` (Enabled: `True`). {{(pid=92057) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 13:03:23.750594 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.fernet_utils [None req-8f2e6324-f1ed-496c-bee5-2b2a198068dd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92057) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:03:23.865537 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 1204/2393] 10.222.0.30 () {58 vars in 957 bytes} [Mon May 23 13:03:23 2022] POST /identity/v3/auth/tokens => generated 3256 bytes in 156 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) May 23 13:03:23.880588 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47002a49-4531-41ba-b0fa-3f4c1c132c81 admin admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:03:23.882960 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.fernet_utils [None req-47002a49-4531-41ba-b0fa-3f4c1c132c81 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92056) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:03:23.987957 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47002a49-4531-41ba-b0fa-3f4c1c132c81 admin admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:03:23.994114 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ffd60d7-ba25-4c8e-9cfe-79369c281284 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1666097f8ed545c6b0e4809c93e4775a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9c389ae65aaa4542bb5a5c4fd0bacf5b', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:03:23.994773 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ffd60d7-ba25-4c8e-9cfe-79369c281284 service cinder] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:03:23.995194 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ffd60d7-ba25-4c8e-9cfe-79369c281284 service cinder] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:03:23.995512 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ffd60d7-ba25-4c8e-9cfe-79369c281284 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:03:24.006001 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ffd60d7-ba25-4c8e-9cfe-79369c281284 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:03:24.007463 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ffd60d7-ba25-4c8e-9cfe-79369c281284 service cinder] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:03:24.025521 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 1190/2394] 10.222.0.30 () {60 vars in 1365 bytes} [Mon May 23 13:03:23 2022] GET /identity/v3/auth/tokens => generated 3256 bytes in 145 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) May 23 13:03:25.166157 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f2e6324-f1ed-496c-bee5-2b2a198068dd None None] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:03:25.167555 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.fernet_utils [None req-8f2e6324-f1ed-496c-bee5-2b2a198068dd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92057) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:03:25.222159 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 13:03:25.270925 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f2e6324-f1ed-496c-bee5-2b2a198068dd None None] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:03:25.276509 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73d51e4a-2f2a-49b6-91d2-39b72779b0cb service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd5fb51f8d0fa46439cb29666970b885b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9c389ae65aaa4542bb5a5c4fd0bacf5b', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'service', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:03:25.281130 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73d51e4a-2f2a-49b6-91d2-39b72779b0cb service placement] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:03:25.281130 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73d51e4a-2f2a-49b6-91d2-39b72779b0cb service placement] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:03:25.281130 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73d51e4a-2f2a-49b6-91d2-39b72779b0cb service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:03:25.281130 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.fernet_utils [None req-73d51e4a-2f2a-49b6-91d2-39b72779b0cb service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92057) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:03:25.292310 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-73d51e4a-2f2a-49b6-91d2-39b72779b0cb service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:03:25.294031 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-73d51e4a-2f2a-49b6-91d2-39b72779b0cb service placement] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:03:25.308220 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 1205/2395] 10.222.0.30 () {60 vars in 1365 bytes} [Mon May 23 13:03:25 2022] GET /identity/v3/auth/tokens => generated 3298 bytes in 143 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) May 23 13:03:26.117815 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6025fda-ed38-4d4e-9457-07835e685e0a None None] REQUEST_METHOD: `POST` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:03:26.118330 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6025fda-ed38-4d4e-9457-07835e685e0a None None] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:03:26.118433 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6025fda-ed38-4d4e-9457-07835e685e0a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:03:26.135918 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.auth.core [None req-e6025fda-ed38-4d4e-9457-07835e685e0a None None] MFA Rules not processed for user `6d21a56126fd4ccd837c33ccc08ab609`. Rule list: `[]` (Enabled: `True`). {{(pid=92056) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 13:03:26.137107 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.fernet_utils [None req-e6025fda-ed38-4d4e-9457-07835e685e0a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92056) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:03:26.192734 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 1191/2396] 10.222.0.30 () {58 vars in 967 bytes} [Mon May 23 13:03:26 2022] POST /identity/v3/auth/tokens => generated 3185 bytes in 76 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) May 23 13:03:26.205982 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73d51e4a-2f2a-49b6-91d2-39b72779b0cb service placement] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:03:26.207393 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.fernet_utils [None req-73d51e4a-2f2a-49b6-91d2-39b72779b0cb service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92057) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:03:26.275900 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73d51e4a-2f2a-49b6-91d2-39b72779b0cb service placement] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:03:26.282090 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15bc185b-afb1-420d-8566-4e0cbc7d3c94 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '138d91edb99f450e846765f9eef00d62', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9c389ae65aaa4542bb5a5c4fd0bacf5b', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'service', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:03:26.282725 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15bc185b-afb1-420d-8566-4e0cbc7d3c94 service swift] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:03:26.283009 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15bc185b-afb1-420d-8566-4e0cbc7d3c94 service swift] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:03:26.283286 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15bc185b-afb1-420d-8566-4e0cbc7d3c94 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:03:26.291208 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15bc185b-afb1-420d-8566-4e0cbc7d3c94 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:03:26.292799 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15bc185b-afb1-420d-8566-4e0cbc7d3c94 service swift] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:03:26.301180 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 1206/2397] 10.222.0.30 () {60 vars in 1384 bytes} [Mon May 23 13:03:26 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 96 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) May 23 13:03:30.465963 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-597ba22e-cfd3-4b2e-bd6b-928b3ad204e4 None None] REQUEST_METHOD: `POST` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:03:30.467653 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-597ba22e-cfd3-4b2e-bd6b-928b3ad204e4 None None] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:03:30.467653 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-597ba22e-cfd3-4b2e-bd6b-928b3ad204e4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:03:30.478731 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:DELETE=5,keystone:SELECT=92,keystone:INSERT=3 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 13:03:30.493110 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.auth.core [None req-597ba22e-cfd3-4b2e-bd6b-928b3ad204e4 None None] MFA Rules not processed for user `257251830f5148e5b9e435c2c13828e8`. Rule list: `[]` (Enabled: `True`). {{(pid=92056) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 13:03:30.496240 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.fernet_utils [None req-597ba22e-cfd3-4b2e-bd6b-928b3ad204e4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92056) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:03:30.561488 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 1192/2398] 10.222.0.30 () {58 vars in 957 bytes} [Mon May 23 13:03:30 2022] POST /identity/v3/auth/tokens => generated 3252 bytes in 97 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) May 23 13:03:35.642298 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15bc185b-afb1-420d-8566-4e0cbc7d3c94 service swift] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:03:35.644060 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.fernet_utils [None req-15bc185b-afb1-420d-8566-4e0cbc7d3c94 service swift] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92057) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:03:35.722074 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15bc185b-afb1-420d-8566-4e0cbc7d3c94 service swift] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:03:35.727400 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82ac5cd6-7e92-41f2-b546-ca42b777c365 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:03:35.728271 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82ac5cd6-7e92-41f2-b546-ca42b777c365 admin admin] REQUEST_METHOD: `DELETE` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:03:35.728563 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82ac5cd6-7e92-41f2-b546-ca42b777c365 admin admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:03:35.728820 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82ac5cd6-7e92-41f2-b546-ca42b777c365 admin admin] PATH_INFO: `/v3/users/7f5e411f4d8e4f13b662cf2846720cd3` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:03:35.730201 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82ac5cd6-7e92-41f2-b546-ca42b777c365 admin admin] RBAC: Authorizing `identity:delete_user(user_id=7f5e411f4d8e4f13b662cf2846720cd3)` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:03:35.732834 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82ac5cd6-7e92-41f2-b546-ca42b777c365 admin admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:03:35.902991 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=94,keystone:DELETE=6 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 13:03:35.908658 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [None req-82ac5cd6-7e92-41f2-b546-ca42b777c365 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '7f5e411f4d8e4f13b662cf2846720cd3'} {{(pid=92057) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:03:35.968465 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [None req-82ac5cd6-7e92-41f2-b546-ca42b777c365 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '7f5e411f4d8e4f13b662cf2846720cd3'} {{(pid=92057) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:03:35.976064 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [None req-82ac5cd6-7e92-41f2-b546-ca42b777c365 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '7f5e411f4d8e4f13b662cf2846720cd3'} {{(pid=92057) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:03:35.976707 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [None req-82ac5cd6-7e92-41f2-b546-ca42b777c365 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '7f5e411f4d8e4f13b662cf2846720cd3'} {{(pid=92057) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:03:35.989473 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [None req-82ac5cd6-7e92-41f2-b546-ca42b777c365 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '7f5e411f4d8e4f13b662cf2846720cd3'} {{(pid=92057) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:03:35.995399 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 1207/2399] 10.222.0.30 () {58 vars in 1223 bytes} [Mon May 23 13:03:35 2022] DELETE /identity/v3/users/7f5e411f4d8e4f13b662cf2846720cd3 => generated 0 bytes in 354 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) May 23 13:03:36.272784 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-597ba22e-cfd3-4b2e-bd6b-928b3ad204e4 None None] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:03:36.303827 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-597ba22e-cfd3-4b2e-bd6b-928b3ad204e4 None None] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:03:36.362094 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4781d9d-fec1-4714-aa86-3d5d47f28701 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:03:36.362946 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4781d9d-fec1-4714-aa86-3d5d47f28701 admin admin] REQUEST_METHOD: `DELETE` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:03:36.363231 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4781d9d-fec1-4714-aa86-3d5d47f28701 admin admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:03:36.363503 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4781d9d-fec1-4714-aa86-3d5d47f28701 admin admin] PATH_INFO: `/v3/projects/3b3719ab772449d79dc00dc96487dc36` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:03:36.364681 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4781d9d-fec1-4714-aa86-3d5d47f28701 admin admin] RBAC: Authorizing `identity:delete_project(project_id=3b3719ab772449d79dc00dc96487dc36)` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:03:36.366614 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4781d9d-fec1-4714-aa86-3d5d47f28701 admin admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:03:36.461828 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-b4781d9d-fec1-4714-aa86-3d5d47f28701 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '3b3719ab772449d79dc00dc96487dc36', 'request_id': 'req-b4781d9d-fec1-4714-aa86-3d5d47f28701'} {{(pid=92056) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:03:36.532841 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-b4781d9d-fec1-4714-aa86-3d5d47f28701 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '3b3719ab772449d79dc00dc96487dc36', 'request_id': 'req-b4781d9d-fec1-4714-aa86-3d5d47f28701'} {{(pid=92056) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:03:36.539283 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-b4781d9d-fec1-4714-aa86-3d5d47f28701 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '3b3719ab772449d79dc00dc96487dc36', 'request_id': 'req-b4781d9d-fec1-4714-aa86-3d5d47f28701'} {{(pid=92056) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:03:36.550532 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-b4781d9d-fec1-4714-aa86-3d5d47f28701 admin admin] The token cache is being invalidate because project 3b3719ab772449d79dc00dc96487dc36 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=92056) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} May 23 13:03:36.550876 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-b4781d9d-fec1-4714-aa86-3d5d47f28701 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=92056) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:03:36.551509 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-b4781d9d-fec1-4714-aa86-3d5d47f28701 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=92056) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:03:36.553069 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 1193/2400] 10.222.0.30 () {58 vars in 1232 bytes} [Mon May 23 13:03:36 2022] DELETE /identity/v3/projects/3b3719ab772449d79dc00dc96487dc36 => generated 0 bytes in 281 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) May 23 13:03:36.558891 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e728b0b8-d757-4efa-b1c9-fcaee0db12da None None] REQUEST_METHOD: `POST` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:03:36.559182 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e728b0b8-d757-4efa-b1c9-fcaee0db12da None None] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:03:36.559456 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e728b0b8-d757-4efa-b1c9-fcaee0db12da None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:03:36.577794 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.auth.core [None req-e728b0b8-d757-4efa-b1c9-fcaee0db12da None None] MFA Rules not processed for user `0f7119f8fa5c4445977671140ded91a4`. Rule list: `[]` (Enabled: `True`). {{(pid=92057) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 13:03:36.579049 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.fernet_utils [None req-e728b0b8-d757-4efa-b1c9-fcaee0db12da None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92057) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:03:36.642709 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 1208/2401] 10.222.0.30 () {58 vars in 957 bytes} [Mon May 23 13:03:36 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) May 23 13:03:36.650523 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2681d3fb-1f2d-413c-ad6d-3a14369bc765 None None] REQUEST_METHOD: `POST` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:03:36.650812 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2681d3fb-1f2d-413c-ad6d-3a14369bc765 None None] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:03:36.651095 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2681d3fb-1f2d-413c-ad6d-3a14369bc765 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:03:36.670917 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.auth.core [None req-2681d3fb-1f2d-413c-ad6d-3a14369bc765 None None] MFA Rules not processed for user `0f7119f8fa5c4445977671140ded91a4`. Rule list: `[]` (Enabled: `True`). {{(pid=92056) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 13:03:36.672101 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.fernet_utils [None req-2681d3fb-1f2d-413c-ad6d-3a14369bc765 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92056) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:03:36.680139 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 1194/2402] 10.222.0.30 () {58 vars in 957 bytes} [Mon May 23 13:03:36 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 30 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) May 23 13:03:36.684653 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e728b0b8-d757-4efa-b1c9-fcaee0db12da None None] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:03:36.695241 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e728b0b8-d757-4efa-b1c9-fcaee0db12da None None] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:03:36.700688 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-052f5576-6b99-44d8-b99d-4d621d958e68 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:03:36.701607 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-052f5576-6b99-44d8-b99d-4d621d958e68 admin admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:03:36.701837 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-052f5576-6b99-44d8-b99d-4d621d958e68 admin admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:03:36.702143 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-052f5576-6b99-44d8-b99d-4d621d958e68 admin admin] PATH_INFO: `/v3/domains` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:03:36.702880 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-052f5576-6b99-44d8-b99d-4d621d958e68 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=92057) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} May 23 13:03:36.703186 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-052f5576-6b99-44d8-b99d-4d621d958e68 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:03:36.704889 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-052f5576-6b99-44d8-b99d-4d621d958e68 admin admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:03:36.717761 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 1209/2403] 10.222.0.30 () {58 vars in 1152 bytes} [Mon May 23 13:03:36 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) May 23 13:03:36.722526 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2681d3fb-1f2d-413c-ad6d-3a14369bc765 None None] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:03:36.731893 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2681d3fb-1f2d-413c-ad6d-3a14369bc765 None None] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:03:36.736643 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-568b9104-9e45-41fa-ac6a-62782a9d94bd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:03:36.737302 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-568b9104-9e45-41fa-ac6a-62782a9d94bd admin admin] REQUEST_METHOD: `POST` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:03:36.737711 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-568b9104-9e45-41fa-ac6a-62782a9d94bd admin admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:03:36.738032 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-568b9104-9e45-41fa-ac6a-62782a9d94bd admin admin] PATH_INFO: `/v3/projects` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:03:36.738727 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-568b9104-9e45-41fa-ac6a-62782a9d94bd admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:03:36.740637 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-568b9104-9e45-41fa-ac6a-62782a9d94bd admin admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:03:36.817949 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 1195/2404] 10.222.0.30 () {60 vars in 1152 bytes} [Mon May 23 13:03:36 2022] POST /identity/v3/projects => generated 365 bytes in 96 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) May 23 13:03:36.822154 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-052f5576-6b99-44d8-b99d-4d621d958e68 admin admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:03:36.833035 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-052f5576-6b99-44d8-b99d-4d621d958e68 admin admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:03:36.890407 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-086cad05-9c8d-43fb-bfd0-6dcd36b60f92 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:03:36.891071 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-086cad05-9c8d-43fb-bfd0-6dcd36b60f92 admin admin] REQUEST_METHOD: `POST` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:03:36.891352 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-086cad05-9c8d-43fb-bfd0-6dcd36b60f92 admin admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:03:36.891624 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-086cad05-9c8d-43fb-bfd0-6dcd36b60f92 admin admin] PATH_INFO: `/v3/users` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:03:36.892335 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-086cad05-9c8d-43fb-bfd0-6dcd36b60f92 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:03:36.894596 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-086cad05-9c8d-43fb-bfd0-6dcd36b60f92 admin admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:03:36.959039 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 1210/2405] 10.222.0.30 () {60 vars in 1143 bytes} [Mon May 23 13:03:36 2022] POST /identity/v3/users => generated 283 bytes in 137 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) May 23 13:03:36.963058 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-568b9104-9e45-41fa-ac6a-62782a9d94bd admin admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:03:36.974712 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-568b9104-9e45-41fa-ac6a-62782a9d94bd admin admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:03:36.979364 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19d1014f-a239-4f5d-b222-9f9a5af0911c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:03:36.980030 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19d1014f-a239-4f5d-b222-9f9a5af0911c admin admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:03:36.980316 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19d1014f-a239-4f5d-b222-9f9a5af0911c admin admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:03:36.980593 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19d1014f-a239-4f5d-b222-9f9a5af0911c admin admin] PATH_INFO: `/v3/roles` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:03:36.981355 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19d1014f-a239-4f5d-b222-9f9a5af0911c admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:03:36.983694 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19d1014f-a239-4f5d-b222-9f9a5af0911c admin admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:03:36.994873 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 1196/2406] 10.222.0.30 () {58 vars in 1121 bytes} [Mon May 23 13:03:36 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) May 23 13:03:36.998710 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-086cad05-9c8d-43fb-bfd0-6dcd36b60f92 admin admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:03:37.008603 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-086cad05-9c8d-43fb-bfd0-6dcd36b60f92 admin admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:03:37.013705 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbde52a9-eaa6-41c0-9b7e-ee0d9694a0a3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:03:37.014518 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbde52a9-eaa6-41c0-9b7e-ee0d9694a0a3 admin admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:03:37.014813 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbde52a9-eaa6-41c0-9b7e-ee0d9694a0a3 admin admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:03:37.015091 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbde52a9-eaa6-41c0-9b7e-ee0d9694a0a3 admin admin] PATH_INFO: `/v3/roles` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:03:37.015807 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bbde52a9-eaa6-41c0-9b7e-ee0d9694a0a3 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:03:37.017395 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bbde52a9-eaa6-41c0-9b7e-ee0d9694a0a3 admin admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:03:37.027358 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 1211/2407] 10.222.0.30 () {58 vars in 1121 bytes} [Mon May 23 13:03:36 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) May 23 13:03:37.032385 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19d1014f-a239-4f5d-b222-9f9a5af0911c admin admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:03:37.037633 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=91,keystone:DELETE=7,keystone:INSERT=2 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 13:03:37.042805 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19d1014f-a239-4f5d-b222-9f9a5af0911c admin admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:03:37.048256 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3b30d7b-7f2f-469d-9107-315af943c829 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:03:37.049163 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3b30d7b-7f2f-469d-9107-315af943c829 admin admin] REQUEST_METHOD: `PUT` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:03:37.049438 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3b30d7b-7f2f-469d-9107-315af943c829 admin admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:03:37.049766 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3b30d7b-7f2f-469d-9107-315af943c829 admin admin] PATH_INFO: `/v3/projects/b544b79932064d969a04f4cc18eb082d/users/d21b6d37cec8480b8045219fe0f429b2/roles/80fb22507c9a4ff09d44d8f12f6fff1a` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:03:37.065763 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3b30d7b-7f2f-469d-9107-315af943c829 admin admin] RBAC: Authorizing `identity:create_grant(project_id=b544b79932064d969a04f4cc18eb082d, user_id=d21b6d37cec8480b8045219fe0f429b2, role_id=80fb22507c9a4ff09d44d8f12f6fff1a)` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:03:37.068509 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3b30d7b-7f2f-469d-9107-315af943c829 admin admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:03:37.113053 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 1197/2408] 10.222.0.30 () {60 vars in 1482 bytes} [Mon May 23 13:03:37 2022] PUT /identity/v3/projects/b544b79932064d969a04f4cc18eb082d/users/d21b6d37cec8480b8045219fe0f429b2/roles/80fb22507c9a4ff09d44d8f12f6fff1a => generated 0 bytes in 81 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) May 23 13:03:37.123639 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b92dd6cd-3c98-49cf-9af1-8573b889da72 None None] REQUEST_METHOD: `POST` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:03:37.123936 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b92dd6cd-3c98-49cf-9af1-8573b889da72 None None] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:03:37.124215 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b92dd6cd-3c98-49cf-9af1-8573b889da72 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:03:37.151241 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.auth.core [None req-b92dd6cd-3c98-49cf-9af1-8573b889da72 None None] MFA Rules not processed for user `d21b6d37cec8480b8045219fe0f429b2`. Rule list: `[]` (Enabled: `True`). {{(pid=92057) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 13:03:37.152433 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.fernet_utils [None req-b92dd6cd-3c98-49cf-9af1-8573b889da72 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92057) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:03:37.164036 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=92,keystone:DELETE=4,keystone:INSERT=4 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 13:03:37.216218 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 1212/2409] 10.222.0.30 () {58 vars in 957 bytes} [Mon May 23 13:03:37 2022] POST /identity/v3/auth/tokens => generated 3232 bytes in 94 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) May 23 13:03:37.227650 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3b30d7b-7f2f-469d-9107-315af943c829 admin admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:03:37.229322 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.fernet_utils [None req-b3b30d7b-7f2f-469d-9107-315af943c829 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92056) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:03:37.288454 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3b30d7b-7f2f-469d-9107-315af943c829 admin admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:03:37.292088 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65468363-94cb-4944-a910-2e520d9a172e service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1666097f8ed545c6b0e4809c93e4775a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9c389ae65aaa4542bb5a5c4fd0bacf5b', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:03:37.305919 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65468363-94cb-4944-a910-2e520d9a172e service cinder] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:03:37.305919 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65468363-94cb-4944-a910-2e520d9a172e service cinder] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:03:37.305919 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65468363-94cb-4944-a910-2e520d9a172e service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:03:37.305919 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65468363-94cb-4944-a910-2e520d9a172e service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:03:37.305919 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65468363-94cb-4944-a910-2e520d9a172e service cinder] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:03:37.312681 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 1198/2410] 10.222.0.30 () {60 vars in 1365 bytes} [Mon May 23 13:03:37 2022] GET /identity/v3/auth/tokens => generated 3232 bytes in 86 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) May 23 13:03:47.269819 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=21 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 13:03:47.307768 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=35,keystone:INSERT=1 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 13:03:54.326403 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b92dd6cd-3c98-49cf-9af1-8573b889da72 None None] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:03:54.329171 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.fernet_utils [None req-b92dd6cd-3c98-49cf-9af1-8573b889da72 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92057) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:03:54.407337 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b92dd6cd-3c98-49cf-9af1-8573b889da72 None None] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:03:54.414156 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d0cd495-ed15-4363-b439-f9b2c6295b31 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:03:54.414156 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d0cd495-ed15-4363-b439-f9b2c6295b31 admin admin] REQUEST_METHOD: `DELETE` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:03:54.414156 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d0cd495-ed15-4363-b439-f9b2c6295b31 admin admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:03:54.414156 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d0cd495-ed15-4363-b439-f9b2c6295b31 admin admin] PATH_INFO: `/v3/users/68c12740b64549fdab49b511d23ad01f` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:03:54.415665 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d0cd495-ed15-4363-b439-f9b2c6295b31 admin admin] RBAC: Authorizing `identity:delete_user(user_id=68c12740b64549fdab49b511d23ad01f)` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:03:54.417853 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d0cd495-ed15-4363-b439-f9b2c6295b31 admin admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:03:54.638382 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [None req-6d0cd495-ed15-4363-b439-f9b2c6295b31 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '68c12740b64549fdab49b511d23ad01f'} {{(pid=92057) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:03:54.721563 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [None req-6d0cd495-ed15-4363-b439-f9b2c6295b31 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '68c12740b64549fdab49b511d23ad01f'} {{(pid=92057) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:03:54.729571 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [None req-6d0cd495-ed15-4363-b439-f9b2c6295b31 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '68c12740b64549fdab49b511d23ad01f'} {{(pid=92057) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:03:54.730326 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [None req-6d0cd495-ed15-4363-b439-f9b2c6295b31 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '68c12740b64549fdab49b511d23ad01f'} {{(pid=92057) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:03:54.743924 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [None req-6d0cd495-ed15-4363-b439-f9b2c6295b31 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '68c12740b64549fdab49b511d23ad01f'} {{(pid=92057) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:03:54.753006 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 1213/2411] 10.222.0.30 () {58 vars in 1223 bytes} [Mon May 23 13:03:54 2022] DELETE /identity/v3/users/68c12740b64549fdab49b511d23ad01f => generated 0 bytes in 426 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) May 23 13:03:54.755281 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49449d8f-7882-4500-ac75-c4c39270b4ab None None] REQUEST_METHOD: `POST` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:03:54.756106 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49449d8f-7882-4500-ac75-c4c39270b4ab None None] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:03:54.756389 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49449d8f-7882-4500-ac75-c4c39270b4ab None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:03:54.775188 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.auth.core [None req-49449d8f-7882-4500-ac75-c4c39270b4ab None None] MFA Rules not processed for user `0f7119f8fa5c4445977671140ded91a4`. Rule list: `[]` (Enabled: `True`). {{(pid=92056) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 13:03:54.776404 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.fernet_utils [None req-49449d8f-7882-4500-ac75-c4c39270b4ab None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92056) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:03:54.837684 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 1199/2412] 10.222.0.30 () {58 vars in 957 bytes} [Mon May 23 13:03:54 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 84 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) May 23 13:03:54.848103 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d0cd495-ed15-4363-b439-f9b2c6295b31 admin admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:03:54.849610 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.fernet_utils [None req-6d0cd495-ed15-4363-b439-f9b2c6295b31 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92057) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:03:54.909479 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d0cd495-ed15-4363-b439-f9b2c6295b31 admin admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:03:54.913049 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0624c2ba-ede6-48f1-8edc-d774a535157e service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2eb83ca38e79449095f3a2f947bbb06a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9c389ae65aaa4542bb5a5c4fd0bacf5b', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:03:54.913686 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0624c2ba-ede6-48f1-8edc-d774a535157e service neutron] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:03:54.914013 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0624c2ba-ede6-48f1-8edc-d774a535157e service neutron] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:03:54.914292 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0624c2ba-ede6-48f1-8edc-d774a535157e service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:03:54.923402 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0624c2ba-ede6-48f1-8edc-d774a535157e service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:03:54.924885 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0624c2ba-ede6-48f1-8edc-d774a535157e service neutron] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:03:54.934277 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 1214/2413] 10.222.0.30 () {60 vars in 1365 bytes} [Mon May 23 13:03:54 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 87 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) May 23 13:03:55.382694 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49449d8f-7882-4500-ac75-c4c39270b4ab None None] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:03:55.384478 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.fernet_utils [None req-49449d8f-7882-4500-ac75-c4c39270b4ab None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92056) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:03:55.397855 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49449d8f-7882-4500-ac75-c4c39270b4ab None None] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:03:55.411217 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff386436-c7fa-4042-95e1-4971139064af admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:03:55.412097 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff386436-c7fa-4042-95e1-4971139064af admin admin] REQUEST_METHOD: `DELETE` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:03:55.412383 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff386436-c7fa-4042-95e1-4971139064af admin admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:03:55.412713 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff386436-c7fa-4042-95e1-4971139064af admin admin] PATH_INFO: `/v3/projects/ef486840486f408bab70af5c07f3d683` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:03:55.413998 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff386436-c7fa-4042-95e1-4971139064af admin admin] RBAC: Authorizing `identity:delete_project(project_id=ef486840486f408bab70af5c07f3d683)` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:03:55.419422 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff386436-c7fa-4042-95e1-4971139064af admin admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:03:55.544794 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-ff386436-c7fa-4042-95e1-4971139064af admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'ef486840486f408bab70af5c07f3d683', 'request_id': 'req-ff386436-c7fa-4042-95e1-4971139064af'} {{(pid=92056) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:03:55.610668 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-ff386436-c7fa-4042-95e1-4971139064af admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'ef486840486f408bab70af5c07f3d683', 'request_id': 'req-ff386436-c7fa-4042-95e1-4971139064af'} {{(pid=92056) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:03:55.617290 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-ff386436-c7fa-4042-95e1-4971139064af admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'ef486840486f408bab70af5c07f3d683', 'request_id': 'req-ff386436-c7fa-4042-95e1-4971139064af'} {{(pid=92056) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:03:55.628489 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-ff386436-c7fa-4042-95e1-4971139064af admin admin] The token cache is being invalidate because project ef486840486f408bab70af5c07f3d683 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=92056) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} May 23 13:03:55.628774 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-ff386436-c7fa-4042-95e1-4971139064af admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=92056) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:03:55.629429 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-ff386436-c7fa-4042-95e1-4971139064af admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=92056) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:03:55.631004 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 1200/2414] 10.222.0.30 () {58 vars in 1232 bytes} [Mon May 23 13:03:55 2022] DELETE /identity/v3/projects/ef486840486f408bab70af5c07f3d683 => generated 0 bytes in 249 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) May 23 13:04:04.930235 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=79,keystone:DELETE=10,keystone:INSERT=1 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 13:04:05.625367 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=54,keystone:DELETE=7,keystone:INSERT=1 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 13:04:50.113004 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d48be1f2-1f84-481f-a848-d0f732a76d9c None None] REQUEST_METHOD: `POST` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:04:50.113501 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d48be1f2-1f84-481f-a848-d0f732a76d9c None None] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:04:50.113703 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d48be1f2-1f84-481f-a848-d0f732a76d9c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:04:50.131168 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.auth.core [None req-d48be1f2-1f84-481f-a848-d0f732a76d9c None None] MFA Rules not processed for user `5c6147fed25a4212b6175aacc460943d`. Rule list: `[]` (Enabled: `True`). {{(pid=92057) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 13:04:50.132567 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.fernet_utils [None req-d48be1f2-1f84-481f-a848-d0f732a76d9c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92057) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:04:50.189208 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 1215/2415] 10.222.0.30 () {58 vars in 957 bytes} [Mon May 23 13:04:50 2022] POST /identity/v3/auth/tokens => generated 3248 bytes in 78 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) May 23 13:04:57.950497 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff386436-c7fa-4042-95e1-4971139064af admin admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:04:57.951988 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.fernet_utils [None req-ff386436-c7fa-4042-95e1-4971139064af admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92056) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:04:58.009116 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff386436-c7fa-4042-95e1-4971139064af admin admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:04:58.012750 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b11a96d3-2bf0-4958-8add-0f8a91b88b4c service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2eb83ca38e79449095f3a2f947bbb06a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9c389ae65aaa4542bb5a5c4fd0bacf5b', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:04:58.013394 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b11a96d3-2bf0-4958-8add-0f8a91b88b4c service neutron] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:04:58.013680 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b11a96d3-2bf0-4958-8add-0f8a91b88b4c service neutron] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:04:58.013987 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b11a96d3-2bf0-4958-8add-0f8a91b88b4c service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:04:58.015746 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.fernet_utils [None req-b11a96d3-2bf0-4958-8add-0f8a91b88b4c service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92056) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:04:58.058416 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d48be1f2-1f84-481f-a848-d0f732a76d9c None None] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:04:58.059794 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.fernet_utils [None req-d48be1f2-1f84-481f-a848-d0f732a76d9c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92057) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:04:58.102640 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b11a96d3-2bf0-4958-8add-0f8a91b88b4c service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:04:58.104442 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b11a96d3-2bf0-4958-8add-0f8a91b88b4c service neutron] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:04:58.123978 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d48be1f2-1f84-481f-a848-d0f732a76d9c None None] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:04:58.125211 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 1201/2416] 10.222.0.30 () {60 vars in 1365 bytes} [Mon May 23 13:04:57 2022] GET /identity/v3/auth/tokens => generated 3362 bytes in 175 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) May 23 13:04:58.128713 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-531ffb8c-6995-4e50-b53c-5bb49923fdf9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:04:58.129574 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-531ffb8c-6995-4e50-b53c-5bb49923fdf9 admin admin] REQUEST_METHOD: `DELETE` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:04:58.129861 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-531ffb8c-6995-4e50-b53c-5bb49923fdf9 admin admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:04:58.130676 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-531ffb8c-6995-4e50-b53c-5bb49923fdf9 admin admin] PATH_INFO: `/v3/users/d21b6d37cec8480b8045219fe0f429b2` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:04:58.131875 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-531ffb8c-6995-4e50-b53c-5bb49923fdf9 admin admin] RBAC: Authorizing `identity:delete_user(user_id=d21b6d37cec8480b8045219fe0f429b2)` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:04:58.133829 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-531ffb8c-6995-4e50-b53c-5bb49923fdf9 admin admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:04:58.321430 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [None req-531ffb8c-6995-4e50-b53c-5bb49923fdf9 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'd21b6d37cec8480b8045219fe0f429b2'} {{(pid=92057) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:04:58.384097 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [None req-531ffb8c-6995-4e50-b53c-5bb49923fdf9 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'd21b6d37cec8480b8045219fe0f429b2'} {{(pid=92057) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:04:58.391038 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [None req-531ffb8c-6995-4e50-b53c-5bb49923fdf9 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'd21b6d37cec8480b8045219fe0f429b2'} {{(pid=92057) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:04:58.391660 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [None req-531ffb8c-6995-4e50-b53c-5bb49923fdf9 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'd21b6d37cec8480b8045219fe0f429b2'} {{(pid=92057) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:04:58.404320 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [None req-531ffb8c-6995-4e50-b53c-5bb49923fdf9 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'd21b6d37cec8480b8045219fe0f429b2'} {{(pid=92057) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:04:58.410293 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 1216/2417] 10.222.0.30 () {58 vars in 1223 bytes} [Mon May 23 13:04:58 2022] DELETE /identity/v3/users/d21b6d37cec8480b8045219fe0f429b2 => generated 0 bytes in 353 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) May 23 13:04:58.415468 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9eb35211-5818-439d-a4e9-f5fe71ed0dda None None] REQUEST_METHOD: `POST` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:04:58.415753 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9eb35211-5818-439d-a4e9-f5fe71ed0dda None None] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:04:58.416029 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9eb35211-5818-439d-a4e9-f5fe71ed0dda None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:04:58.433141 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.auth.core [None req-9eb35211-5818-439d-a4e9-f5fe71ed0dda None None] MFA Rules not processed for user `0f7119f8fa5c4445977671140ded91a4`. Rule list: `[]` (Enabled: `True`). {{(pid=92056) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 13:04:58.434726 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.fernet_utils [None req-9eb35211-5818-439d-a4e9-f5fe71ed0dda None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92056) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:04:58.493048 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 1202/2418] 10.222.0.30 () {58 vars in 957 bytes} [Mon May 23 13:04:58 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 79 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) May 23 13:04:58.502497 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-531ffb8c-6995-4e50-b53c-5bb49923fdf9 admin admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:04:58.503876 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.fernet_utils [None req-531ffb8c-6995-4e50-b53c-5bb49923fdf9 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92057) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:04:58.531953 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 13:04:58.560851 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-531ffb8c-6995-4e50-b53c-5bb49923fdf9 admin admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:04:58.565196 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa19c4b4-3dea-4796-b557-ab46c0e0dc1b service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2eb83ca38e79449095f3a2f947bbb06a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9c389ae65aaa4542bb5a5c4fd0bacf5b', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:04:58.565916 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa19c4b4-3dea-4796-b557-ab46c0e0dc1b service neutron] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:04:58.566201 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa19c4b4-3dea-4796-b557-ab46c0e0dc1b service neutron] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:04:58.566473 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa19c4b4-3dea-4796-b557-ab46c0e0dc1b service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:04:58.576219 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa19c4b4-3dea-4796-b557-ab46c0e0dc1b service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:04:58.577646 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa19c4b4-3dea-4796-b557-ab46c0e0dc1b service neutron] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:04:58.587949 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 1217/2419] 10.222.0.30 () {60 vars in 1365 bytes} [Mon May 23 13:04:58 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 86 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) May 23 13:04:58.968993 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9eb35211-5818-439d-a4e9-f5fe71ed0dda None None] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:04:58.973667 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.fernet_utils [None req-9eb35211-5818-439d-a4e9-f5fe71ed0dda None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92056) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:04:58.983602 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9eb35211-5818-439d-a4e9-f5fe71ed0dda None None] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:04:58.990318 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9369b4df-b683-4b26-9036-93467ea790ac admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:04:58.991212 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9369b4df-b683-4b26-9036-93467ea790ac admin admin] REQUEST_METHOD: `DELETE` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:04:58.991491 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9369b4df-b683-4b26-9036-93467ea790ac admin admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:04:58.991764 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9369b4df-b683-4b26-9036-93467ea790ac admin admin] PATH_INFO: `/v3/projects/b544b79932064d969a04f4cc18eb082d` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:04:58.993148 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9369b4df-b683-4b26-9036-93467ea790ac admin admin] RBAC: Authorizing `identity:delete_project(project_id=b544b79932064d969a04f4cc18eb082d)` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:04:58.995206 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9369b4df-b683-4b26-9036-93467ea790ac admin admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:04:59.013415 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=100 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 13:04:59.101837 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-9369b4df-b683-4b26-9036-93467ea790ac admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'b544b79932064d969a04f4cc18eb082d', 'request_id': 'req-9369b4df-b683-4b26-9036-93467ea790ac'} {{(pid=92056) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:04:59.164580 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-9369b4df-b683-4b26-9036-93467ea790ac admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'b544b79932064d969a04f4cc18eb082d', 'request_id': 'req-9369b4df-b683-4b26-9036-93467ea790ac'} {{(pid=92056) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:04:59.170952 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-9369b4df-b683-4b26-9036-93467ea790ac admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'b544b79932064d969a04f4cc18eb082d', 'request_id': 'req-9369b4df-b683-4b26-9036-93467ea790ac'} {{(pid=92056) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:04:59.181850 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-9369b4df-b683-4b26-9036-93467ea790ac admin admin] The token cache is being invalidate because project b544b79932064d969a04f4cc18eb082d was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=92056) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} May 23 13:04:59.182168 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-9369b4df-b683-4b26-9036-93467ea790ac admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=92056) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:04:59.182796 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-9369b4df-b683-4b26-9036-93467ea790ac admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=92056) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:04:59.184340 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 1203/2420] 10.222.0.30 () {58 vars in 1232 bytes} [Mon May 23 13:04:58 2022] DELETE /identity/v3/projects/b544b79932064d969a04f4cc18eb082d => generated 0 bytes in 216 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) May 23 13:04:59.297872 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa19c4b4-3dea-4796-b557-ab46c0e0dc1b service neutron] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:04:59.306909 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa19c4b4-3dea-4796-b557-ab46c0e0dc1b service neutron] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:04:59.359418 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d2d9c30-3005-4abb-a20d-7d62e5203a22 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2eb83ca38e79449095f3a2f947bbb06a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9c389ae65aaa4542bb5a5c4fd0bacf5b', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:04:59.360066 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d2d9c30-3005-4abb-a20d-7d62e5203a22 service neutron] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:04:59.360352 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d2d9c30-3005-4abb-a20d-7d62e5203a22 service neutron] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:04:59.360629 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d2d9c30-3005-4abb-a20d-7d62e5203a22 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:04:59.362439 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.fernet_utils [None req-3d2d9c30-3005-4abb-a20d-7d62e5203a22 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92057) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:04:59.434383 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d2d9c30-3005-4abb-a20d-7d62e5203a22 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:04:59.435834 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d2d9c30-3005-4abb-a20d-7d62e5203a22 service neutron] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:04:59.447300 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 1218/2421] 10.222.0.30 () {60 vars in 1365 bytes} [Mon May 23 13:04:59 2022] GET /identity/v3/auth/tokens => generated 3362 bytes in 150 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) May 23 13:05:09.178885 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=17,keystone:DELETE=7,keystone:INSERT=1 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 13:05:09.441904 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=69 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 13:05:37.124375 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9369b4df-b683-4b26-9036-93467ea790ac admin admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:05:37.126276 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.fernet_utils [None req-9369b4df-b683-4b26-9036-93467ea790ac admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92056) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:05:37.142122 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9369b4df-b683-4b26-9036-93467ea790ac admin admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:05:37.148179 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0481868f-0588-4837-8bc6-d7ae0d0be4cf service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cbe9931d94d44cc7ace0e83a7e5015fc', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9c389ae65aaa4542bb5a5c4fd0bacf5b', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'member', 'reader', 'service', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:05:37.149397 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0481868f-0588-4837-8bc6-d7ae0d0be4cf service nova] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:05:37.149686 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0481868f-0588-4837-8bc6-d7ae0d0be4cf service nova] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:05:37.150080 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0481868f-0588-4837-8bc6-d7ae0d0be4cf service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:05:37.151987 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.fernet_utils [None req-0481868f-0588-4837-8bc6-d7ae0d0be4cf service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92056) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:05:37.163839 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0481868f-0588-4837-8bc6-d7ae0d0be4cf service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:05:37.166155 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0481868f-0588-4837-8bc6-d7ae0d0be4cf service nova] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:05:37.176436 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 1204/2422] 10.222.0.30 () {60 vars in 1365 bytes} [Mon May 23 13:05:37 2022] GET /identity/v3/auth/tokens => generated 3362 bytes in 53 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) May 23 13:05:47.172166 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=6 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 13:05:50.594882 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d2d9c30-3005-4abb-a20d-7d62e5203a22 service neutron] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:05:50.606786 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d2d9c30-3005-4abb-a20d-7d62e5203a22 service neutron] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:05:50.612329 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8394980-d3b9-4caa-942c-1ad3cef41d03 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cbe9931d94d44cc7ace0e83a7e5015fc', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9c389ae65aaa4542bb5a5c4fd0bacf5b', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'member', 'reader', 'service', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:05:50.612970 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8394980-d3b9-4caa-942c-1ad3cef41d03 service nova] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:05:50.619949 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8394980-d3b9-4caa-942c-1ad3cef41d03 service nova] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:05:50.619949 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8394980-d3b9-4caa-942c-1ad3cef41d03 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:05:50.619949 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.fernet_utils [None req-f8394980-d3b9-4caa-942c-1ad3cef41d03 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92057) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:05:50.627671 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8394980-d3b9-4caa-942c-1ad3cef41d03 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:05:50.629257 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8394980-d3b9-4caa-942c-1ad3cef41d03 service nova] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:05:50.640088 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 1219/2423] 10.222.0.30 () {60 vars in 1365 bytes} [Mon May 23 13:05:50 2022] GET /identity/v3/auth/tokens => generated 3362 bytes in 46 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) May 23 13:05:52.417520 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ca1ddf4-2a44-4add-965e-48b4afb4adfb None None] REQUEST_METHOD: `POST` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:05:52.418049 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ca1ddf4-2a44-4add-965e-48b4afb4adfb None None] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:05:52.418153 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ca1ddf4-2a44-4add-965e-48b4afb4adfb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:05:52.436884 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.auth.core [None req-3ca1ddf4-2a44-4add-965e-48b4afb4adfb None None] MFA Rules not processed for user `6d21a56126fd4ccd837c33ccc08ab609`. Rule list: `[]` (Enabled: `True`). {{(pid=92056) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 13:05:52.439858 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.fernet_utils [None req-3ca1ddf4-2a44-4add-965e-48b4afb4adfb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92056) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:05:52.494808 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 1205/2424] 10.222.0.30 () {58 vars in 1009 bytes} [Mon May 23 13:05:52 2022] POST /identity/v3/auth/tokens => generated 3185 bytes in 79 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) May 23 13:05:52.634375 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8394980-d3b9-4caa-942c-1ad3cef41d03 service nova] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:05:52.635973 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.fernet_utils [None req-f8394980-d3b9-4caa-942c-1ad3cef41d03 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92057) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:05:52.705548 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8394980-d3b9-4caa-942c-1ad3cef41d03 service nova] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:05:52.710754 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94445ba9-8a30-48df-b407-5d6a63ff02a2 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '138d91edb99f450e846765f9eef00d62', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9c389ae65aaa4542bb5a5c4fd0bacf5b', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'service', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:05:52.711380 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94445ba9-8a30-48df-b407-5d6a63ff02a2 service swift] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:05:52.711667 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94445ba9-8a30-48df-b407-5d6a63ff02a2 service swift] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:05:52.711948 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94445ba9-8a30-48df-b407-5d6a63ff02a2 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:05:52.720198 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94445ba9-8a30-48df-b407-5d6a63ff02a2 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:05:52.721795 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94445ba9-8a30-48df-b407-5d6a63ff02a2 service swift] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:05:52.730413 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 1220/2425] 10.222.0.30 () {60 vars in 1384 bytes} [Mon May 23 13:05:52 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 97 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) May 23 13:05:59.633463 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ca1ddf4-2a44-4add-965e-48b4afb4adfb None None] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:05:59.638621 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.fernet_utils [None req-3ca1ddf4-2a44-4add-965e-48b4afb4adfb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92056) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:05:59.655760 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ca1ddf4-2a44-4add-965e-48b4afb4adfb None None] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:05:59.661212 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d0b4798-4d81-468f-9c41-1423364d30f6 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2eb83ca38e79449095f3a2f947bbb06a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9c389ae65aaa4542bb5a5c4fd0bacf5b', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:05:59.661212 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d0b4798-4d81-468f-9c41-1423364d30f6 service neutron] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:05:59.661212 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d0b4798-4d81-468f-9c41-1423364d30f6 service neutron] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:05:59.661212 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d0b4798-4d81-468f-9c41-1423364d30f6 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:05:59.662723 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.fernet_utils [None req-9d0b4798-4d81-468f-9c41-1423364d30f6 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92056) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:05:59.673336 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d0b4798-4d81-468f-9c41-1423364d30f6 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:05:59.674839 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d0b4798-4d81-468f-9c41-1423364d30f6 service neutron] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:05:59.688102 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 1206/2426] 10.222.0.30 () {60 vars in 1365 bytes} [Mon May 23 13:05:59 2022] GET /identity/v3/auth/tokens => generated 3111 bytes in 55 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) May 23 13:06:02.726688 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=38 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 13:06:09.549839 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d8aa500-8ecb-4cc5-b1a1-b0907dd9323c None None] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:06:09.550527 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d8aa500-8ecb-4cc5-b1a1-b0907dd9323c None None] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:06:09.550527 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d8aa500-8ecb-4cc5-b1a1-b0907dd9323c None None] PATH_INFO: `/` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:06:09.552089 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 1221/2427] 10.222.0.30 () {54 vars in 921 bytes} [Mon May 23 13:06:09 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) May 23 13:06:09.557870 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-937fd9fb-a90b-44a9-9d77-7706bdb3fbe5 None None] REQUEST_METHOD: `POST` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:06:09.558417 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-937fd9fb-a90b-44a9-9d77-7706bdb3fbe5 None None] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:06:09.558509 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-937fd9fb-a90b-44a9-9d77-7706bdb3fbe5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:06:09.606996 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.auth.core [None req-937fd9fb-a90b-44a9-9d77-7706bdb3fbe5 None None] MFA Rules not processed for user `cbe9931d94d44cc7ace0e83a7e5015fc`. Rule list: `[]` (Enabled: `True`). {{(pid=92056) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 13:06:09.606996 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.fernet_utils [None req-937fd9fb-a90b-44a9-9d77-7706bdb3fbe5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92056) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:06:09.609059 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 1207/2428] 10.222.0.30 () {58 vars in 1018 bytes} [Mon May 23 13:06:09 2022] POST /identity/v3/auth/tokens => generated 3362 bytes in 52 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) May 23 13:06:09.620403 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d8aa500-8ecb-4cc5-b1a1-b0907dd9323c None None] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:06:09.622167 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.fernet_utils [None req-3d8aa500-8ecb-4cc5-b1a1-b0907dd9323c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92057) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:06:09.704241 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d8aa500-8ecb-4cc5-b1a1-b0907dd9323c None None] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:06:09.707770 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15f03677-6969-4043-8fcc-a51532d411e2 service glance] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '5cea0feed4254ec085854986f99acf30', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9c389ae65aaa4542bb5a5c4fd0bacf5b', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:06:09.708404 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15f03677-6969-4043-8fcc-a51532d411e2 service glance] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:06:09.708682 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15f03677-6969-4043-8fcc-a51532d411e2 service glance] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:06:09.708954 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15f03677-6969-4043-8fcc-a51532d411e2 service glance] PATH_INFO: `/v3/auth/tokens` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:06:09.718942 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15f03677-6969-4043-8fcc-a51532d411e2 service glance] RBAC: Authorizing `identity:validate_token()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:06:09.720402 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15f03677-6969-4043-8fcc-a51532d411e2 service glance] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:06:09.730795 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 1222/2429] 10.222.0.30 () {60 vars in 1365 bytes} [Mon May 23 13:06:09 2022] GET /identity/v3/auth/tokens => generated 3362 bytes in 111 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) May 23 13:06:09.960236 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb72ea71-33a8-40ce-b790-fee10b4ee0c1 None None] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:06:09.960622 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb72ea71-33a8-40ce-b790-fee10b4ee0c1 None None] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:06:09.960911 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb72ea71-33a8-40ce-b790-fee10b4ee0c1 None None] PATH_INFO: `/` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:06:09.962578 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 1208/2430] 10.222.0.30 () {54 vars in 921 bytes} [Mon May 23 13:06:09 2022] GET /identity => generated 270 bytes in 12 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) May 23 13:06:09.969028 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19cadf6d-720f-4d7b-874d-a4a43dcc2756 None None] REQUEST_METHOD: `POST` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:06:09.969425 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19cadf6d-720f-4d7b-874d-a4a43dcc2756 None None] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:06:09.969781 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19cadf6d-720f-4d7b-874d-a4a43dcc2756 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:06:09.988522 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.auth.core [None req-19cadf6d-720f-4d7b-874d-a4a43dcc2756 None None] MFA Rules not processed for user `2eb83ca38e79449095f3a2f947bbb06a`. Rule list: `[]` (Enabled: `True`). {{(pid=92057) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 13:06:09.989730 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.fernet_utils [None req-19cadf6d-720f-4d7b-874d-a4a43dcc2756 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92057) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:06:09.996260 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 1223/2431] 10.222.0.30 () {58 vars in 1018 bytes} [Mon May 23 13:06:09 2022] POST /identity/v3/auth/tokens => generated 3111 bytes in 30 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) May 23 13:06:10.009226 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb72ea71-33a8-40ce-b790-fee10b4ee0c1 None None] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:06:10.010943 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.fernet_utils [None req-fb72ea71-33a8-40ce-b790-fee10b4ee0c1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92056) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:06:10.024331 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb72ea71-33a8-40ce-b790-fee10b4ee0c1 None None] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:06:10.027919 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d47a72d-fbe1-400e-8c0a-30e6500c13db service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2eb83ca38e79449095f3a2f947bbb06a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9c389ae65aaa4542bb5a5c4fd0bacf5b', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:06:10.028557 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d47a72d-fbe1-400e-8c0a-30e6500c13db service neutron] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:06:10.028845 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d47a72d-fbe1-400e-8c0a-30e6500c13db service neutron] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:06:10.029121 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d47a72d-fbe1-400e-8c0a-30e6500c13db service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:06:10.038419 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d47a72d-fbe1-400e-8c0a-30e6500c13db service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:06:10.041294 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d47a72d-fbe1-400e-8c0a-30e6500c13db service neutron] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:06:10.050218 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 1209/2432] 10.222.0.30 () {60 vars in 1365 bytes} [Mon May 23 13:06:10 2022] GET /identity/v3/auth/tokens => generated 3111 bytes in 41 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) May 23 13:06:11.144830 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-654df387-b46f-46fc-999b-ffaaba046738 None None] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:06:11.145253 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-654df387-b46f-46fc-999b-ffaaba046738 None None] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:06:11.145356 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-654df387-b46f-46fc-999b-ffaaba046738 None None] PATH_INFO: `/` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:06:11.147023 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 1224/2433] 10.222.0.30 () {54 vars in 921 bytes} [Mon May 23 13:06:11 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) May 23 13:06:11.152240 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b425ec0a-7fc4-4dcc-8b06-f18eefda76fc None None] REQUEST_METHOD: `POST` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:06:11.152663 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b425ec0a-7fc4-4dcc-8b06-f18eefda76fc None None] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:06:11.152800 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b425ec0a-7fc4-4dcc-8b06-f18eefda76fc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:06:11.171373 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.auth.core [None req-b425ec0a-7fc4-4dcc-8b06-f18eefda76fc None None] MFA Rules not processed for user `cbe9931d94d44cc7ace0e83a7e5015fc`. Rule list: `[]` (Enabled: `True`). {{(pid=92056) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 13:06:11.172543 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.fernet_utils [None req-b425ec0a-7fc4-4dcc-8b06-f18eefda76fc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92056) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:06:11.181562 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 1210/2434] 10.222.0.30 () {58 vars in 1018 bytes} [Mon May 23 13:06:11 2022] POST /identity/v3/auth/tokens => generated 3362 bytes in 30 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) May 23 13:06:11.191880 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-654df387-b46f-46fc-999b-ffaaba046738 None None] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:06:11.203170 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-654df387-b46f-46fc-999b-ffaaba046738 None None] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:06:11.206663 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-395647ee-713e-4363-90f5-7ec77eb9902a service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2eb83ca38e79449095f3a2f947bbb06a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9c389ae65aaa4542bb5a5c4fd0bacf5b', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:06:11.207283 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-395647ee-713e-4363-90f5-7ec77eb9902a service neutron] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:06:11.207564 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-395647ee-713e-4363-90f5-7ec77eb9902a service neutron] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:06:11.207838 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-395647ee-713e-4363-90f5-7ec77eb9902a service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:06:11.218463 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-395647ee-713e-4363-90f5-7ec77eb9902a service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:06:11.219892 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-395647ee-713e-4363-90f5-7ec77eb9902a service neutron] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:06:11.230419 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 1225/2435] 10.222.0.30 () {60 vars in 1365 bytes} [Mon May 23 13:06:11 2022] GET /identity/v3/auth/tokens => generated 3362 bytes in 39 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) May 23 13:06:11.294461 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b425ec0a-7fc4-4dcc-8b06-f18eefda76fc None None] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:06:11.295698 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.fernet_utils [None req-b425ec0a-7fc4-4dcc-8b06-f18eefda76fc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92056) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:06:11.365851 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b425ec0a-7fc4-4dcc-8b06-f18eefda76fc None None] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:06:11.370889 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce4d595d-9453-4ecf-af24-f9b8f9b1b874 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd5fb51f8d0fa46439cb29666970b885b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9c389ae65aaa4542bb5a5c4fd0bacf5b', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'service', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:06:11.371527 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce4d595d-9453-4ecf-af24-f9b8f9b1b874 service placement] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:06:11.371813 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce4d595d-9453-4ecf-af24-f9b8f9b1b874 service placement] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:06:11.372086 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce4d595d-9453-4ecf-af24-f9b8f9b1b874 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:06:11.374825 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.fernet_utils [None req-ce4d595d-9453-4ecf-af24-f9b8f9b1b874 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92056) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:06:11.385116 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce4d595d-9453-4ecf-af24-f9b8f9b1b874 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:06:11.386711 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce4d595d-9453-4ecf-af24-f9b8f9b1b874 service placement] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:06:11.397955 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 1211/2436] 10.222.0.30 () {60 vars in 1365 bytes} [Mon May 23 13:06:11 2022] GET /identity/v3/auth/tokens => generated 3298 bytes in 104 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) May 23 13:06:13.163586 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-395647ee-713e-4363-90f5-7ec77eb9902a service neutron] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:06:13.165072 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.fernet_utils [None req-395647ee-713e-4363-90f5-7ec77eb9902a service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92057) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:06:13.181440 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-395647ee-713e-4363-90f5-7ec77eb9902a service neutron] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:06:13.188310 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7fe2e781-ed98-4287-84b9-db452385d21d service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd5fb51f8d0fa46439cb29666970b885b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9c389ae65aaa4542bb5a5c4fd0bacf5b', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'service', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:06:13.188310 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fe2e781-ed98-4287-84b9-db452385d21d service placement] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:06:13.188538 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fe2e781-ed98-4287-84b9-db452385d21d service placement] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:06:13.188635 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fe2e781-ed98-4287-84b9-db452385d21d service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:06:13.190449 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.fernet_utils [None req-7fe2e781-ed98-4287-84b9-db452385d21d service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92057) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:06:13.204166 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7fe2e781-ed98-4287-84b9-db452385d21d service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:06:13.205768 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7fe2e781-ed98-4287-84b9-db452385d21d service placement] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:06:13.216175 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 1226/2437] 10.222.0.30 () {60 vars in 1365 bytes} [Mon May 23 13:06:13 2022] GET /identity/v3/auth/tokens => generated 3298 bytes in 53 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) May 23 13:06:13.319087 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce4d595d-9453-4ecf-af24-f9b8f9b1b874 service placement] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:06:13.320766 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.fernet_utils [None req-ce4d595d-9453-4ecf-af24-f9b8f9b1b874 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92056) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:06:13.367202 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=100 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 13:06:13.386345 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce4d595d-9453-4ecf-af24-f9b8f9b1b874 service placement] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:06:13.391074 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-479fad6e-bffb-43e1-99c6-b8ee734c65f0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:06:13.391927 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-479fad6e-bffb-43e1-99c6-b8ee734c65f0 admin admin] REQUEST_METHOD: `DELETE` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:06:13.392204 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-479fad6e-bffb-43e1-99c6-b8ee734c65f0 admin admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:06:13.392480 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-479fad6e-bffb-43e1-99c6-b8ee734c65f0 admin admin] PATH_INFO: `/v3/users/5c6147fed25a4212b6175aacc460943d` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:06:13.393642 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-479fad6e-bffb-43e1-99c6-b8ee734c65f0 admin admin] RBAC: Authorizing `identity:delete_user(user_id=5c6147fed25a4212b6175aacc460943d)` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:06:13.395573 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-479fad6e-bffb-43e1-99c6-b8ee734c65f0 admin admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:06:13.535125 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-479fad6e-bffb-43e1-99c6-b8ee734c65f0 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '5c6147fed25a4212b6175aacc460943d'} {{(pid=92056) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:06:13.535777 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-479fad6e-bffb-43e1-99c6-b8ee734c65f0 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '5c6147fed25a4212b6175aacc460943d'} {{(pid=92056) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:06:13.600993 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-479fad6e-bffb-43e1-99c6-b8ee734c65f0 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '5c6147fed25a4212b6175aacc460943d'} {{(pid=92056) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:06:13.613366 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-479fad6e-bffb-43e1-99c6-b8ee734c65f0 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '5c6147fed25a4212b6175aacc460943d'} {{(pid=92056) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:06:13.620650 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-479fad6e-bffb-43e1-99c6-b8ee734c65f0 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '5c6147fed25a4212b6175aacc460943d'} {{(pid=92056) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:06:13.630784 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 1212/2438] 10.222.0.30 () {58 vars in 1223 bytes} [Mon May 23 13:06:13 2022] DELETE /identity/v3/users/5c6147fed25a4212b6175aacc460943d => generated 0 bytes in 312 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) May 23 13:06:13.877719 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7fe2e781-ed98-4287-84b9-db452385d21d service placement] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:06:13.890137 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7fe2e781-ed98-4287-84b9-db452385d21d service placement] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:06:13.954847 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ed2c9f8-621d-4195-9e6a-9d281078a9e3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:06:13.955698 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ed2c9f8-621d-4195-9e6a-9d281078a9e3 admin admin] REQUEST_METHOD: `DELETE` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:06:13.955977 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ed2c9f8-621d-4195-9e6a-9d281078a9e3 admin admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:06:13.956247 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ed2c9f8-621d-4195-9e6a-9d281078a9e3 admin admin] PATH_INFO: `/v3/projects/ccd27933078b44b48323995e98b21318` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:06:13.957438 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ed2c9f8-621d-4195-9e6a-9d281078a9e3 admin admin] RBAC: Authorizing `identity:delete_project(project_id=ccd27933078b44b48323995e98b21318)` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:06:13.959374 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ed2c9f8-621d-4195-9e6a-9d281078a9e3 admin admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:06:14.058640 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [None req-3ed2c9f8-621d-4195-9e6a-9d281078a9e3 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'ccd27933078b44b48323995e98b21318', 'request_id': 'req-3ed2c9f8-621d-4195-9e6a-9d281078a9e3'} {{(pid=92057) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:06:14.060582 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=95,keystone:DELETE=5 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 13:06:14.064805 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [None req-3ed2c9f8-621d-4195-9e6a-9d281078a9e3 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'ccd27933078b44b48323995e98b21318', 'request_id': 'req-3ed2c9f8-621d-4195-9e6a-9d281078a9e3'} {{(pid=92057) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:06:14.158561 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [None req-3ed2c9f8-621d-4195-9e6a-9d281078a9e3 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'ccd27933078b44b48323995e98b21318', 'request_id': 'req-3ed2c9f8-621d-4195-9e6a-9d281078a9e3'} {{(pid=92057) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:06:14.169608 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [None req-3ed2c9f8-621d-4195-9e6a-9d281078a9e3 admin admin] The token cache is being invalidate because project ccd27933078b44b48323995e98b21318 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=92057) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} May 23 13:06:14.173801 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [None req-3ed2c9f8-621d-4195-9e6a-9d281078a9e3 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=92057) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:06:14.173801 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [None req-3ed2c9f8-621d-4195-9e6a-9d281078a9e3 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=92057) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:06:14.173801 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 1227/2439] 10.222.0.30 () {58 vars in 1232 bytes} [Mon May 23 13:06:13 2022] DELETE /identity/v3/projects/ccd27933078b44b48323995e98b21318 => generated 0 bytes in 296 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) May 23 13:06:15.927582 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-479fad6e-bffb-43e1-99c6-b8ee734c65f0 admin admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:06:15.929453 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.fernet_utils [None req-479fad6e-bffb-43e1-99c6-b8ee734c65f0 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92056) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:06:16.002853 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-479fad6e-bffb-43e1-99c6-b8ee734c65f0 admin admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:06:16.054482 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 13:06:16.065357 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5239acc-441f-4a9a-b4b9-4fc610e9c1f0 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cbe9931d94d44cc7ace0e83a7e5015fc', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9c389ae65aaa4542bb5a5c4fd0bacf5b', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'member', 'reader', 'service', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:06:16.066041 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5239acc-441f-4a9a-b4b9-4fc610e9c1f0 service nova] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:06:16.066327 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5239acc-441f-4a9a-b4b9-4fc610e9c1f0 service nova] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:06:16.066607 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5239acc-441f-4a9a-b4b9-4fc610e9c1f0 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:06:16.068386 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.fernet_utils [None req-b5239acc-441f-4a9a-b4b9-4fc610e9c1f0 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92056) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:06:16.129621 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5239acc-441f-4a9a-b4b9-4fc610e9c1f0 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:06:16.131334 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5239acc-441f-4a9a-b4b9-4fc610e9c1f0 service nova] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:06:16.141153 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 1213/2440] 10.222.0.30 () {60 vars in 1365 bytes} [Mon May 23 13:06:15 2022] GET /identity/v3/auth/tokens => generated 3258 bytes in 214 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) May 23 13:06:18.106519 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6d3c104-277b-4aa0-af0b-ab227bce5e43 None None] REQUEST_METHOD: `POST` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:06:18.106994 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6d3c104-277b-4aa0-af0b-ab227bce5e43 None None] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:06:18.107121 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6d3c104-277b-4aa0-af0b-ab227bce5e43 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:06:18.125846 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.auth.core [None req-b6d3c104-277b-4aa0-af0b-ab227bce5e43 None None] MFA Rules not processed for user `6d21a56126fd4ccd837c33ccc08ab609`. Rule list: `[]` (Enabled: `True`). {{(pid=92057) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 13:06:18.127081 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.fernet_utils [None req-b6d3c104-277b-4aa0-af0b-ab227bce5e43 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92057) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:06:18.183858 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 1228/2441] 10.222.0.30 () {58 vars in 1009 bytes} [Mon May 23 13:06:18 2022] POST /identity/v3/auth/tokens => generated 3185 bytes in 78 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) May 23 13:06:18.196851 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5239acc-441f-4a9a-b4b9-4fc610e9c1f0 service nova] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:06:18.198287 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.fernet_utils [None req-b5239acc-441f-4a9a-b4b9-4fc610e9c1f0 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92056) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:06:18.266752 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5239acc-441f-4a9a-b4b9-4fc610e9c1f0 service nova] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:06:18.271845 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b888c274-dc73-41f5-bc6e-7859653e3f13 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '138d91edb99f450e846765f9eef00d62', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9c389ae65aaa4542bb5a5c4fd0bacf5b', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'service', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:06:18.272483 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b888c274-dc73-41f5-bc6e-7859653e3f13 service swift] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:06:18.272764 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b888c274-dc73-41f5-bc6e-7859653e3f13 service swift] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:06:18.275454 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b888c274-dc73-41f5-bc6e-7859653e3f13 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:06:18.281662 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b888c274-dc73-41f5-bc6e-7859653e3f13 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:06:18.283306 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b888c274-dc73-41f5-bc6e-7859653e3f13 service swift] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:06:18.291452 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 1214/2442] 10.222.0.30 () {60 vars in 1384 bytes} [Mon May 23 13:06:18 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 95 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) May 23 13:06:27.975883 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6d3c104-277b-4aa0-af0b-ab227bce5e43 None None] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:06:27.977211 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.fernet_utils [None req-b6d3c104-277b-4aa0-af0b-ab227bce5e43 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92057) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:06:27.993906 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6d3c104-277b-4aa0-af0b-ab227bce5e43 None None] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:06:27.999681 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3bc166d-2c75-40d2-b496-775bca633925 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cbe9931d94d44cc7ace0e83a7e5015fc', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9c389ae65aaa4542bb5a5c4fd0bacf5b', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'member', 'reader', 'service', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:06:28.000407 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3bc166d-2c75-40d2-b496-775bca633925 service nova] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:06:28.000768 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3bc166d-2c75-40d2-b496-775bca633925 service nova] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:06:28.001107 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3bc166d-2c75-40d2-b496-775bca633925 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:06:28.002955 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.fernet_utils [None req-a3bc166d-2c75-40d2-b496-775bca633925 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92057) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:06:28.072460 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a3bc166d-2c75-40d2-b496-775bca633925 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:06:28.077594 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a3bc166d-2c75-40d2-b496-775bca633925 service nova] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:06:28.085662 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 1229/2443] 10.222.0.30 () {60 vars in 1365 bytes} [Mon May 23 13:06:27 2022] GET /identity/v3/auth/tokens => generated 3252 bytes in 110 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) May 23 13:06:28.288201 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=61 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 13:06:38.081661 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=59,keystone:DELETE=2,keystone:INSERT=1 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 13:06:38.392676 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b888c274-dc73-41f5-bc6e-7859653e3f13 service swift] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:06:38.394336 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.fernet_utils [None req-b888c274-dc73-41f5-bc6e-7859653e3f13 service swift] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92056) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:06:38.410622 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b888c274-dc73-41f5-bc6e-7859653e3f13 service swift] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:06:38.416018 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d20d77f-a00b-4d43-a66d-8bba4e3f0c19 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cbe9931d94d44cc7ace0e83a7e5015fc', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9c389ae65aaa4542bb5a5c4fd0bacf5b', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'member', 'reader', 'service', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:06:38.416649 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d20d77f-a00b-4d43-a66d-8bba4e3f0c19 service nova] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:06:38.416934 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d20d77f-a00b-4d43-a66d-8bba4e3f0c19 service nova] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:06:38.417208 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d20d77f-a00b-4d43-a66d-8bba4e3f0c19 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:06:38.419097 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.fernet_utils [None req-3d20d77f-a00b-4d43-a66d-8bba4e3f0c19 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92056) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:06:38.477665 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d20d77f-a00b-4d43-a66d-8bba4e3f0c19 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:06:38.479347 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d20d77f-a00b-4d43-a66d-8bba4e3f0c19 service nova] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:06:38.488766 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 1215/2444] 10.222.0.30 () {60 vars in 1365 bytes} [Mon May 23 13:06:38 2022] GET /identity/v3/auth/tokens => generated 3254 bytes in 96 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) May 23 13:06:47.697608 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bb5854d-fe96-4781-9c8a-fb2a580441bb None None] REQUEST_METHOD: `POST` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:06:47.698055 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bb5854d-fe96-4781-9c8a-fb2a580441bb None None] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:06:47.698202 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bb5854d-fe96-4781-9c8a-fb2a580441bb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:06:47.716192 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.auth.core [None req-1bb5854d-fe96-4781-9c8a-fb2a580441bb None None] MFA Rules not processed for user `6d21a56126fd4ccd837c33ccc08ab609`. Rule list: `[]` (Enabled: `True`). {{(pid=92057) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 13:06:47.717453 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.fernet_utils [None req-1bb5854d-fe96-4781-9c8a-fb2a580441bb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92057) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:06:47.726505 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 1230/2445] 10.222.0.30 () {58 vars in 967 bytes} [Mon May 23 13:06:47 2022] POST /identity/v3/auth/tokens => generated 3185 bytes in 30 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) May 23 13:06:47.739887 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d20d77f-a00b-4d43-a66d-8bba4e3f0c19 service nova] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:06:47.751699 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d20d77f-a00b-4d43-a66d-8bba4e3f0c19 service nova] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:06:47.756944 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2917db37-7145-479b-a5c9-87439fe75965 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '138d91edb99f450e846765f9eef00d62', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9c389ae65aaa4542bb5a5c4fd0bacf5b', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'service', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:06:47.757581 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2917db37-7145-479b-a5c9-87439fe75965 service swift] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:06:47.757860 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2917db37-7145-479b-a5c9-87439fe75965 service swift] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:06:47.758162 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2917db37-7145-479b-a5c9-87439fe75965 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:06:47.767156 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2917db37-7145-479b-a5c9-87439fe75965 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:06:47.768796 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2917db37-7145-479b-a5c9-87439fe75965 service swift] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:06:47.777536 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 1216/2446] 10.222.0.30 () {60 vars in 1384 bytes} [Mon May 23 13:06:47 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 38 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) May 23 13:06:48.821223 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1bb5854d-fe96-4781-9c8a-fb2a580441bb None None] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:06:48.834928 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1bb5854d-fe96-4781-9c8a-fb2a580441bb None None] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:06:48.840860 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2819e2da-419e-4fd8-99de-b4ae2f7a6790 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cbe9931d94d44cc7ace0e83a7e5015fc', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9c389ae65aaa4542bb5a5c4fd0bacf5b', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'member', 'reader', 'service', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:06:48.841671 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2819e2da-419e-4fd8-99de-b4ae2f7a6790 service nova] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:06:48.842013 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2819e2da-419e-4fd8-99de-b4ae2f7a6790 service nova] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:06:48.842290 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2819e2da-419e-4fd8-99de-b4ae2f7a6790 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:06:48.844208 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.fernet_utils [None req-2819e2da-419e-4fd8-99de-b4ae2f7a6790 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92057) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:06:48.916880 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2819e2da-419e-4fd8-99de-b4ae2f7a6790 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:06:48.922551 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2819e2da-419e-4fd8-99de-b4ae2f7a6790 service nova] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:06:48.932763 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 1231/2447] 10.222.0.30 () {60 vars in 1365 bytes} [Mon May 23 13:06:48 2022] GET /identity/v3/auth/tokens => generated 3246 bytes in 112 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) May 23 13:06:52.598646 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf711e20-e824-4f79-80cf-674a01c611b1 None None] REQUEST_METHOD: `POST` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:06:52.599170 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf711e20-e824-4f79-80cf-674a01c611b1 None None] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:06:52.599268 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf711e20-e824-4f79-80cf-674a01c611b1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:06:52.620401 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.auth.core [None req-cf711e20-e824-4f79-80cf-674a01c611b1 None None] MFA Rules not processed for user `257251830f5148e5b9e435c2c13828e8`. Rule list: `[]` (Enabled: `True`). {{(pid=92056) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 13:06:52.621613 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.fernet_utils [None req-cf711e20-e824-4f79-80cf-674a01c611b1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92056) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:06:52.634774 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 1217/2448] 10.222.0.30 () {58 vars in 957 bytes} [Mon May 23 13:06:52 2022] POST /identity/v3/auth/tokens => generated 3252 bytes in 37 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) May 23 13:06:52.954878 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2819e2da-419e-4fd8-99de-b4ae2f7a6790 service nova] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:06:52.956464 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.fernet_utils [None req-2819e2da-419e-4fd8-99de-b4ae2f7a6790 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92057) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:06:53.038847 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2819e2da-419e-4fd8-99de-b4ae2f7a6790 service nova] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:06:53.042423 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-326493a9-695e-4fca-88e4-e6669386722b service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2eb83ca38e79449095f3a2f947bbb06a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9c389ae65aaa4542bb5a5c4fd0bacf5b', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:06:53.043060 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-326493a9-695e-4fca-88e4-e6669386722b service neutron] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:06:53.043342 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-326493a9-695e-4fca-88e4-e6669386722b service neutron] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:06:53.043615 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-326493a9-695e-4fca-88e4-e6669386722b service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:06:53.045391 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.fernet_utils [None req-326493a9-695e-4fca-88e4-e6669386722b service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92057) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:06:53.055908 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-326493a9-695e-4fca-88e4-e6669386722b service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:06:53.057371 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-326493a9-695e-4fca-88e4-e6669386722b service neutron] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:06:53.067003 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 1232/2449] 10.222.0.30 () {60 vars in 1365 bytes} [Mon May 23 13:06:52 2022] GET /identity/v3/auth/tokens => generated 3111 bytes in 113 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) May 23 13:06:54.120658 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad6e3ec0-d544-47a1-b927-f6dbdd7d3025 None None] REQUEST_METHOD: `POST` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:06:54.121160 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad6e3ec0-d544-47a1-b927-f6dbdd7d3025 None None] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:06:54.121263 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad6e3ec0-d544-47a1-b927-f6dbdd7d3025 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:06:54.139077 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.auth.core [None req-ad6e3ec0-d544-47a1-b927-f6dbdd7d3025 None None] MFA Rules not processed for user `c6a6096760734ac49ce2f26c6065b701`. Rule list: `[]` (Enabled: `True`). {{(pid=92056) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 13:06:54.140810 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.fernet_utils [None req-ad6e3ec0-d544-47a1-b927-f6dbdd7d3025 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92056) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:06:54.147911 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 1218/2450] 10.222.0.30 () {58 vars in 957 bytes} [Mon May 23 13:06:54 2022] POST /identity/v3/auth/tokens => generated 3246 bytes in 28 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) May 23 13:06:54.465459 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-326493a9-695e-4fca-88e4-e6669386722b service neutron] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:06:54.474363 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-326493a9-695e-4fca-88e4-e6669386722b service neutron] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:06:54.480129 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb86bb9e-aae3-4e07-9f0c-cd0e9a6f5970 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2eb83ca38e79449095f3a2f947bbb06a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9c389ae65aaa4542bb5a5c4fd0bacf5b', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:06:54.480804 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb86bb9e-aae3-4e07-9f0c-cd0e9a6f5970 service neutron] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:06:54.481086 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb86bb9e-aae3-4e07-9f0c-cd0e9a6f5970 service neutron] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:06:54.481366 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb86bb9e-aae3-4e07-9f0c-cd0e9a6f5970 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:06:54.483139 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.fernet_utils [None req-fb86bb9e-aae3-4e07-9f0c-cd0e9a6f5970 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92057) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:06:54.493079 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb86bb9e-aae3-4e07-9f0c-cd0e9a6f5970 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:06:54.494910 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb86bb9e-aae3-4e07-9f0c-cd0e9a6f5970 service neutron] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:06:54.504479 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 1233/2451] 10.222.0.30 () {60 vars in 1365 bytes} [Mon May 23 13:06:54 2022] GET /identity/v3/auth/tokens => generated 3111 bytes in 40 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) May 23 13:06:58.363466 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a6c48f1-5f60-49d7-8355-045145e73450 None None] REQUEST_METHOD: `POST` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:06:58.363912 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a6c48f1-5f60-49d7-8355-045145e73450 None None] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:06:58.363912 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a6c48f1-5f60-49d7-8355-045145e73450 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:06:58.382649 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.auth.core [None req-6a6c48f1-5f60-49d7-8355-045145e73450 None None] MFA Rules not processed for user `b18e5f1b24634096995068e41d94cad6`. Rule list: `[]` (Enabled: `True`). {{(pid=92056) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 13:06:58.383913 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.fernet_utils [None req-6a6c48f1-5f60-49d7-8355-045145e73450 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92056) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:06:58.391245 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 1219/2452] 10.222.0.30 () {58 vars in 957 bytes} [Mon May 23 13:06:58 2022] POST /identity/v3/auth/tokens => generated 3258 bytes in 29 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) May 23 13:07:04.500117 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=63 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 13:07:05.974372 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb86bb9e-aae3-4e07-9f0c-cd0e9a6f5970 service neutron] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:07:05.989860 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb86bb9e-aae3-4e07-9f0c-cd0e9a6f5970 service neutron] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:07:05.996064 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b2050d2-e120-4c77-97f9-e7f6475ca15b service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cbe9931d94d44cc7ace0e83a7e5015fc', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9c389ae65aaa4542bb5a5c4fd0bacf5b', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'member', 'reader', 'service', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:07:05.996493 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b2050d2-e120-4c77-97f9-e7f6475ca15b service nova] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:07:05.996785 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b2050d2-e120-4c77-97f9-e7f6475ca15b service nova] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:07:05.997366 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b2050d2-e120-4c77-97f9-e7f6475ca15b service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:07:05.999739 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.fernet_utils [None req-9b2050d2-e120-4c77-97f9-e7f6475ca15b service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92057) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:07:06.015962 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b2050d2-e120-4c77-97f9-e7f6475ca15b service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:07:06.017714 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b2050d2-e120-4c77-97f9-e7f6475ca15b service nova] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:07:06.029312 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 1234/2453] 10.222.0.30 () {60 vars in 1365 bytes} [Mon May 23 13:07:05 2022] GET /identity/v3/auth/tokens => generated 3362 bytes in 55 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) May 23 13:07:08.377190 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=43 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 13:07:16.025073 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=6 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 13:08:00.092850 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57f541c6-2a59-4aff-805f-7033605ff4c7 None None] REQUEST_METHOD: `POST` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:08:00.093391 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57f541c6-2a59-4aff-805f-7033605ff4c7 None None] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:08:00.093391 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57f541c6-2a59-4aff-805f-7033605ff4c7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:08:00.111771 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.auth.core [None req-57f541c6-2a59-4aff-805f-7033605ff4c7 None None] MFA Rules not processed for user `f78ab5496c1e4008b83fa3866601c06c`. Rule list: `[]` (Enabled: `True`). {{(pid=92056) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 13:08:00.112963 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.fernet_utils [None req-57f541c6-2a59-4aff-805f-7033605ff4c7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92056) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:08:00.122009 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 1220/2454] 10.222.0.30 () {58 vars in 957 bytes} [Mon May 23 13:08:00 2022] POST /identity/v3/auth/tokens => generated 3254 bytes in 30 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) May 23 13:08:10.106436 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=3 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 13:08:26.946798 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b2050d2-e120-4c77-97f9-e7f6475ca15b service nova] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:08:26.948310 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.fernet_utils [None req-9b2050d2-e120-4c77-97f9-e7f6475ca15b service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92057) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:08:27.016566 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b2050d2-e120-4c77-97f9-e7f6475ca15b service nova] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:08:27.021416 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15d2f8a7-80bb-4384-a806-42ca02cc4255 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd5fb51f8d0fa46439cb29666970b885b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9c389ae65aaa4542bb5a5c4fd0bacf5b', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'service', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:08:27.022087 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15d2f8a7-80bb-4384-a806-42ca02cc4255 service placement] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:08:27.022626 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15d2f8a7-80bb-4384-a806-42ca02cc4255 service placement] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:08:27.022908 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15d2f8a7-80bb-4384-a806-42ca02cc4255 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:08:27.024637 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.fernet_utils [None req-15d2f8a7-80bb-4384-a806-42ca02cc4255 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92057) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:08:27.036309 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15d2f8a7-80bb-4384-a806-42ca02cc4255 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:08:27.037914 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15d2f8a7-80bb-4384-a806-42ca02cc4255 service placement] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:08:27.047964 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 1235/2455] 10.222.0.30 () {60 vars in 1365 bytes} [Mon May 23 13:08:26 2022] GET /identity/v3/auth/tokens => generated 3298 bytes in 102 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) May 23 13:08:32.397366 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57f541c6-2a59-4aff-805f-7033605ff4c7 None None] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:08:32.408234 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57f541c6-2a59-4aff-805f-7033605ff4c7 None None] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:08:32.412098 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4f05099-1dde-46bb-8c50-493e09bd2269 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2eb83ca38e79449095f3a2f947bbb06a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9c389ae65aaa4542bb5a5c4fd0bacf5b', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:08:32.412742 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4f05099-1dde-46bb-8c50-493e09bd2269 service neutron] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:08:32.413021 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4f05099-1dde-46bb-8c50-493e09bd2269 service neutron] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:08:32.413293 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4f05099-1dde-46bb-8c50-493e09bd2269 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:08:32.415124 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.fernet_utils [None req-e4f05099-1dde-46bb-8c50-493e09bd2269 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92056) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:08:32.477521 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4f05099-1dde-46bb-8c50-493e09bd2269 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:08:32.478992 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4f05099-1dde-46bb-8c50-493e09bd2269 service neutron] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:08:32.488693 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 1221/2456] 10.222.0.30 () {60 vars in 1365 bytes} [Mon May 23 13:08:32 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 92 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) May 23 13:08:36.262977 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15d2f8a7-80bb-4384-a806-42ca02cc4255 service placement] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:08:36.265128 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.fernet_utils [None req-15d2f8a7-80bb-4384-a806-42ca02cc4255 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92057) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:08:36.281180 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15d2f8a7-80bb-4384-a806-42ca02cc4255 service placement] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:08:36.285865 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89fc0451-1357-4902-ad3a-0fc8b10564e6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:08:36.286757 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89fc0451-1357-4902-ad3a-0fc8b10564e6 admin admin] REQUEST_METHOD: `DELETE` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:08:36.287044 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89fc0451-1357-4902-ad3a-0fc8b10564e6 admin admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:08:36.287368 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89fc0451-1357-4902-ad3a-0fc8b10564e6 admin admin] PATH_INFO: `/v3/users/b18e5f1b24634096995068e41d94cad6` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:08:36.288540 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89fc0451-1357-4902-ad3a-0fc8b10564e6 admin admin] RBAC: Authorizing `identity:delete_user(user_id=b18e5f1b24634096995068e41d94cad6)` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:08:36.290581 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89fc0451-1357-4902-ad3a-0fc8b10564e6 admin admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:08:36.401822 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [None req-89fc0451-1357-4902-ad3a-0fc8b10564e6 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'b18e5f1b24634096995068e41d94cad6'} {{(pid=92057) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:08:36.460046 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [None req-89fc0451-1357-4902-ad3a-0fc8b10564e6 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'b18e5f1b24634096995068e41d94cad6'} {{(pid=92057) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:08:36.467543 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [None req-89fc0451-1357-4902-ad3a-0fc8b10564e6 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'b18e5f1b24634096995068e41d94cad6'} {{(pid=92057) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:08:36.468196 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [None req-89fc0451-1357-4902-ad3a-0fc8b10564e6 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'b18e5f1b24634096995068e41d94cad6'} {{(pid=92057) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:08:36.480068 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [None req-89fc0451-1357-4902-ad3a-0fc8b10564e6 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'b18e5f1b24634096995068e41d94cad6'} {{(pid=92057) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:08:36.486198 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 1236/2457] 10.222.0.30 () {58 vars in 1223 bytes} [Mon May 23 13:08:36 2022] DELETE /identity/v3/users/b18e5f1b24634096995068e41d94cad6 => generated 0 bytes in 224 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) May 23 13:08:36.745930 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4f05099-1dde-46bb-8c50-493e09bd2269 service neutron] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:08:36.757614 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4f05099-1dde-46bb-8c50-493e09bd2269 service neutron] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:08:36.816332 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a43f58b-4c30-44e0-ad90-c2a1bc989d0f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:08:36.817194 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a43f58b-4c30-44e0-ad90-c2a1bc989d0f admin admin] REQUEST_METHOD: `DELETE` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:08:36.817477 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a43f58b-4c30-44e0-ad90-c2a1bc989d0f admin admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:08:36.817752 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a43f58b-4c30-44e0-ad90-c2a1bc989d0f admin admin] PATH_INFO: `/v3/projects/5bda5ee012d84d88a3015baa3b80a7d7` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:08:36.818969 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a43f58b-4c30-44e0-ad90-c2a1bc989d0f admin admin] RBAC: Authorizing `identity:delete_project(project_id=5bda5ee012d84d88a3015baa3b80a7d7)` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:08:36.820873 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a43f58b-4c30-44e0-ad90-c2a1bc989d0f admin admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:08:36.944164 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-9a43f58b-4c30-44e0-ad90-c2a1bc989d0f admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '5bda5ee012d84d88a3015baa3b80a7d7', 'request_id': 'req-9a43f58b-4c30-44e0-ad90-c2a1bc989d0f'} {{(pid=92056) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:08:37.017186 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-9a43f58b-4c30-44e0-ad90-c2a1bc989d0f admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '5bda5ee012d84d88a3015baa3b80a7d7', 'request_id': 'req-9a43f58b-4c30-44e0-ad90-c2a1bc989d0f'} {{(pid=92056) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:08:37.026191 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-9a43f58b-4c30-44e0-ad90-c2a1bc989d0f admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '5bda5ee012d84d88a3015baa3b80a7d7', 'request_id': 'req-9a43f58b-4c30-44e0-ad90-c2a1bc989d0f'} {{(pid=92056) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:08:37.039709 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-9a43f58b-4c30-44e0-ad90-c2a1bc989d0f admin admin] The token cache is being invalidate because project 5bda5ee012d84d88a3015baa3b80a7d7 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=92056) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} May 23 13:08:37.040179 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-9a43f58b-4c30-44e0-ad90-c2a1bc989d0f admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=92056) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:08:37.043454 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-9a43f58b-4c30-44e0-ad90-c2a1bc989d0f admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=92056) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:08:37.043454 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 1222/2458] 10.222.0.30 () {58 vars in 1232 bytes} [Mon May 23 13:08:36 2022] DELETE /identity/v3/projects/5bda5ee012d84d88a3015baa3b80a7d7 => generated 0 bytes in 298 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) May 23 13:08:37.049218 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea18e28b-a0b5-4bb1-9671-46887943cd7e None None] REQUEST_METHOD: `POST` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:08:37.049381 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea18e28b-a0b5-4bb1-9671-46887943cd7e None None] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:08:37.049740 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea18e28b-a0b5-4bb1-9671-46887943cd7e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:08:37.069408 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.auth.core [None req-ea18e28b-a0b5-4bb1-9671-46887943cd7e None None] MFA Rules not processed for user `0f7119f8fa5c4445977671140ded91a4`. Rule list: `[]` (Enabled: `True`). {{(pid=92057) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 13:08:37.072215 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.fernet_utils [None req-ea18e28b-a0b5-4bb1-9671-46887943cd7e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92057) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:08:37.146878 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 1237/2459] 10.222.0.30 () {58 vars in 957 bytes} [Mon May 23 13:08:37 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 99 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) May 23 13:08:37.158289 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bee6f65-4e1c-40ee-be71-54dd7c7de8a7 None None] REQUEST_METHOD: `POST` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:08:37.158579 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bee6f65-4e1c-40ee-be71-54dd7c7de8a7 None None] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:08:37.158860 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bee6f65-4e1c-40ee-be71-54dd7c7de8a7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:08:37.178345 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.auth.core [None req-1bee6f65-4e1c-40ee-be71-54dd7c7de8a7 None None] MFA Rules not processed for user `0f7119f8fa5c4445977671140ded91a4`. Rule list: `[]` (Enabled: `True`). {{(pid=92056) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 13:08:37.179565 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.fernet_utils [None req-1bee6f65-4e1c-40ee-be71-54dd7c7de8a7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92056) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:08:37.190189 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 1223/2460] 10.222.0.30 () {58 vars in 957 bytes} [Mon May 23 13:08:37 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 33 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) May 23 13:08:37.194755 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea18e28b-a0b5-4bb1-9671-46887943cd7e None None] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:08:37.201519 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 13:08:37.207732 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea18e28b-a0b5-4bb1-9671-46887943cd7e None None] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:08:37.213790 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4eeda0cc-9766-446f-82ed-6a4a75b26d5d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:08:37.221484 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4eeda0cc-9766-446f-82ed-6a4a75b26d5d admin admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:08:37.221484 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4eeda0cc-9766-446f-82ed-6a4a75b26d5d admin admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:08:37.221484 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4eeda0cc-9766-446f-82ed-6a4a75b26d5d admin admin] PATH_INFO: `/v3/domains` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:08:37.221484 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4eeda0cc-9766-446f-82ed-6a4a75b26d5d admin admin] RBAC: Adding query filter params (name=Default) {{(pid=92057) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} May 23 13:08:37.221484 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4eeda0cc-9766-446f-82ed-6a4a75b26d5d admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:08:37.221484 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4eeda0cc-9766-446f-82ed-6a4a75b26d5d admin admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:08:37.229755 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 1238/2461] 10.222.0.30 () {58 vars in 1152 bytes} [Mon May 23 13:08:37 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 36 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) May 23 13:08:37.234611 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1bee6f65-4e1c-40ee-be71-54dd7c7de8a7 None None] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:08:37.244174 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1bee6f65-4e1c-40ee-be71-54dd7c7de8a7 None None] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:08:37.249605 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c370bd28-ffff-4567-9eb5-57efe15d410a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:08:37.250293 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c370bd28-ffff-4567-9eb5-57efe15d410a admin admin] REQUEST_METHOD: `POST` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:08:37.250577 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c370bd28-ffff-4567-9eb5-57efe15d410a admin admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:08:37.250854 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c370bd28-ffff-4567-9eb5-57efe15d410a admin admin] PATH_INFO: `/v3/projects` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:08:37.251615 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c370bd28-ffff-4567-9eb5-57efe15d410a admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:08:37.253549 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c370bd28-ffff-4567-9eb5-57efe15d410a admin admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:08:37.268955 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 13:08:37.363317 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 1224/2462] 10.222.0.30 () {60 vars in 1152 bytes} [Mon May 23 13:08:37 2022] POST /identity/v3/projects => generated 365 bytes in 129 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) May 23 13:08:37.367711 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4eeda0cc-9766-446f-82ed-6a4a75b26d5d admin admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:08:37.380301 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4eeda0cc-9766-446f-82ed-6a4a75b26d5d admin admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:08:37.451674 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f70409ab-29fa-4e40-8359-499406ec5521 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:08:37.452266 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f70409ab-29fa-4e40-8359-499406ec5521 admin admin] REQUEST_METHOD: `POST` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:08:37.452635 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f70409ab-29fa-4e40-8359-499406ec5521 admin admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:08:37.452912 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f70409ab-29fa-4e40-8359-499406ec5521 admin admin] PATH_INFO: `/v3/users` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:08:37.453605 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f70409ab-29fa-4e40-8359-499406ec5521 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:08:37.455604 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f70409ab-29fa-4e40-8359-499406ec5521 admin admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:08:37.537466 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 1239/2463] 10.222.0.30 () {60 vars in 1143 bytes} [Mon May 23 13:08:37 2022] POST /identity/v3/users => generated 283 bytes in 170 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) May 23 13:08:37.541831 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c370bd28-ffff-4567-9eb5-57efe15d410a admin admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:08:37.553363 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c370bd28-ffff-4567-9eb5-57efe15d410a admin admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:08:37.558516 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04603fb3-41eb-484b-8164-9e59c6a11f45 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:08:37.559208 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04603fb3-41eb-484b-8164-9e59c6a11f45 admin admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:08:37.559493 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04603fb3-41eb-484b-8164-9e59c6a11f45 admin admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:08:37.559772 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04603fb3-41eb-484b-8164-9e59c6a11f45 admin admin] PATH_INFO: `/v3/roles` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:08:37.560509 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04603fb3-41eb-484b-8164-9e59c6a11f45 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:08:37.562825 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04603fb3-41eb-484b-8164-9e59c6a11f45 admin admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:08:37.573301 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 1225/2464] 10.222.0.30 () {58 vars in 1121 bytes} [Mon May 23 13:08:37 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) May 23 13:08:37.577489 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f70409ab-29fa-4e40-8359-499406ec5521 admin admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:08:37.587616 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f70409ab-29fa-4e40-8359-499406ec5521 admin admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:08:37.592174 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b13a5faa-fbca-4aae-967e-d3fe1ed25020 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:08:37.592891 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b13a5faa-fbca-4aae-967e-d3fe1ed25020 admin admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:08:37.593192 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b13a5faa-fbca-4aae-967e-d3fe1ed25020 admin admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:08:37.593469 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b13a5faa-fbca-4aae-967e-d3fe1ed25020 admin admin] PATH_INFO: `/v3/roles` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:08:37.594226 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b13a5faa-fbca-4aae-967e-d3fe1ed25020 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:08:37.595826 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b13a5faa-fbca-4aae-967e-d3fe1ed25020 admin admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:08:37.605848 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 1240/2465] 10.222.0.30 () {58 vars in 1121 bytes} [Mon May 23 13:08:37 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) May 23 13:08:37.610509 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04603fb3-41eb-484b-8164-9e59c6a11f45 admin admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:08:37.620322 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04603fb3-41eb-484b-8164-9e59c6a11f45 admin admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:08:37.626095 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9174df5-2531-4c3f-a350-df63bced0edc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:08:37.626972 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9174df5-2531-4c3f-a350-df63bced0edc admin admin] REQUEST_METHOD: `PUT` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:08:37.628949 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9174df5-2531-4c3f-a350-df63bced0edc admin admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:08:37.629257 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9174df5-2531-4c3f-a350-df63bced0edc admin admin] PATH_INFO: `/v3/projects/5bb82b0a31ba4d619ebc3d36444f326f/users/1da377a30ae34e72b548613e886f8d59/roles/80fb22507c9a4ff09d44d8f12f6fff1a` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:08:37.651352 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9174df5-2531-4c3f-a350-df63bced0edc admin admin] RBAC: Authorizing `identity:create_grant(project_id=5bb82b0a31ba4d619ebc3d36444f326f, user_id=1da377a30ae34e72b548613e886f8d59, role_id=80fb22507c9a4ff09d44d8f12f6fff1a)` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:08:37.654049 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9174df5-2531-4c3f-a350-df63bced0edc admin admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:08:37.726256 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 1226/2466] 10.222.0.30 () {60 vars in 1482 bytes} [Mon May 23 13:08:37 2022] PUT /identity/v3/projects/5bb82b0a31ba4d619ebc3d36444f326f/users/1da377a30ae34e72b548613e886f8d59/roles/80fb22507c9a4ff09d44d8f12f6fff1a => generated 0 bytes in 116 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) May 23 13:08:37.737767 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f93798cd-bba3-4bf5-aad5-7753eb9dc5fd None None] REQUEST_METHOD: `POST` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:08:37.738100 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f93798cd-bba3-4bf5-aad5-7753eb9dc5fd None None] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:08:37.738380 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f93798cd-bba3-4bf5-aad5-7753eb9dc5fd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:08:37.765282 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.auth.core [None req-f93798cd-bba3-4bf5-aad5-7753eb9dc5fd None None] MFA Rules not processed for user `1da377a30ae34e72b548613e886f8d59`. Rule list: `[]` (Enabled: `True`). {{(pid=92057) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 13:08:37.766789 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.fernet_utils [None req-f93798cd-bba3-4bf5-aad5-7753eb9dc5fd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92057) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:08:37.831784 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 1241/2467] 10.222.0.30 () {58 vars in 957 bytes} [Mon May 23 13:08:37 2022] POST /identity/v3/auth/tokens => generated 3232 bytes in 95 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) May 23 13:08:37.836298 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9174df5-2531-4c3f-a350-df63bced0edc admin admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:08:37.896470 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9174df5-2531-4c3f-a350-df63bced0edc admin admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:08:37.901356 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cefd9adc-8ec5-4620-81cb-bfb32e07b4d7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:08:37.902038 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cefd9adc-8ec5-4620-81cb-bfb32e07b4d7 admin admin] REQUEST_METHOD: `POST` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:08:37.902322 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cefd9adc-8ec5-4620-81cb-bfb32e07b4d7 admin admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:08:37.902595 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cefd9adc-8ec5-4620-81cb-bfb32e07b4d7 admin admin] PATH_INFO: `/v3/projects` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:08:37.903290 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cefd9adc-8ec5-4620-81cb-bfb32e07b4d7 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:08:37.905714 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cefd9adc-8ec5-4620-81cb-bfb32e07b4d7 admin admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:08:37.990403 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 1227/2468] 10.222.0.30 () {60 vars in 1152 bytes} [Mon May 23 13:08:37 2022] POST /identity/v3/projects => generated 365 bytes in 154 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) May 23 13:08:37.994344 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f93798cd-bba3-4bf5-aad5-7753eb9dc5fd None None] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:08:38.005851 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f93798cd-bba3-4bf5-aad5-7753eb9dc5fd None None] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:08:38.056947 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 13:08:38.063823 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d375c22a-98fc-4235-855a-e0cbe3be1ad0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:08:38.066436 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d375c22a-98fc-4235-855a-e0cbe3be1ad0 admin admin] REQUEST_METHOD: `POST` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:08:38.066436 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d375c22a-98fc-4235-855a-e0cbe3be1ad0 admin admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:08:38.066436 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d375c22a-98fc-4235-855a-e0cbe3be1ad0 admin admin] PATH_INFO: `/v3/users` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:08:38.066436 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d375c22a-98fc-4235-855a-e0cbe3be1ad0 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:08:38.067899 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d375c22a-98fc-4235-855a-e0cbe3be1ad0 admin admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:08:38.159925 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 1242/2469] 10.222.0.30 () {60 vars in 1143 bytes} [Mon May 23 13:08:37 2022] POST /identity/v3/users => generated 289 bytes in 166 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) May 23 13:08:38.166965 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cefd9adc-8ec5-4620-81cb-bfb32e07b4d7 admin admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:08:38.178002 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cefd9adc-8ec5-4620-81cb-bfb32e07b4d7 admin admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:08:38.182784 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a40f37c8-675c-4620-83a2-4fd78e03eb2e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:08:38.183440 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a40f37c8-675c-4620-83a2-4fd78e03eb2e admin admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:08:38.183722 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a40f37c8-675c-4620-83a2-4fd78e03eb2e admin admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:08:38.183998 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a40f37c8-675c-4620-83a2-4fd78e03eb2e admin admin] PATH_INFO: `/v3/roles` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:08:38.184719 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a40f37c8-675c-4620-83a2-4fd78e03eb2e admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:08:38.187085 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a40f37c8-675c-4620-83a2-4fd78e03eb2e admin admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:08:38.198392 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 1228/2470] 10.222.0.30 () {58 vars in 1121 bytes} [Mon May 23 13:08:38 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) May 23 13:08:38.203781 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d375c22a-98fc-4235-855a-e0cbe3be1ad0 admin admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:08:38.213720 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d375c22a-98fc-4235-855a-e0cbe3be1ad0 admin admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:08:38.219483 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b15b649d-5adf-4b74-bdaf-0462efaf5d78 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:08:38.220349 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b15b649d-5adf-4b74-bdaf-0462efaf5d78 admin admin] REQUEST_METHOD: `PUT` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:08:38.220692 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b15b649d-5adf-4b74-bdaf-0462efaf5d78 admin admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:08:38.220974 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b15b649d-5adf-4b74-bdaf-0462efaf5d78 admin admin] PATH_INFO: `/v3/projects/75ab60c480704e999689098becba5eab/users/5f4119a36dac44c7bd14c22aa4337087/roles/bc3c3c9d654940cbaa5177a6be23d1bc` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:08:38.235694 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b15b649d-5adf-4b74-bdaf-0462efaf5d78 admin admin] RBAC: Authorizing `identity:create_grant(project_id=75ab60c480704e999689098becba5eab, user_id=5f4119a36dac44c7bd14c22aa4337087, role_id=bc3c3c9d654940cbaa5177a6be23d1bc)` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:08:38.238423 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b15b649d-5adf-4b74-bdaf-0462efaf5d78 admin admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:08:38.307678 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 1243/2471] 10.222.0.30 () {60 vars in 1482 bytes} [Mon May 23 13:08:38 2022] PUT /identity/v3/projects/75ab60c480704e999689098becba5eab/users/5f4119a36dac44c7bd14c22aa4337087/roles/bc3c3c9d654940cbaa5177a6be23d1bc => generated 0 bytes in 105 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) May 23 13:08:38.311595 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a40f37c8-675c-4620-83a2-4fd78e03eb2e admin admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:08:38.321477 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a40f37c8-675c-4620-83a2-4fd78e03eb2e admin admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:08:38.377303 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73c5104e-eac1-44e9-ab2c-a21ec90c1fc6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:08:38.377981 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73c5104e-eac1-44e9-ab2c-a21ec90c1fc6 admin admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:08:38.378265 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73c5104e-eac1-44e9-ab2c-a21ec90c1fc6 admin admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:08:38.378541 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73c5104e-eac1-44e9-ab2c-a21ec90c1fc6 admin admin] PATH_INFO: `/v3/roles` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:08:38.379255 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-73c5104e-eac1-44e9-ab2c-a21ec90c1fc6 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:08:38.380969 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-73c5104e-eac1-44e9-ab2c-a21ec90c1fc6 admin admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:08:38.390971 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 1229/2472] 10.222.0.30 () {58 vars in 1121 bytes} [Mon May 23 13:08:38 2022] GET /identity/v3/roles => generated 1420 bytes in 80 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) May 23 13:08:38.395150 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b15b649d-5adf-4b74-bdaf-0462efaf5d78 admin admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:08:38.406269 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b15b649d-5adf-4b74-bdaf-0462efaf5d78 admin admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:08:38.410858 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25908fa5-daef-4444-b474-b6571babaf68 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:08:38.411726 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25908fa5-daef-4444-b474-b6571babaf68 admin admin] REQUEST_METHOD: `PUT` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:08:38.412011 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25908fa5-daef-4444-b474-b6571babaf68 admin admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:08:38.412290 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25908fa5-daef-4444-b474-b6571babaf68 admin admin] PATH_INFO: `/v3/projects/75ab60c480704e999689098becba5eab/users/5f4119a36dac44c7bd14c22aa4337087/roles/bc3c3c9d654940cbaa5177a6be23d1bc` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:08:38.414288 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25908fa5-daef-4444-b474-b6571babaf68 admin admin] RBAC: Authorizing `identity:create_grant(project_id=75ab60c480704e999689098becba5eab, user_id=5f4119a36dac44c7bd14c22aa4337087, role_id=bc3c3c9d654940cbaa5177a6be23d1bc)` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:08:38.416925 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25908fa5-daef-4444-b474-b6571babaf68 admin admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:08:38.428088 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 1244/2473] 10.222.0.30 () {60 vars in 1482 bytes} [Mon May 23 13:08:38 2022] PUT /identity/v3/projects/75ab60c480704e999689098becba5eab/users/5f4119a36dac44c7bd14c22aa4337087/roles/bc3c3c9d654940cbaa5177a6be23d1bc => generated 0 bytes in 33 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) May 23 13:08:38.444761 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d7c05d3-29a8-4baa-b2b8-e8f4490af470 None None] REQUEST_METHOD: `POST` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:08:38.445056 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d7c05d3-29a8-4baa-b2b8-e8f4490af470 None None] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:08:38.445344 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d7c05d3-29a8-4baa-b2b8-e8f4490af470 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:08:38.474728 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.auth.core [None req-8d7c05d3-29a8-4baa-b2b8-e8f4490af470 None None] MFA Rules not processed for user `5f4119a36dac44c7bd14c22aa4337087`. Rule list: `[]` (Enabled: `True`). {{(pid=92056) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 13:08:38.479313 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.fernet_utils [None req-8d7c05d3-29a8-4baa-b2b8-e8f4490af470 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92056) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:08:38.485414 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 13:08:38.546270 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 1230/2474] 10.222.0.30 () {58 vars in 957 bytes} [Mon May 23 13:08:38 2022] POST /identity/v3/auth/tokens => generated 3299 bytes in 102 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) May 23 13:08:38.557526 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25908fa5-daef-4444-b474-b6571babaf68 admin admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:08:38.559275 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.fernet_utils [None req-25908fa5-daef-4444-b474-b6571babaf68 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92057) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:08:38.618468 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25908fa5-daef-4444-b474-b6571babaf68 admin admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:08:38.622096 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06656996-e480-4e72-be32-36c1421db640 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1666097f8ed545c6b0e4809c93e4775a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9c389ae65aaa4542bb5a5c4fd0bacf5b', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:08:38.622728 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06656996-e480-4e72-be32-36c1421db640 service cinder] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:08:38.623009 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06656996-e480-4e72-be32-36c1421db640 service cinder] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:08:38.623281 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06656996-e480-4e72-be32-36c1421db640 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:08:38.632720 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06656996-e480-4e72-be32-36c1421db640 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:08:38.634221 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06656996-e480-4e72-be32-36c1421db640 service cinder] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:08:38.644448 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 1245/2475] 10.222.0.30 () {60 vars in 1365 bytes} [Mon May 23 13:08:38 2022] GET /identity/v3/auth/tokens => generated 3299 bytes in 87 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) May 23 13:08:38.846059 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d7c05d3-29a8-4baa-b2b8-e8f4490af470 None None] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:08:38.855879 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d7c05d3-29a8-4baa-b2b8-e8f4490af470 None None] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:08:38.860715 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ddef4f3e-51ca-4cd9-aed3-ad5375c97878 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1666097f8ed545c6b0e4809c93e4775a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9c389ae65aaa4542bb5a5c4fd0bacf5b', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:08:38.861352 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddef4f3e-51ca-4cd9-aed3-ad5375c97878 service cinder] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:08:38.861637 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddef4f3e-51ca-4cd9-aed3-ad5375c97878 service cinder] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:08:38.861946 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddef4f3e-51ca-4cd9-aed3-ad5375c97878 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:08:38.919331 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ddef4f3e-51ca-4cd9-aed3-ad5375c97878 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:08:38.920816 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ddef4f3e-51ca-4cd9-aed3-ad5375c97878 service cinder] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:08:38.929761 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 1231/2476] 10.222.0.30 () {60 vars in 1365 bytes} [Mon May 23 13:08:38 2022] GET /identity/v3/auth/tokens => generated 3232 bytes in 84 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) May 23 13:08:48.639329 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=44,keystone:INSERT=5 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 13:08:48.925682 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=52 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 13:08:51.772413 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06656996-e480-4e72-be32-36c1421db640 service cinder] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:08:51.836728 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06656996-e480-4e72-be32-36c1421db640 service cinder] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:08:51.841381 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0276f78b-c812-4174-99f4-de47653d1d11 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:08:51.842278 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0276f78b-c812-4174-99f4-de47653d1d11 admin admin] REQUEST_METHOD: `DELETE` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:08:51.842568 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0276f78b-c812-4174-99f4-de47653d1d11 admin admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:08:51.842848 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0276f78b-c812-4174-99f4-de47653d1d11 admin admin] PATH_INFO: `/v3/users/1da377a30ae34e72b548613e886f8d59` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:08:51.844052 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0276f78b-c812-4174-99f4-de47653d1d11 admin admin] RBAC: Authorizing `identity:delete_user(user_id=1da377a30ae34e72b548613e886f8d59)` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:08:51.846540 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0276f78b-c812-4174-99f4-de47653d1d11 admin admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:08:52.045975 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [None req-0276f78b-c812-4174-99f4-de47653d1d11 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '1da377a30ae34e72b548613e886f8d59'} {{(pid=92057) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:08:52.119183 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [None req-0276f78b-c812-4174-99f4-de47653d1d11 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '1da377a30ae34e72b548613e886f8d59'} {{(pid=92057) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:08:52.127152 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [None req-0276f78b-c812-4174-99f4-de47653d1d11 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '1da377a30ae34e72b548613e886f8d59'} {{(pid=92057) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:08:52.127831 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [None req-0276f78b-c812-4174-99f4-de47653d1d11 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '1da377a30ae34e72b548613e886f8d59'} {{(pid=92057) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:08:52.140129 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [None req-0276f78b-c812-4174-99f4-de47653d1d11 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '1da377a30ae34e72b548613e886f8d59'} {{(pid=92057) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:08:52.147022 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 1246/2477] 10.222.0.30 () {58 vars in 1223 bytes} [Mon May 23 13:08:51 2022] DELETE /identity/v3/users/1da377a30ae34e72b548613e886f8d59 => generated 0 bytes in 375 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) May 23 13:08:52.152051 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-adb1bdf3-6109-4a48-b23b-73ca5dfc4914 None None] REQUEST_METHOD: `POST` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:08:52.152540 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-adb1bdf3-6109-4a48-b23b-73ca5dfc4914 None None] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:08:52.152636 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-adb1bdf3-6109-4a48-b23b-73ca5dfc4914 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:08:52.170170 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.auth.core [None req-adb1bdf3-6109-4a48-b23b-73ca5dfc4914 None None] MFA Rules not processed for user `0f7119f8fa5c4445977671140ded91a4`. Rule list: `[]` (Enabled: `True`). {{(pid=92056) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 13:08:52.171463 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.fernet_utils [None req-adb1bdf3-6109-4a48-b23b-73ca5dfc4914 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92056) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:08:52.232764 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 1232/2478] 10.222.0.30 () {58 vars in 957 bytes} [Mon May 23 13:08:52 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 82 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) May 23 13:08:52.242960 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0276f78b-c812-4174-99f4-de47653d1d11 admin admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:08:52.244404 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.fernet_utils [None req-0276f78b-c812-4174-99f4-de47653d1d11 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92057) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:08:52.306866 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0276f78b-c812-4174-99f4-de47653d1d11 admin admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:08:52.310575 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13039486-46f8-4841-a27b-7fff95abd2cc service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2eb83ca38e79449095f3a2f947bbb06a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9c389ae65aaa4542bb5a5c4fd0bacf5b', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:08:52.311232 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13039486-46f8-4841-a27b-7fff95abd2cc service neutron] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:08:52.311512 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13039486-46f8-4841-a27b-7fff95abd2cc service neutron] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:08:52.311786 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13039486-46f8-4841-a27b-7fff95abd2cc service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:08:52.323228 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13039486-46f8-4841-a27b-7fff95abd2cc service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:08:52.324868 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13039486-46f8-4841-a27b-7fff95abd2cc service neutron] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:08:52.336791 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 1247/2479] 10.222.0.30 () {60 vars in 1365 bytes} [Mon May 23 13:08:52 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 94 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) May 23 13:08:52.989688 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-adb1bdf3-6109-4a48-b23b-73ca5dfc4914 None None] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:08:52.991520 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.fernet_utils [None req-adb1bdf3-6109-4a48-b23b-73ca5dfc4914 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92056) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:08:53.008656 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-adb1bdf3-6109-4a48-b23b-73ca5dfc4914 None None] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:08:53.014600 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4a9809f-e5b3-45e8-aed0-0b8f73c6b1bc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:08:53.015482 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4a9809f-e5b3-45e8-aed0-0b8f73c6b1bc admin admin] REQUEST_METHOD: `DELETE` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:08:53.015771 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4a9809f-e5b3-45e8-aed0-0b8f73c6b1bc admin admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:08:53.016078 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4a9809f-e5b3-45e8-aed0-0b8f73c6b1bc admin admin] PATH_INFO: `/v3/projects/5bb82b0a31ba4d619ebc3d36444f326f` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:08:53.017313 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4a9809f-e5b3-45e8-aed0-0b8f73c6b1bc admin admin] RBAC: Authorizing `identity:delete_project(project_id=5bb82b0a31ba4d619ebc3d36444f326f)` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:08:53.019290 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4a9809f-e5b3-45e8-aed0-0b8f73c6b1bc admin admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:08:53.149337 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-a4a9809f-e5b3-45e8-aed0-0b8f73c6b1bc admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '5bb82b0a31ba4d619ebc3d36444f326f', 'request_id': 'req-a4a9809f-e5b3-45e8-aed0-0b8f73c6b1bc'} {{(pid=92056) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:08:53.260109 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-a4a9809f-e5b3-45e8-aed0-0b8f73c6b1bc admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '5bb82b0a31ba4d619ebc3d36444f326f', 'request_id': 'req-a4a9809f-e5b3-45e8-aed0-0b8f73c6b1bc'} {{(pid=92056) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:08:53.267239 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-a4a9809f-e5b3-45e8-aed0-0b8f73c6b1bc admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '5bb82b0a31ba4d619ebc3d36444f326f', 'request_id': 'req-a4a9809f-e5b3-45e8-aed0-0b8f73c6b1bc'} {{(pid=92056) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:08:53.279230 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-a4a9809f-e5b3-45e8-aed0-0b8f73c6b1bc admin admin] The token cache is being invalidate because project 5bb82b0a31ba4d619ebc3d36444f326f was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=92056) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} May 23 13:08:53.279623 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-a4a9809f-e5b3-45e8-aed0-0b8f73c6b1bc admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=92056) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:08:53.280418 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-a4a9809f-e5b3-45e8-aed0-0b8f73c6b1bc admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=92056) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:08:53.282148 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 1233/2480] 10.222.0.30 () {58 vars in 1232 bytes} [Mon May 23 13:08:52 2022] DELETE /identity/v3/projects/5bb82b0a31ba4d619ebc3d36444f326f => generated 0 bytes in 293 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) May 23 13:08:53.286174 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13039486-46f8-4841-a27b-7fff95abd2cc service neutron] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:08:53.296240 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13039486-46f8-4841-a27b-7fff95abd2cc service neutron] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:08:53.316845 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 13:08:53.362531 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf746f6b-5ff7-420a-a7fb-b61613dddfd6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:08:53.364020 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf746f6b-5ff7-420a-a7fb-b61613dddfd6 admin admin] REQUEST_METHOD: `DELETE` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:08:53.364525 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf746f6b-5ff7-420a-a7fb-b61613dddfd6 admin admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:08:53.365019 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf746f6b-5ff7-420a-a7fb-b61613dddfd6 admin admin] PATH_INFO: `/v3/users/5f4119a36dac44c7bd14c22aa4337087` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:08:53.367105 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf746f6b-5ff7-420a-a7fb-b61613dddfd6 admin admin] RBAC: Authorizing `identity:delete_user(user_id=5f4119a36dac44c7bd14c22aa4337087)` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:08:53.370747 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf746f6b-5ff7-420a-a7fb-b61613dddfd6 admin admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:08:53.551972 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [None req-cf746f6b-5ff7-420a-a7fb-b61613dddfd6 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '5f4119a36dac44c7bd14c22aa4337087'} {{(pid=92057) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:08:53.634466 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [None req-cf746f6b-5ff7-420a-a7fb-b61613dddfd6 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '5f4119a36dac44c7bd14c22aa4337087'} {{(pid=92057) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:08:53.638507 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [None req-cf746f6b-5ff7-420a-a7fb-b61613dddfd6 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '5f4119a36dac44c7bd14c22aa4337087'} {{(pid=92057) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:08:53.639225 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [None req-cf746f6b-5ff7-420a-a7fb-b61613dddfd6 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '5f4119a36dac44c7bd14c22aa4337087'} {{(pid=92057) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:08:53.654049 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [None req-cf746f6b-5ff7-420a-a7fb-b61613dddfd6 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '5f4119a36dac44c7bd14c22aa4337087'} {{(pid=92057) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:08:53.661617 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 1248/2481] 10.222.0.30 () {58 vars in 1223 bytes} [Mon May 23 13:08:53 2022] DELETE /identity/v3/users/5f4119a36dac44c7bd14c22aa4337087 => generated 0 bytes in 376 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) May 23 13:08:54.172167 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4a9809f-e5b3-45e8-aed0-0b8f73c6b1bc admin admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:08:54.172167 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.fernet_utils [None req-a4a9809f-e5b3-45e8-aed0-0b8f73c6b1bc admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92056) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:08:54.187520 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4a9809f-e5b3-45e8-aed0-0b8f73c6b1bc admin admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:08:54.249113 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a94b686b-109c-4937-8b83-2b14b7cd13ad admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:08:54.250513 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a94b686b-109c-4937-8b83-2b14b7cd13ad admin admin] REQUEST_METHOD: `DELETE` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:08:54.250798 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a94b686b-109c-4937-8b83-2b14b7cd13ad admin admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:08:54.251077 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a94b686b-109c-4937-8b83-2b14b7cd13ad admin admin] PATH_INFO: `/v3/projects/75ab60c480704e999689098becba5eab` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:08:54.252347 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a94b686b-109c-4937-8b83-2b14b7cd13ad admin admin] RBAC: Authorizing `identity:delete_project(project_id=75ab60c480704e999689098becba5eab)` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:08:54.254352 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a94b686b-109c-4937-8b83-2b14b7cd13ad admin admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:08:54.384075 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=91,keystone:DELETE=8,keystone:INSERT=1 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 13:08:54.401909 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-a94b686b-109c-4937-8b83-2b14b7cd13ad admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '75ab60c480704e999689098becba5eab', 'request_id': 'req-a94b686b-109c-4937-8b83-2b14b7cd13ad'} {{(pid=92056) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:08:54.483112 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-a94b686b-109c-4937-8b83-2b14b7cd13ad admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '75ab60c480704e999689098becba5eab', 'request_id': 'req-a94b686b-109c-4937-8b83-2b14b7cd13ad'} {{(pid=92056) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:08:54.490744 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-a94b686b-109c-4937-8b83-2b14b7cd13ad admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '75ab60c480704e999689098becba5eab', 'request_id': 'req-a94b686b-109c-4937-8b83-2b14b7cd13ad'} {{(pid=92056) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:08:54.504198 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-a94b686b-109c-4937-8b83-2b14b7cd13ad admin admin] The token cache is being invalidate because project 75ab60c480704e999689098becba5eab was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=92056) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} May 23 13:08:54.504511 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-a94b686b-109c-4937-8b83-2b14b7cd13ad admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=92056) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:08:54.505145 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-a94b686b-109c-4937-8b83-2b14b7cd13ad admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=92056) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:08:54.510107 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 1234/2482] 10.222.0.30 () {58 vars in 1232 bytes} [Mon May 23 13:08:54 2022] DELETE /identity/v3/projects/75ab60c480704e999689098becba5eab => generated 0 bytes in 337 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) May 23 13:08:54.511825 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cc2910f-d1e0-4f2f-b1b1-eda62ca84227 None None] REQUEST_METHOD: `POST` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:08:54.512228 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cc2910f-d1e0-4f2f-b1b1-eda62ca84227 None None] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:08:54.512377 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cc2910f-d1e0-4f2f-b1b1-eda62ca84227 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:08:54.533421 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.auth.core [None req-8cc2910f-d1e0-4f2f-b1b1-eda62ca84227 None None] MFA Rules not processed for user `0f7119f8fa5c4445977671140ded91a4`. Rule list: `[]` (Enabled: `True`). {{(pid=92057) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 13:08:54.534804 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.fernet_utils [None req-8cc2910f-d1e0-4f2f-b1b1-eda62ca84227 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92057) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:08:54.602212 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 1249/2483] 10.222.0.30 () {58 vars in 957 bytes} [Mon May 23 13:08:54 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 91 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) May 23 13:08:54.609810 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84f5db7c-4fe1-483c-b849-844437bf424f None None] REQUEST_METHOD: `POST` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:08:54.610131 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84f5db7c-4fe1-483c-b849-844437bf424f None None] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:08:54.610408 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84f5db7c-4fe1-483c-b849-844437bf424f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:08:54.629465 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.auth.core [None req-84f5db7c-4fe1-483c-b849-844437bf424f None None] MFA Rules not processed for user `0f7119f8fa5c4445977671140ded91a4`. Rule list: `[]` (Enabled: `True`). {{(pid=92056) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 13:08:54.630780 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.fernet_utils [None req-84f5db7c-4fe1-483c-b849-844437bf424f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92056) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:08:54.638586 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 1235/2484] 10.222.0.30 () {58 vars in 957 bytes} [Mon May 23 13:08:54 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 30 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) May 23 13:08:54.644676 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8cc2910f-d1e0-4f2f-b1b1-eda62ca84227 None None] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:08:54.654601 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8cc2910f-d1e0-4f2f-b1b1-eda62ca84227 None None] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:08:54.659370 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8ac61b6-948b-4420-8d50-31f733ca82fe admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:08:54.660023 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8ac61b6-948b-4420-8d50-31f733ca82fe admin admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:08:54.660304 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8ac61b6-948b-4420-8d50-31f733ca82fe admin admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:08:54.660586 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8ac61b6-948b-4420-8d50-31f733ca82fe admin admin] PATH_INFO: `/v3/domains` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:08:54.661302 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8ac61b6-948b-4420-8d50-31f733ca82fe admin admin] RBAC: Adding query filter params (name=Default) {{(pid=92057) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} May 23 13:08:54.661606 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8ac61b6-948b-4420-8d50-31f733ca82fe admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:08:54.663309 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8ac61b6-948b-4420-8d50-31f733ca82fe admin admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:08:54.675619 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 1250/2485] 10.222.0.30 () {58 vars in 1152 bytes} [Mon May 23 13:08:54 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) May 23 13:08:54.679835 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84f5db7c-4fe1-483c-b849-844437bf424f None None] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:08:54.689706 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84f5db7c-4fe1-483c-b849-844437bf424f None None] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:08:54.694479 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70f53640-0c85-4250-bcfc-04ec68338fab admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:08:54.695139 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70f53640-0c85-4250-bcfc-04ec68338fab admin admin] REQUEST_METHOD: `POST` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:08:54.695412 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70f53640-0c85-4250-bcfc-04ec68338fab admin admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:08:54.695682 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70f53640-0c85-4250-bcfc-04ec68338fab admin admin] PATH_INFO: `/v3/projects` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:08:54.696366 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70f53640-0c85-4250-bcfc-04ec68338fab admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:08:54.698328 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70f53640-0c85-4250-bcfc-04ec68338fab admin admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:08:54.798254 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 1236/2486] 10.222.0.30 () {60 vars in 1152 bytes} [Mon May 23 13:08:54 2022] POST /identity/v3/projects => generated 385 bytes in 116 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) May 23 13:08:54.799736 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8ac61b6-948b-4420-8d50-31f733ca82fe admin admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:08:54.811893 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8ac61b6-948b-4420-8d50-31f733ca82fe admin admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:08:54.840220 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 13:08:54.873400 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1fc2c0e0-b221-4417-97db-288466c2bf15 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:08:54.874476 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fc2c0e0-b221-4417-97db-288466c2bf15 admin admin] REQUEST_METHOD: `POST` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:08:54.874925 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fc2c0e0-b221-4417-97db-288466c2bf15 admin admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:08:54.875203 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fc2c0e0-b221-4417-97db-288466c2bf15 admin admin] PATH_INFO: `/v3/users` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:08:54.875894 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1fc2c0e0-b221-4417-97db-288466c2bf15 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:08:54.878952 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1fc2c0e0-b221-4417-97db-288466c2bf15 admin admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:08:54.948834 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 1251/2487] 10.222.0.30 () {60 vars in 1143 bytes} [Mon May 23 13:08:54 2022] POST /identity/v3/users => generated 293 bytes in 150 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) May 23 13:08:54.954369 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70f53640-0c85-4250-bcfc-04ec68338fab admin admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:08:54.965374 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70f53640-0c85-4250-bcfc-04ec68338fab admin admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:08:54.970006 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-273ecd84-f276-4faa-b7ce-18a25b3d6efa admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:08:54.970784 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-273ecd84-f276-4faa-b7ce-18a25b3d6efa admin admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:08:54.971055 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-273ecd84-f276-4faa-b7ce-18a25b3d6efa admin admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:08:54.971438 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-273ecd84-f276-4faa-b7ce-18a25b3d6efa admin admin] PATH_INFO: `/v3/roles` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:08:54.972156 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-273ecd84-f276-4faa-b7ce-18a25b3d6efa admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:08:54.973753 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-273ecd84-f276-4faa-b7ce-18a25b3d6efa admin admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:08:54.984129 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 1237/2488] 10.222.0.30 () {58 vars in 1121 bytes} [Mon May 23 13:08:54 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) May 23 13:08:54.997782 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1fc2c0e0-b221-4417-97db-288466c2bf15 admin admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:08:54.998343 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1fc2c0e0-b221-4417-97db-288466c2bf15 admin admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:08:55.003622 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3fbfcad-90c6-43e7-b5d1-1e37af397199 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:08:55.004281 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3fbfcad-90c6-43e7-b5d1-1e37af397199 admin admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:08:55.004571 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3fbfcad-90c6-43e7-b5d1-1e37af397199 admin admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:08:55.004877 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3fbfcad-90c6-43e7-b5d1-1e37af397199 admin admin] PATH_INFO: `/v3/roles` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:08:55.005607 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3fbfcad-90c6-43e7-b5d1-1e37af397199 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:08:55.007397 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3fbfcad-90c6-43e7-b5d1-1e37af397199 admin admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:08:55.020110 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 1252/2489] 10.222.0.30 () {58 vars in 1121 bytes} [Mon May 23 13:08:54 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) May 23 13:08:55.024568 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-273ecd84-f276-4faa-b7ce-18a25b3d6efa admin admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:08:55.035093 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-273ecd84-f276-4faa-b7ce-18a25b3d6efa admin admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:08:55.039758 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d773903b-91c0-447b-ac48-b725f8b080fc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:08:55.040637 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d773903b-91c0-447b-ac48-b725f8b080fc admin admin] REQUEST_METHOD: `PUT` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:08:55.040916 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d773903b-91c0-447b-ac48-b725f8b080fc admin admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:08:55.041189 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d773903b-91c0-447b-ac48-b725f8b080fc admin admin] PATH_INFO: `/v3/projects/79666b25325f459b8b38dff1565cbabb/users/b30c02ae86da42b1a5f036a3faec2bf0/roles/80fb22507c9a4ff09d44d8f12f6fff1a` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:08:55.056478 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d773903b-91c0-447b-ac48-b725f8b080fc admin admin] RBAC: Authorizing `identity:create_grant(project_id=79666b25325f459b8b38dff1565cbabb, user_id=b30c02ae86da42b1a5f036a3faec2bf0, role_id=80fb22507c9a4ff09d44d8f12f6fff1a)` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:08:55.059189 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d773903b-91c0-447b-ac48-b725f8b080fc admin admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:08:55.143506 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 1238/2490] 10.222.0.30 () {60 vars in 1482 bytes} [Mon May 23 13:08:55 2022] PUT /identity/v3/projects/79666b25325f459b8b38dff1565cbabb/users/b30c02ae86da42b1a5f036a3faec2bf0/roles/80fb22507c9a4ff09d44d8f12f6fff1a => generated 0 bytes in 120 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) May 23 13:08:55.160909 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83c4d85a-54dc-4cc7-89d2-64a1ffce8536 None None] REQUEST_METHOD: `POST` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:08:55.161200 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83c4d85a-54dc-4cc7-89d2-64a1ffce8536 None None] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:08:55.161478 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83c4d85a-54dc-4cc7-89d2-64a1ffce8536 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:08:55.162065 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-880b08c0-3671-46ca-8bf2-e25d080aa5cb None None] REQUEST_METHOD: `POST` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:08:55.162566 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-880b08c0-3671-46ca-8bf2-e25d080aa5cb None None] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:08:55.163115 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-880b08c0-3671-46ca-8bf2-e25d080aa5cb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:08:55.183853 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.auth.core [None req-880b08c0-3671-46ca-8bf2-e25d080aa5cb None None] MFA Rules not processed for user `257251830f5148e5b9e435c2c13828e8`. Rule list: `[]` (Enabled: `True`). {{(pid=92056) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 13:08:55.185065 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.fernet_utils [None req-880b08c0-3671-46ca-8bf2-e25d080aa5cb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92056) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:08:55.194772 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.auth.core [None req-83c4d85a-54dc-4cc7-89d2-64a1ffce8536 None None] MFA Rules not processed for user `b30c02ae86da42b1a5f036a3faec2bf0`. Rule list: `[]` (Enabled: `True`). {{(pid=92057) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 13:08:55.195983 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.fernet_utils [None req-83c4d85a-54dc-4cc7-89d2-64a1ffce8536 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92057) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:08:55.245151 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 1239/2491] 10.222.0.30 () {58 vars in 957 bytes} [Mon May 23 13:08:55 2022] POST /identity/v3/auth/tokens => generated 3252 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) May 23 13:08:55.263654 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 1253/2492] 10.222.0.30 () {58 vars in 957 bytes} [Mon May 23 13:08:55 2022] POST /identity/v3/auth/tokens => generated 3252 bytes in 104 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) May 23 13:08:55.268974 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-880b08c0-3671-46ca-8bf2-e25d080aa5cb None None] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:08:55.332787 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:DELETE=6,keystone:SELECT=91,keystone:INSERT=3 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 13:08:55.338719 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-880b08c0-3671-46ca-8bf2-e25d080aa5cb None None] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:08:55.344045 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b755097-2a68-4619-88ea-dd4fac078d9b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:08:55.344713 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b755097-2a68-4619-88ea-dd4fac078d9b admin admin] REQUEST_METHOD: `POST` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:08:55.347304 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b755097-2a68-4619-88ea-dd4fac078d9b admin admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:08:55.347304 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b755097-2a68-4619-88ea-dd4fac078d9b admin admin] PATH_INFO: `/v3/projects` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:08:55.347304 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b755097-2a68-4619-88ea-dd4fac078d9b admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:08:55.347981 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b755097-2a68-4619-88ea-dd4fac078d9b admin admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:08:55.421146 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 1240/2493] 10.222.0.30 () {60 vars in 1152 bytes} [Mon May 23 13:08:55 2022] POST /identity/v3/projects => generated 387 bytes in 150 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) May 23 13:08:55.423009 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83c4d85a-54dc-4cc7-89d2-64a1ffce8536 None None] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:08:55.434829 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83c4d85a-54dc-4cc7-89d2-64a1ffce8536 None None] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:08:55.494975 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b396a6ac-47da-4ef0-88ee-69e520c5bb39 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:08:55.495639 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b396a6ac-47da-4ef0-88ee-69e520c5bb39 admin admin] REQUEST_METHOD: `POST` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:08:55.495922 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b396a6ac-47da-4ef0-88ee-69e520c5bb39 admin admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:08:55.496223 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b396a6ac-47da-4ef0-88ee-69e520c5bb39 admin admin] PATH_INFO: `/v3/users` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:08:55.496889 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b396a6ac-47da-4ef0-88ee-69e520c5bb39 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:08:55.498856 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b396a6ac-47da-4ef0-88ee-69e520c5bb39 admin admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:08:55.576843 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 1254/2494] 10.222.0.30 () {60 vars in 1143 bytes} [Mon May 23 13:08:55 2022] POST /identity/v3/users => generated 300 bytes in 154 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) May 23 13:08:55.581051 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b755097-2a68-4619-88ea-dd4fac078d9b admin admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:08:55.591520 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b755097-2a68-4619-88ea-dd4fac078d9b admin admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:08:55.596916 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d64d316-1f28-420f-a14c-0030e39f99c0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:08:55.597578 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d64d316-1f28-420f-a14c-0030e39f99c0 admin admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:08:55.614380 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d64d316-1f28-420f-a14c-0030e39f99c0 admin admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:08:55.614380 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d64d316-1f28-420f-a14c-0030e39f99c0 admin admin] PATH_INFO: `/v3/roles` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:08:55.614380 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d64d316-1f28-420f-a14c-0030e39f99c0 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:08:55.614380 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d64d316-1f28-420f-a14c-0030e39f99c0 admin admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:08:55.614380 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 1241/2495] 10.222.0.30 () {58 vars in 1121 bytes} [Mon May 23 13:08:55 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) May 23 13:08:55.615943 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b396a6ac-47da-4ef0-88ee-69e520c5bb39 admin admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:08:55.626315 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b396a6ac-47da-4ef0-88ee-69e520c5bb39 admin admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:08:55.631279 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b952d19-8bdb-421a-b8d5-770802cf7e34 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:08:55.632166 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b952d19-8bdb-421a-b8d5-770802cf7e34 admin admin] REQUEST_METHOD: `PUT` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:08:55.632454 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b952d19-8bdb-421a-b8d5-770802cf7e34 admin admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:08:55.632766 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b952d19-8bdb-421a-b8d5-770802cf7e34 admin admin] PATH_INFO: `/v3/projects/abfb547132c049ec9d875706ae326463/users/4c8d617f86b54e69bd571d9d2ed95acc/roles/bc3c3c9d654940cbaa5177a6be23d1bc` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:08:55.649155 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b952d19-8bdb-421a-b8d5-770802cf7e34 admin admin] RBAC: Authorizing `identity:create_grant(project_id=abfb547132c049ec9d875706ae326463, user_id=4c8d617f86b54e69bd571d9d2ed95acc, role_id=bc3c3c9d654940cbaa5177a6be23d1bc)` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:08:55.651881 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b952d19-8bdb-421a-b8d5-770802cf7e34 admin admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:08:55.714687 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 1255/2496] 10.222.0.30 () {60 vars in 1482 bytes} [Mon May 23 13:08:55 2022] PUT /identity/v3/projects/abfb547132c049ec9d875706ae326463/users/4c8d617f86b54e69bd571d9d2ed95acc/roles/bc3c3c9d654940cbaa5177a6be23d1bc => generated 0 bytes in 99 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) May 23 13:08:55.718860 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d64d316-1f28-420f-a14c-0030e39f99c0 admin admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:08:55.729649 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d64d316-1f28-420f-a14c-0030e39f99c0 admin admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:08:55.788004 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f42bb506-b60e-4016-aba6-df522b9ae061 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:08:55.788675 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f42bb506-b60e-4016-aba6-df522b9ae061 admin admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:08:55.788955 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f42bb506-b60e-4016-aba6-df522b9ae061 admin admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:08:55.789236 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f42bb506-b60e-4016-aba6-df522b9ae061 admin admin] PATH_INFO: `/v3/roles` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:08:55.789991 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f42bb506-b60e-4016-aba6-df522b9ae061 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:08:55.791586 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f42bb506-b60e-4016-aba6-df522b9ae061 admin admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:08:55.806056 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 1242/2497] 10.222.0.30 () {58 vars in 1121 bytes} [Mon May 23 13:08:55 2022] GET /identity/v3/roles => generated 1420 bytes in 85 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) May 23 13:08:55.807128 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b952d19-8bdb-421a-b8d5-770802cf7e34 admin admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:08:55.812792 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=93,keystone:INSERT=7 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 13:08:55.818785 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b952d19-8bdb-421a-b8d5-770802cf7e34 admin admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:08:55.823575 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9fd6c325-cd8d-4e78-971a-16f46464453a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:08:55.824554 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fd6c325-cd8d-4e78-971a-16f46464453a admin admin] REQUEST_METHOD: `PUT` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:08:55.824909 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fd6c325-cd8d-4e78-971a-16f46464453a admin admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:08:55.825199 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fd6c325-cd8d-4e78-971a-16f46464453a admin admin] PATH_INFO: `/v3/projects/abfb547132c049ec9d875706ae326463/users/4c8d617f86b54e69bd571d9d2ed95acc/roles/bc3c3c9d654940cbaa5177a6be23d1bc` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:08:55.827225 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9fd6c325-cd8d-4e78-971a-16f46464453a admin admin] RBAC: Authorizing `identity:create_grant(project_id=abfb547132c049ec9d875706ae326463, user_id=4c8d617f86b54e69bd571d9d2ed95acc, role_id=bc3c3c9d654940cbaa5177a6be23d1bc)` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:08:55.830429 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9fd6c325-cd8d-4e78-971a-16f46464453a admin admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:08:55.840820 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 1256/2498] 10.222.0.30 () {60 vars in 1482 bytes} [Mon May 23 13:08:55 2022] PUT /identity/v3/projects/abfb547132c049ec9d875706ae326463/users/4c8d617f86b54e69bd571d9d2ed95acc/roles/bc3c3c9d654940cbaa5177a6be23d1bc => generated 0 bytes in 34 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) May 23 13:08:55.852050 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7750c08-8226-48f4-ac73-22abbf4c92e5 None None] REQUEST_METHOD: `POST` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:08:55.852329 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7750c08-8226-48f4-ac73-22abbf4c92e5 None None] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:08:55.857327 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7750c08-8226-48f4-ac73-22abbf4c92e5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:08:55.879871 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.auth.core [None req-e7750c08-8226-48f4-ac73-22abbf4c92e5 None None] MFA Rules not processed for user `4c8d617f86b54e69bd571d9d2ed95acc`. Rule list: `[]` (Enabled: `True`). {{(pid=92056) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 13:08:55.881184 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.fernet_utils [None req-e7750c08-8226-48f4-ac73-22abbf4c92e5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92056) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:08:55.951475 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 1243/2499] 10.222.0.30 () {58 vars in 957 bytes} [Mon May 23 13:08:55 2022] POST /identity/v3/auth/tokens => generated 3321 bytes in 100 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) May 23 13:08:55.963567 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9fd6c325-cd8d-4e78-971a-16f46464453a admin admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:08:55.965272 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.fernet_utils [None req-9fd6c325-cd8d-4e78-971a-16f46464453a admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92057) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:08:56.041171 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9fd6c325-cd8d-4e78-971a-16f46464453a admin admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:08:56.044925 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c025f303-44f2-4c65-981a-01aeea29b797 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1666097f8ed545c6b0e4809c93e4775a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9c389ae65aaa4542bb5a5c4fd0bacf5b', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:08:56.045560 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c025f303-44f2-4c65-981a-01aeea29b797 service cinder] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:08:56.045843 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c025f303-44f2-4c65-981a-01aeea29b797 service cinder] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:08:56.046152 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c025f303-44f2-4c65-981a-01aeea29b797 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:08:56.056053 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c025f303-44f2-4c65-981a-01aeea29b797 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:08:56.057578 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c025f303-44f2-4c65-981a-01aeea29b797 service cinder] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:08:56.070612 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 1257/2500] 10.222.0.30 () {60 vars in 1365 bytes} [Mon May 23 13:08:55 2022] GET /identity/v3/auth/tokens => generated 3321 bytes in 107 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) May 23 13:08:56.099329 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7750c08-8226-48f4-ac73-22abbf4c92e5 None None] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:08:56.154400 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 13:08:56.166752 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7750c08-8226-48f4-ac73-22abbf4c92e5 None None] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:08:56.171340 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-642d67e5-5909-493e-b956-f7adbc37eff2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:08:56.172197 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-642d67e5-5909-493e-b956-f7adbc37eff2 admin admin] REQUEST_METHOD: `DELETE` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:08:56.172480 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-642d67e5-5909-493e-b956-f7adbc37eff2 admin admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:08:56.172757 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-642d67e5-5909-493e-b956-f7adbc37eff2 admin admin] PATH_INFO: `/v3/users/b30c02ae86da42b1a5f036a3faec2bf0` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:08:56.174475 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-642d67e5-5909-493e-b956-f7adbc37eff2 admin admin] RBAC: Authorizing `identity:delete_user(user_id=b30c02ae86da42b1a5f036a3faec2bf0)` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:08:56.176443 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-642d67e5-5909-493e-b956-f7adbc37eff2 admin admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:08:56.346288 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-642d67e5-5909-493e-b956-f7adbc37eff2 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'b30c02ae86da42b1a5f036a3faec2bf0'} {{(pid=92056) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:08:56.346967 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-642d67e5-5909-493e-b956-f7adbc37eff2 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'b30c02ae86da42b1a5f036a3faec2bf0'} {{(pid=92056) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:08:56.462216 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-642d67e5-5909-493e-b956-f7adbc37eff2 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'b30c02ae86da42b1a5f036a3faec2bf0'} {{(pid=92056) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:08:56.477016 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-642d67e5-5909-493e-b956-f7adbc37eff2 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'b30c02ae86da42b1a5f036a3faec2bf0'} {{(pid=92056) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:08:56.484151 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-642d67e5-5909-493e-b956-f7adbc37eff2 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'b30c02ae86da42b1a5f036a3faec2bf0'} {{(pid=92056) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:08:56.490949 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 1244/2501] 10.222.0.30 () {58 vars in 1223 bytes} [Mon May 23 13:08:56 2022] DELETE /identity/v3/users/b30c02ae86da42b1a5f036a3faec2bf0 => generated 0 bytes in 392 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) May 23 13:08:56.508114 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51489489-d4a1-403b-b20e-796f3bba64f4 None None] REQUEST_METHOD: `POST` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:08:56.522246 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51489489-d4a1-403b-b20e-796f3bba64f4 None None] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:08:56.522246 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51489489-d4a1-403b-b20e-796f3bba64f4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:08:56.528836 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.auth.core [None req-51489489-d4a1-403b-b20e-796f3bba64f4 None None] MFA Rules not processed for user `0f7119f8fa5c4445977671140ded91a4`. Rule list: `[]` (Enabled: `True`). {{(pid=92057) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 13:08:56.530122 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.fernet_utils [None req-51489489-d4a1-403b-b20e-796f3bba64f4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92057) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:08:56.592763 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 1258/2502] 10.222.0.30 () {58 vars in 957 bytes} [Mon May 23 13:08:56 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) May 23 13:08:56.602800 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-642d67e5-5909-493e-b956-f7adbc37eff2 admin admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:08:56.604314 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.fernet_utils [None req-642d67e5-5909-493e-b956-f7adbc37eff2 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92056) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:08:56.661640 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-642d67e5-5909-493e-b956-f7adbc37eff2 admin admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:08:56.665335 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a69d2f8d-141f-4201-b9e2-56cbe6abd4cd service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2eb83ca38e79449095f3a2f947bbb06a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9c389ae65aaa4542bb5a5c4fd0bacf5b', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:08:56.666081 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a69d2f8d-141f-4201-b9e2-56cbe6abd4cd service neutron] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:08:56.666367 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a69d2f8d-141f-4201-b9e2-56cbe6abd4cd service neutron] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:08:56.666646 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a69d2f8d-141f-4201-b9e2-56cbe6abd4cd service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:08:56.676687 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a69d2f8d-141f-4201-b9e2-56cbe6abd4cd service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:08:56.682795 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a69d2f8d-141f-4201-b9e2-56cbe6abd4cd service neutron] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:08:56.688291 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 1245/2503] 10.222.0.30 () {60 vars in 1365 bytes} [Mon May 23 13:08:56 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 86 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) May 23 13:08:57.116517 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51489489-d4a1-403b-b20e-796f3bba64f4 None None] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:08:57.118416 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.fernet_utils [None req-51489489-d4a1-403b-b20e-796f3bba64f4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92057) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:08:57.132051 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51489489-d4a1-403b-b20e-796f3bba64f4 None None] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:08:57.139692 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bdd35be0-b7f2-4e8c-bf72-387f43f17440 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:08:57.140658 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdd35be0-b7f2-4e8c-bf72-387f43f17440 admin admin] REQUEST_METHOD: `DELETE` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:08:57.140938 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdd35be0-b7f2-4e8c-bf72-387f43f17440 admin admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:08:57.141207 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdd35be0-b7f2-4e8c-bf72-387f43f17440 admin admin] PATH_INFO: `/v3/projects/79666b25325f459b8b38dff1565cbabb` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:08:57.142451 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bdd35be0-b7f2-4e8c-bf72-387f43f17440 admin admin] RBAC: Authorizing `identity:delete_project(project_id=79666b25325f459b8b38dff1565cbabb)` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:08:57.145023 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bdd35be0-b7f2-4e8c-bf72-387f43f17440 admin admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:08:57.270372 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [None req-bdd35be0-b7f2-4e8c-bf72-387f43f17440 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '79666b25325f459b8b38dff1565cbabb', 'request_id': 'req-bdd35be0-b7f2-4e8c-bf72-387f43f17440'} {{(pid=92057) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:08:57.277057 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [None req-bdd35be0-b7f2-4e8c-bf72-387f43f17440 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '79666b25325f459b8b38dff1565cbabb', 'request_id': 'req-bdd35be0-b7f2-4e8c-bf72-387f43f17440'} {{(pid=92057) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:08:57.334303 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [None req-bdd35be0-b7f2-4e8c-bf72-387f43f17440 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '79666b25325f459b8b38dff1565cbabb', 'request_id': 'req-bdd35be0-b7f2-4e8c-bf72-387f43f17440'} {{(pid=92057) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:08:57.346246 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [None req-bdd35be0-b7f2-4e8c-bf72-387f43f17440 admin admin] The token cache is being invalidate because project 79666b25325f459b8b38dff1565cbabb was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=92057) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} May 23 13:08:57.346486 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [None req-bdd35be0-b7f2-4e8c-bf72-387f43f17440 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=92057) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:08:57.347200 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [None req-bdd35be0-b7f2-4e8c-bf72-387f43f17440 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=92057) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:08:57.348749 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 1259/2504] 10.222.0.30 () {58 vars in 1232 bytes} [Mon May 23 13:08:57 2022] DELETE /identity/v3/projects/79666b25325f459b8b38dff1565cbabb => generated 0 bytes in 233 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) May 23 13:08:57.352805 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a69d2f8d-141f-4201-b9e2-56cbe6abd4cd service neutron] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:08:57.362695 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a69d2f8d-141f-4201-b9e2-56cbe6abd4cd service neutron] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:08:57.419340 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb955fcc-e55e-4dee-9199-9b2334d04b35 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:08:57.420301 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb955fcc-e55e-4dee-9199-9b2334d04b35 admin admin] REQUEST_METHOD: `DELETE` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:08:57.420574 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb955fcc-e55e-4dee-9199-9b2334d04b35 admin admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:08:57.420849 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb955fcc-e55e-4dee-9199-9b2334d04b35 admin admin] PATH_INFO: `/v3/users/4c8d617f86b54e69bd571d9d2ed95acc` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:08:57.422110 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb955fcc-e55e-4dee-9199-9b2334d04b35 admin admin] RBAC: Authorizing `identity:delete_user(user_id=4c8d617f86b54e69bd571d9d2ed95acc)` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:08:57.424017 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb955fcc-e55e-4dee-9199-9b2334d04b35 admin admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:08:57.438883 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=88,keystone:DELETE=11,keystone:INSERT=1 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 13:08:57.659617 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-fb955fcc-e55e-4dee-9199-9b2334d04b35 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '4c8d617f86b54e69bd571d9d2ed95acc'} {{(pid=92056) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:08:57.660472 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-fb955fcc-e55e-4dee-9199-9b2334d04b35 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '4c8d617f86b54e69bd571d9d2ed95acc'} {{(pid=92056) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:08:57.742080 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-fb955fcc-e55e-4dee-9199-9b2334d04b35 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '4c8d617f86b54e69bd571d9d2ed95acc'} {{(pid=92056) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:08:57.755023 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-fb955fcc-e55e-4dee-9199-9b2334d04b35 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '4c8d617f86b54e69bd571d9d2ed95acc'} {{(pid=92056) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:08:57.761680 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-fb955fcc-e55e-4dee-9199-9b2334d04b35 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '4c8d617f86b54e69bd571d9d2ed95acc'} {{(pid=92056) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:08:57.767530 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 1246/2505] 10.222.0.30 () {58 vars in 1223 bytes} [Mon May 23 13:08:57 2022] DELETE /identity/v3/users/4c8d617f86b54e69bd571d9d2ed95acc => generated 0 bytes in 415 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) May 23 13:08:58.263948 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bdd35be0-b7f2-4e8c-bf72-387f43f17440 admin admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:08:58.266027 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.fernet_utils [None req-bdd35be0-b7f2-4e8c-bf72-387f43f17440 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92057) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:08:58.281176 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bdd35be0-b7f2-4e8c-bf72-387f43f17440 admin admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:08:58.307516 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=91,keystone:INSERT=2,keystone:DELETE=7 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 13:08:58.353239 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7810b4b0-2bad-4a2d-a7e5-3ca8df3cdc9a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:08:58.354141 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7810b4b0-2bad-4a2d-a7e5-3ca8df3cdc9a admin admin] REQUEST_METHOD: `DELETE` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:08:58.354426 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7810b4b0-2bad-4a2d-a7e5-3ca8df3cdc9a admin admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:08:58.354701 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7810b4b0-2bad-4a2d-a7e5-3ca8df3cdc9a admin admin] PATH_INFO: `/v3/projects/abfb547132c049ec9d875706ae326463` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:08:58.355927 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7810b4b0-2bad-4a2d-a7e5-3ca8df3cdc9a admin admin] RBAC: Authorizing `identity:delete_project(project_id=abfb547132c049ec9d875706ae326463)` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:08:58.357843 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7810b4b0-2bad-4a2d-a7e5-3ca8df3cdc9a admin admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:08:58.445727 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [None req-7810b4b0-2bad-4a2d-a7e5-3ca8df3cdc9a admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'abfb547132c049ec9d875706ae326463', 'request_id': 'req-7810b4b0-2bad-4a2d-a7e5-3ca8df3cdc9a'} {{(pid=92057) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:08:58.451385 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [None req-7810b4b0-2bad-4a2d-a7e5-3ca8df3cdc9a admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'abfb547132c049ec9d875706ae326463', 'request_id': 'req-7810b4b0-2bad-4a2d-a7e5-3ca8df3cdc9a'} {{(pid=92057) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:08:58.501837 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [None req-7810b4b0-2bad-4a2d-a7e5-3ca8df3cdc9a admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'abfb547132c049ec9d875706ae326463', 'request_id': 'req-7810b4b0-2bad-4a2d-a7e5-3ca8df3cdc9a'} {{(pid=92057) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:08:58.513853 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [None req-7810b4b0-2bad-4a2d-a7e5-3ca8df3cdc9a admin admin] The token cache is being invalidate because project abfb547132c049ec9d875706ae326463 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=92057) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} May 23 13:08:58.514260 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [None req-7810b4b0-2bad-4a2d-a7e5-3ca8df3cdc9a admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=92057) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:08:58.514914 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [None req-7810b4b0-2bad-4a2d-a7e5-3ca8df3cdc9a admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=92057) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:08:58.520392 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 1260/2506] 10.222.0.30 () {58 vars in 1232 bytes} [Mon May 23 13:08:58 2022] DELETE /identity/v3/projects/abfb547132c049ec9d875706ae326463 => generated 0 bytes in 253 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) May 23 13:08:58.522105 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-071b1ee5-3a20-477b-b3d4-8554c3da984f None None] REQUEST_METHOD: `POST` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:08:58.522400 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-071b1ee5-3a20-477b-b3d4-8554c3da984f None None] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:08:58.523491 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-071b1ee5-3a20-477b-b3d4-8554c3da984f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:08:58.542031 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.auth.core [None req-071b1ee5-3a20-477b-b3d4-8554c3da984f None None] MFA Rules not processed for user `0f7119f8fa5c4445977671140ded91a4`. Rule list: `[]` (Enabled: `True`). {{(pid=92056) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 13:08:58.543242 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.fernet_utils [None req-071b1ee5-3a20-477b-b3d4-8554c3da984f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92056) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:08:58.610663 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 1247/2507] 10.222.0.30 () {58 vars in 957 bytes} [Mon May 23 13:08:58 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 90 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) May 23 13:08:58.618686 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ce89ab8-719e-44d6-ad9e-b9156b4e5ba9 None None] REQUEST_METHOD: `POST` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:08:58.622051 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ce89ab8-719e-44d6-ad9e-b9156b4e5ba9 None None] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:08:58.622051 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ce89ab8-719e-44d6-ad9e-b9156b4e5ba9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:08:58.637906 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.auth.core [None req-2ce89ab8-719e-44d6-ad9e-b9156b4e5ba9 None None] MFA Rules not processed for user `0f7119f8fa5c4445977671140ded91a4`. Rule list: `[]` (Enabled: `True`). {{(pid=92057) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 13:08:58.639099 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.fernet_utils [None req-2ce89ab8-719e-44d6-ad9e-b9156b4e5ba9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92057) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:08:58.647930 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 1261/2508] 10.222.0.30 () {58 vars in 957 bytes} [Mon May 23 13:08:58 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 30 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) May 23 13:08:58.652117 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-071b1ee5-3a20-477b-b3d4-8554c3da984f None None] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:08:58.662229 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-071b1ee5-3a20-477b-b3d4-8554c3da984f None None] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:08:58.666951 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9412a07a-fc8e-4711-a880-68002ff7b309 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:08:58.672202 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9412a07a-fc8e-4711-a880-68002ff7b309 admin admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:08:58.672202 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9412a07a-fc8e-4711-a880-68002ff7b309 admin admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:08:58.672202 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9412a07a-fc8e-4711-a880-68002ff7b309 admin admin] PATH_INFO: `/v3/domains` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:08:58.672202 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9412a07a-fc8e-4711-a880-68002ff7b309 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=92056) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} May 23 13:08:58.672202 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9412a07a-fc8e-4711-a880-68002ff7b309 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:08:58.672202 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9412a07a-fc8e-4711-a880-68002ff7b309 admin admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:08:58.684419 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 1248/2509] 10.222.0.30 () {58 vars in 1152 bytes} [Mon May 23 13:08:58 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) May 23 13:08:58.693840 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ce89ab8-719e-44d6-ad9e-b9156b4e5ba9 None None] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:08:58.699266 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ce89ab8-719e-44d6-ad9e-b9156b4e5ba9 None None] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:08:58.703902 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dcfe790d-2017-4243-84ee-688c4b04555f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:08:58.704677 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dcfe790d-2017-4243-84ee-688c4b04555f admin admin] REQUEST_METHOD: `POST` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:08:58.704960 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dcfe790d-2017-4243-84ee-688c4b04555f admin admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:08:58.705230 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dcfe790d-2017-4243-84ee-688c4b04555f admin admin] PATH_INFO: `/v3/projects` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:08:58.706829 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dcfe790d-2017-4243-84ee-688c4b04555f admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:08:58.708612 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dcfe790d-2017-4243-84ee-688c4b04555f admin admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:08:58.807555 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 1262/2510] 10.222.0.30 () {60 vars in 1152 bytes} [Mon May 23 13:08:58 2022] POST /identity/v3/projects => generated 381 bytes in 120 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) May 23 13:08:58.812035 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9412a07a-fc8e-4711-a880-68002ff7b309 admin admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:08:58.823388 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9412a07a-fc8e-4711-a880-68002ff7b309 admin admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:08:58.895375 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44a6dfdd-9ad1-4fe3-b06e-20b8f84e72d2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:08:58.898050 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44a6dfdd-9ad1-4fe3-b06e-20b8f84e72d2 admin admin] REQUEST_METHOD: `POST` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:08:58.898050 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44a6dfdd-9ad1-4fe3-b06e-20b8f84e72d2 admin admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:08:58.898050 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44a6dfdd-9ad1-4fe3-b06e-20b8f84e72d2 admin admin] PATH_INFO: `/v3/users` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:08:58.898050 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44a6dfdd-9ad1-4fe3-b06e-20b8f84e72d2 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:08:58.899335 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44a6dfdd-9ad1-4fe3-b06e-20b8f84e72d2 admin admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:08:58.983608 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 1249/2511] 10.222.0.30 () {60 vars in 1143 bytes} [Mon May 23 13:08:58 2022] POST /identity/v3/users => generated 291 bytes in 169 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) May 23 13:08:58.984821 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dcfe790d-2017-4243-84ee-688c4b04555f admin admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:08:58.997029 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dcfe790d-2017-4243-84ee-688c4b04555f admin admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:08:59.004905 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28b2e03d-06c5-4618-b05d-df84cdd4a91c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:08:59.005041 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28b2e03d-06c5-4618-b05d-df84cdd4a91c admin admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:08:59.005340 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28b2e03d-06c5-4618-b05d-df84cdd4a91c admin admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:08:59.005559 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28b2e03d-06c5-4618-b05d-df84cdd4a91c admin admin] PATH_INFO: `/v3/roles` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:08:59.006374 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28b2e03d-06c5-4618-b05d-df84cdd4a91c admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:08:59.008628 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28b2e03d-06c5-4618-b05d-df84cdd4a91c admin admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:08:59.023375 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 1263/2512] 10.222.0.30 () {58 vars in 1121 bytes} [Mon May 23 13:08:58 2022] GET /identity/v3/roles => generated 1420 bytes in 36 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) May 23 13:08:59.024957 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44a6dfdd-9ad1-4fe3-b06e-20b8f84e72d2 admin admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:08:59.036051 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44a6dfdd-9ad1-4fe3-b06e-20b8f84e72d2 admin admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:08:59.040931 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-efc097dc-ab87-40a7-8178-bbef47071c61 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:08:59.041633 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efc097dc-ab87-40a7-8178-bbef47071c61 admin admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:08:59.041956 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efc097dc-ab87-40a7-8178-bbef47071c61 admin admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:08:59.042242 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efc097dc-ab87-40a7-8178-bbef47071c61 admin admin] PATH_INFO: `/v3/roles` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:08:59.042994 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-efc097dc-ab87-40a7-8178-bbef47071c61 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:08:59.044641 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-efc097dc-ab87-40a7-8178-bbef47071c61 admin admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:08:59.047145 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=87,keystone:DELETE=9,keystone:INSERT=4 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 13:08:59.056329 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 1250/2513] 10.222.0.30 () {58 vars in 1121 bytes} [Mon May 23 13:08:59 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) May 23 13:08:59.067020 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28b2e03d-06c5-4618-b05d-df84cdd4a91c admin admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:08:59.078209 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28b2e03d-06c5-4618-b05d-df84cdd4a91c admin admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:08:59.084452 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f6aa7f8-82b7-409a-90b1-8461db28a200 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:08:59.085414 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f6aa7f8-82b7-409a-90b1-8461db28a200 admin admin] REQUEST_METHOD: `PUT` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:08:59.085777 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f6aa7f8-82b7-409a-90b1-8461db28a200 admin admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:08:59.086160 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f6aa7f8-82b7-409a-90b1-8461db28a200 admin admin] PATH_INFO: `/v3/projects/9cb259b3f8c240519ececd93edf16fc9/users/1808c319697549a292777f1b6a482f35/roles/80fb22507c9a4ff09d44d8f12f6fff1a` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:08:59.101711 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f6aa7f8-82b7-409a-90b1-8461db28a200 admin admin] RBAC: Authorizing `identity:create_grant(project_id=9cb259b3f8c240519ececd93edf16fc9, user_id=1808c319697549a292777f1b6a482f35, role_id=80fb22507c9a4ff09d44d8f12f6fff1a)` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:08:59.104593 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f6aa7f8-82b7-409a-90b1-8461db28a200 admin admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:08:59.175432 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 1264/2514] 10.222.0.30 () {60 vars in 1482 bytes} [Mon May 23 13:08:59 2022] PUT /identity/v3/projects/9cb259b3f8c240519ececd93edf16fc9/users/1808c319697549a292777f1b6a482f35/roles/80fb22507c9a4ff09d44d8f12f6fff1a => generated 0 bytes in 109 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) May 23 13:08:59.187129 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-418e0a9d-34de-4f32-8367-ad8890d4122e None None] REQUEST_METHOD: `POST` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:08:59.187475 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-418e0a9d-34de-4f32-8367-ad8890d4122e None None] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:08:59.187762 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-418e0a9d-34de-4f32-8367-ad8890d4122e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:08:59.215427 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.auth.core [None req-418e0a9d-34de-4f32-8367-ad8890d4122e None None] MFA Rules not processed for user `1808c319697549a292777f1b6a482f35`. Rule list: `[]` (Enabled: `True`). {{(pid=92056) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 13:08:59.216643 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.fernet_utils [None req-418e0a9d-34de-4f32-8367-ad8890d4122e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92056) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:08:59.294223 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 1251/2515] 10.222.0.30 () {58 vars in 957 bytes} [Mon May 23 13:08:59 2022] POST /identity/v3/auth/tokens => generated 3248 bytes in 108 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) May 23 13:08:59.298857 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f6aa7f8-82b7-409a-90b1-8461db28a200 admin admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:08:59.349082 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 13:08:59.365617 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f6aa7f8-82b7-409a-90b1-8461db28a200 admin admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:08:59.370473 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ffb60ed-0410-4b70-a6dc-2502a0faac1f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:08:59.371127 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ffb60ed-0410-4b70-a6dc-2502a0faac1f admin admin] REQUEST_METHOD: `POST` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:08:59.371412 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ffb60ed-0410-4b70-a6dc-2502a0faac1f admin admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:08:59.371687 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ffb60ed-0410-4b70-a6dc-2502a0faac1f admin admin] PATH_INFO: `/v3/projects` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:08:59.372380 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ffb60ed-0410-4b70-a6dc-2502a0faac1f admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:08:59.374388 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ffb60ed-0410-4b70-a6dc-2502a0faac1f admin admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:08:59.517457 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 1265/2516] 10.222.0.30 () {60 vars in 1152 bytes} [Mon May 23 13:08:59 2022] POST /identity/v3/projects => generated 381 bytes in 216 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) May 23 13:08:59.519327 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-418e0a9d-34de-4f32-8367-ad8890d4122e None None] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:08:59.532963 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-418e0a9d-34de-4f32-8367-ad8890d4122e None None] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:08:59.594911 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28ca6295-6fc7-4a79-87d5-e457bfc0f939 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:08:59.595703 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28ca6295-6fc7-4a79-87d5-e457bfc0f939 admin admin] REQUEST_METHOD: `POST` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:08:59.596050 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28ca6295-6fc7-4a79-87d5-e457bfc0f939 admin admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:08:59.596329 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28ca6295-6fc7-4a79-87d5-e457bfc0f939 admin admin] PATH_INFO: `/v3/users` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:08:59.597594 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28ca6295-6fc7-4a79-87d5-e457bfc0f939 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:08:59.599584 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28ca6295-6fc7-4a79-87d5-e457bfc0f939 admin admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:08:59.701003 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 1252/2517] 10.222.0.30 () {60 vars in 1143 bytes} [Mon May 23 13:08:59 2022] POST /identity/v3/users => generated 297 bytes in 183 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) May 23 13:08:59.705123 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ffb60ed-0410-4b70-a6dc-2502a0faac1f admin admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:08:59.717344 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ffb60ed-0410-4b70-a6dc-2502a0faac1f admin admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:08:59.725125 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f843342-cd9a-4550-ac63-1363eb0026cb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:08:59.729218 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f843342-cd9a-4550-ac63-1363eb0026cb admin admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:08:59.729450 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f843342-cd9a-4550-ac63-1363eb0026cb admin admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:08:59.729747 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f843342-cd9a-4550-ac63-1363eb0026cb admin admin] PATH_INFO: `/v3/roles` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:08:59.730526 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f843342-cd9a-4550-ac63-1363eb0026cb admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:08:59.732294 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f843342-cd9a-4550-ac63-1363eb0026cb admin admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:08:59.748637 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 1266/2518] 10.222.0.30 () {58 vars in 1121 bytes} [Mon May 23 13:08:59 2022] GET /identity/v3/roles => generated 1420 bytes in 41 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) May 23 13:08:59.754323 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28ca6295-6fc7-4a79-87d5-e457bfc0f939 admin admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:08:59.773351 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28ca6295-6fc7-4a79-87d5-e457bfc0f939 admin admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:08:59.778714 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4d5d079-4514-4728-b1cf-e885caf2cff6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:08:59.779648 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4d5d079-4514-4728-b1cf-e885caf2cff6 admin admin] REQUEST_METHOD: `PUT` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:08:59.779935 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4d5d079-4514-4728-b1cf-e885caf2cff6 admin admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:08:59.780237 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4d5d079-4514-4728-b1cf-e885caf2cff6 admin admin] PATH_INFO: `/v3/projects/85f4c27d81d140c6b8cf6a674610697d/users/a38e93d4a5fd44dbbb7699d34dd99715/roles/bc3c3c9d654940cbaa5177a6be23d1bc` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:08:59.798509 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4d5d079-4514-4728-b1cf-e885caf2cff6 admin admin] RBAC: Authorizing `identity:create_grant(project_id=85f4c27d81d140c6b8cf6a674610697d, user_id=a38e93d4a5fd44dbbb7699d34dd99715, role_id=bc3c3c9d654940cbaa5177a6be23d1bc)` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:08:59.801923 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4d5d079-4514-4728-b1cf-e885caf2cff6 admin admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:08:59.885956 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 1253/2519] 10.222.0.30 () {60 vars in 1482 bytes} [Mon May 23 13:08:59 2022] PUT /identity/v3/projects/85f4c27d81d140c6b8cf6a674610697d/users/a38e93d4a5fd44dbbb7699d34dd99715/roles/bc3c3c9d654940cbaa5177a6be23d1bc => generated 0 bytes in 134 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) May 23 13:08:59.892045 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f843342-cd9a-4550-ac63-1363eb0026cb admin admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:08:59.902093 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f843342-cd9a-4550-ac63-1363eb0026cb admin admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:08:59.962789 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91e9d44b-51c9-4144-9538-084fb0ef6de8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:08:59.963638 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91e9d44b-51c9-4144-9538-084fb0ef6de8 admin admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:08:59.963924 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91e9d44b-51c9-4144-9538-084fb0ef6de8 admin admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:08:59.964200 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91e9d44b-51c9-4144-9538-084fb0ef6de8 admin admin] PATH_INFO: `/v3/roles` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:08:59.964955 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-91e9d44b-51c9-4144-9538-084fb0ef6de8 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:08:59.966615 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-91e9d44b-51c9-4144-9538-084fb0ef6de8 admin admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:08:59.981098 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 1267/2520] 10.222.0.30 () {58 vars in 1121 bytes} [Mon May 23 13:08:59 2022] GET /identity/v3/roles => generated 1420 bytes in 89 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) May 23 13:08:59.982389 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4d5d079-4514-4728-b1cf-e885caf2cff6 admin admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:08:59.993405 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4d5d079-4514-4728-b1cf-e885caf2cff6 admin admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:08:59.998351 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-706ff384-8394-4fc0-bdaf-e9424eebe851 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:08:59.999322 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-706ff384-8394-4fc0-bdaf-e9424eebe851 admin admin] REQUEST_METHOD: `PUT` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:08:59.999669 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-706ff384-8394-4fc0-bdaf-e9424eebe851 admin admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:09:00.000010 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-706ff384-8394-4fc0-bdaf-e9424eebe851 admin admin] PATH_INFO: `/v3/projects/85f4c27d81d140c6b8cf6a674610697d/users/a38e93d4a5fd44dbbb7699d34dd99715/roles/bc3c3c9d654940cbaa5177a6be23d1bc` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:09:00.002092 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-706ff384-8394-4fc0-bdaf-e9424eebe851 admin admin] RBAC: Authorizing `identity:create_grant(project_id=85f4c27d81d140c6b8cf6a674610697d, user_id=a38e93d4a5fd44dbbb7699d34dd99715, role_id=bc3c3c9d654940cbaa5177a6be23d1bc)` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:09:00.004833 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-706ff384-8394-4fc0-bdaf-e9424eebe851 admin admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:09:00.015480 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 1254/2521] 10.222.0.30 () {60 vars in 1482 bytes} [Mon May 23 13:08:59 2022] PUT /identity/v3/projects/85f4c27d81d140c6b8cf6a674610697d/users/a38e93d4a5fd44dbbb7699d34dd99715/roles/bc3c3c9d654940cbaa5177a6be23d1bc => generated 0 bytes in 34 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) May 23 13:09:00.026359 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-037767ee-774f-4330-a695-982def811839 None None] REQUEST_METHOD: `POST` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:09:00.026734 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-037767ee-774f-4330-a695-982def811839 None None] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:09:00.027084 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-037767ee-774f-4330-a695-982def811839 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:09:00.054449 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.auth.core [None req-037767ee-774f-4330-a695-982def811839 None None] MFA Rules not processed for user `a38e93d4a5fd44dbbb7699d34dd99715`. Rule list: `[]` (Enabled: `True`). {{(pid=92057) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 13:09:00.055762 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.fernet_utils [None req-037767ee-774f-4330-a695-982def811839 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92057) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:09:00.128423 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 1268/2522] 10.222.0.30 () {58 vars in 957 bytes} [Mon May 23 13:09:00 2022] POST /identity/v3/auth/tokens => generated 3315 bytes in 103 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) May 23 13:09:00.144642 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-706ff384-8394-4fc0-bdaf-e9424eebe851 admin admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:09:00.146464 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.fernet_utils [None req-706ff384-8394-4fc0-bdaf-e9424eebe851 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92056) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:09:00.192722 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 13:09:00.208693 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-706ff384-8394-4fc0-bdaf-e9424eebe851 admin admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:09:00.212449 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb58c0c3-f60c-49e8-a2fa-22a143e0034b service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1666097f8ed545c6b0e4809c93e4775a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9c389ae65aaa4542bb5a5c4fd0bacf5b', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:09:00.213094 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb58c0c3-f60c-49e8-a2fa-22a143e0034b service cinder] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:09:00.213374 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb58c0c3-f60c-49e8-a2fa-22a143e0034b service cinder] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:09:00.213648 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb58c0c3-f60c-49e8-a2fa-22a143e0034b service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:09:00.222900 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb58c0c3-f60c-49e8-a2fa-22a143e0034b service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:09:00.224340 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb58c0c3-f60c-49e8-a2fa-22a143e0034b service cinder] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:09:00.234290 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 1255/2523] 10.222.0.30 () {60 vars in 1365 bytes} [Mon May 23 13:09:00 2022] GET /identity/v3/auth/tokens => generated 3315 bytes in 93 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) May 23 13:09:01.217542 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-037767ee-774f-4330-a695-982def811839 None None] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:09:01.227400 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-037767ee-774f-4330-a695-982def811839 None None] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:09:01.231861 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cfb10968-6d64-4d6b-9d0d-9ed885ac1391 tempest-VolumeQuotaClassesTest-1411907492 tempest-VolumeQuotaClassesTest-1411907492-project-admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a38e93d4a5fd44dbbb7699d34dd99715', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '85f4c27d81d140c6b8cf6a674610697d', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:09:01.232509 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfb10968-6d64-4d6b-9d0d-9ed885ac1391 tempest-VolumeQuotaClassesTest-1411907492 tempest-VolumeQuotaClassesTest-1411907492-project-admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:09:01.232795 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfb10968-6d64-4d6b-9d0d-9ed885ac1391 tempest-VolumeQuotaClassesTest-1411907492 tempest-VolumeQuotaClassesTest-1411907492-project-admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:09:01.233078 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfb10968-6d64-4d6b-9d0d-9ed885ac1391 tempest-VolumeQuotaClassesTest-1411907492 tempest-VolumeQuotaClassesTest-1411907492-project-admin] PATH_INFO: `/v3/domains` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:09:01.233790 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cfb10968-6d64-4d6b-9d0d-9ed885ac1391 tempest-VolumeQuotaClassesTest-1411907492 tempest-VolumeQuotaClassesTest-1411907492-project-admin] RBAC: Adding query filter params (name=Default) {{(pid=92057) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} May 23 13:09:01.234143 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cfb10968-6d64-4d6b-9d0d-9ed885ac1391 tempest-VolumeQuotaClassesTest-1411907492 tempest-VolumeQuotaClassesTest-1411907492-project-admin] RBAC: Authorizing `identity:list_domains()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:09:01.235759 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cfb10968-6d64-4d6b-9d0d-9ed885ac1391 tempest-VolumeQuotaClassesTest-1411907492 tempest-VolumeQuotaClassesTest-1411907492-project-admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:09:01.247371 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 1269/2524] 10.222.0.30 () {58 vars in 1152 bytes} [Mon May 23 13:09:01 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) May 23 13:09:01.251498 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb58c0c3-f60c-49e8-a2fa-22a143e0034b service cinder] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:09:01.260627 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb58c0c3-f60c-49e8-a2fa-22a143e0034b service cinder] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:09:01.265354 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2c7e08b-84bf-4dff-b2b6-a35b4bb3dd0c tempest-VolumeQuotaClassesTest-1411907492 tempest-VolumeQuotaClassesTest-1411907492-project-admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a38e93d4a5fd44dbbb7699d34dd99715', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '85f4c27d81d140c6b8cf6a674610697d', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:09:01.266062 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2c7e08b-84bf-4dff-b2b6-a35b4bb3dd0c tempest-VolumeQuotaClassesTest-1411907492 tempest-VolumeQuotaClassesTest-1411907492-project-admin] REQUEST_METHOD: `POST` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:09:01.266347 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2c7e08b-84bf-4dff-b2b6-a35b4bb3dd0c tempest-VolumeQuotaClassesTest-1411907492 tempest-VolumeQuotaClassesTest-1411907492-project-admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:09:01.266625 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2c7e08b-84bf-4dff-b2b6-a35b4bb3dd0c tempest-VolumeQuotaClassesTest-1411907492 tempest-VolumeQuotaClassesTest-1411907492-project-admin] PATH_INFO: `/v3/projects` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:09:01.267353 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2c7e08b-84bf-4dff-b2b6-a35b4bb3dd0c tempest-VolumeQuotaClassesTest-1411907492 tempest-VolumeQuotaClassesTest-1411907492-project-admin] RBAC: Authorizing `identity:create_project()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:09:01.269978 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2c7e08b-84bf-4dff-b2b6-a35b4bb3dd0c tempest-VolumeQuotaClassesTest-1411907492 tempest-VolumeQuotaClassesTest-1411907492-project-admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:09:01.343197 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 1256/2525] 10.222.0.30 () {60 vars in 1152 bytes} [Mon May 23 13:09:01 2022] POST /identity/v3/projects => generated 354 bytes in 92 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) May 23 13:09:01.348308 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cfb10968-6d64-4d6b-9d0d-9ed885ac1391 tempest-VolumeQuotaClassesTest-1411907492 tempest-VolumeQuotaClassesTest-1411907492-project-admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:09:01.358290 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cfb10968-6d64-4d6b-9d0d-9ed885ac1391 tempest-VolumeQuotaClassesTest-1411907492 tempest-VolumeQuotaClassesTest-1411907492-project-admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:09:01.378549 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 13:09:01.417455 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f91fbad-afe0-4bc2-816e-8ca46ab42020 tempest-VolumeQuotaClassesTest-1411907492 tempest-VolumeQuotaClassesTest-1411907492-project-admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a38e93d4a5fd44dbbb7699d34dd99715', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '85f4c27d81d140c6b8cf6a674610697d', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:09:01.418139 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f91fbad-afe0-4bc2-816e-8ca46ab42020 tempest-VolumeQuotaClassesTest-1411907492 tempest-VolumeQuotaClassesTest-1411907492-project-admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:09:01.418426 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f91fbad-afe0-4bc2-816e-8ca46ab42020 tempest-VolumeQuotaClassesTest-1411907492 tempest-VolumeQuotaClassesTest-1411907492-project-admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:09:01.418706 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f91fbad-afe0-4bc2-816e-8ca46ab42020 tempest-VolumeQuotaClassesTest-1411907492 tempest-VolumeQuotaClassesTest-1411907492-project-admin] PATH_INFO: `/v3/domains` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:09:01.419434 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f91fbad-afe0-4bc2-816e-8ca46ab42020 tempest-VolumeQuotaClassesTest-1411907492 tempest-VolumeQuotaClassesTest-1411907492-project-admin] RBAC: Adding query filter params (name=Default) {{(pid=92057) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} May 23 13:09:01.419745 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f91fbad-afe0-4bc2-816e-8ca46ab42020 tempest-VolumeQuotaClassesTest-1411907492 tempest-VolumeQuotaClassesTest-1411907492-project-admin] RBAC: Authorizing `identity:list_domains()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:09:01.421361 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f91fbad-afe0-4bc2-816e-8ca46ab42020 tempest-VolumeQuotaClassesTest-1411907492 tempest-VolumeQuotaClassesTest-1411907492-project-admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:09:01.434050 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 1270/2526] 10.222.0.30 () {58 vars in 1152 bytes} [Mon May 23 13:09:01 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 87 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) May 23 13:09:01.469293 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2c7e08b-84bf-4dff-b2b6-a35b4bb3dd0c tempest-VolumeQuotaClassesTest-1411907492 tempest-VolumeQuotaClassesTest-1411907492-project-admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:09:01.480260 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2c7e08b-84bf-4dff-b2b6-a35b4bb3dd0c tempest-VolumeQuotaClassesTest-1411907492 tempest-VolumeQuotaClassesTest-1411907492-project-admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:09:01.484786 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3069506e-f16c-4322-bcbd-aaed3a4e0321 tempest-VolumeQuotaClassesTest-1411907492 tempest-VolumeQuotaClassesTest-1411907492-project-admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a38e93d4a5fd44dbbb7699d34dd99715', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '85f4c27d81d140c6b8cf6a674610697d', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:09:01.485630 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3069506e-f16c-4322-bcbd-aaed3a4e0321 tempest-VolumeQuotaClassesTest-1411907492 tempest-VolumeQuotaClassesTest-1411907492-project-admin] REQUEST_METHOD: `DELETE` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:09:01.485931 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3069506e-f16c-4322-bcbd-aaed3a4e0321 tempest-VolumeQuotaClassesTest-1411907492 tempest-VolumeQuotaClassesTest-1411907492-project-admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:09:01.486210 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3069506e-f16c-4322-bcbd-aaed3a4e0321 tempest-VolumeQuotaClassesTest-1411907492 tempest-VolumeQuotaClassesTest-1411907492-project-admin] PATH_INFO: `/v3/projects/eb0a0d182569403bbdcb1e61fe34a793` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:09:01.487466 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3069506e-f16c-4322-bcbd-aaed3a4e0321 tempest-VolumeQuotaClassesTest-1411907492 tempest-VolumeQuotaClassesTest-1411907492-project-admin] RBAC: Authorizing `identity:delete_project(project_id=eb0a0d182569403bbdcb1e61fe34a793)` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:09:01.489505 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3069506e-f16c-4322-bcbd-aaed3a4e0321 tempest-VolumeQuotaClassesTest-1411907492 tempest-VolumeQuotaClassesTest-1411907492-project-admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:09:01.593640 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-3069506e-f16c-4322-bcbd-aaed3a4e0321 tempest-VolumeQuotaClassesTest-1411907492 tempest-VolumeQuotaClassesTest-1411907492-project-admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'eb0a0d182569403bbdcb1e61fe34a793', 'request_id': 'req-3069506e-f16c-4322-bcbd-aaed3a4e0321'} {{(pid=92056) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:09:01.677191 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-3069506e-f16c-4322-bcbd-aaed3a4e0321 tempest-VolumeQuotaClassesTest-1411907492 tempest-VolumeQuotaClassesTest-1411907492-project-admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'eb0a0d182569403bbdcb1e61fe34a793', 'request_id': 'req-3069506e-f16c-4322-bcbd-aaed3a4e0321'} {{(pid=92056) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:09:01.689414 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-3069506e-f16c-4322-bcbd-aaed3a4e0321 tempest-VolumeQuotaClassesTest-1411907492 tempest-VolumeQuotaClassesTest-1411907492-project-admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'eb0a0d182569403bbdcb1e61fe34a793', 'request_id': 'req-3069506e-f16c-4322-bcbd-aaed3a4e0321'} {{(pid=92056) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:09:01.702665 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-3069506e-f16c-4322-bcbd-aaed3a4e0321 tempest-VolumeQuotaClassesTest-1411907492 tempest-VolumeQuotaClassesTest-1411907492-project-admin] The token cache is being invalidate because project eb0a0d182569403bbdcb1e61fe34a793 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=92056) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} May 23 13:09:01.703165 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-3069506e-f16c-4322-bcbd-aaed3a4e0321 tempest-VolumeQuotaClassesTest-1411907492 tempest-VolumeQuotaClassesTest-1411907492-project-admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=92056) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:09:01.703822 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-3069506e-f16c-4322-bcbd-aaed3a4e0321 tempest-VolumeQuotaClassesTest-1411907492 tempest-VolumeQuotaClassesTest-1411907492-project-admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=92056) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:09:01.705678 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 1257/2527] 10.222.0.30 () {58 vars in 1232 bytes} [Mon May 23 13:09:01 2022] DELETE /identity/v3/projects/eb0a0d182569403bbdcb1e61fe34a793 => generated 0 bytes in 237 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) May 23 13:09:02.305770 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f91fbad-afe0-4bc2-816e-8ca46ab42020 tempest-VolumeQuotaClassesTest-1411907492 tempest-VolumeQuotaClassesTest-1411907492-project-admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:09:02.307537 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.fernet_utils [None req-6f91fbad-afe0-4bc2-816e-8ca46ab42020 tempest-VolumeQuotaClassesTest-1411907492 tempest-VolumeQuotaClassesTest-1411907492-project-admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92057) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:09:02.373505 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f91fbad-afe0-4bc2-816e-8ca46ab42020 tempest-VolumeQuotaClassesTest-1411907492 tempest-VolumeQuotaClassesTest-1411907492-project-admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:09:02.429624 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9123bb99-6c7a-47d3-a253-f01b06974fda admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:09:02.430521 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9123bb99-6c7a-47d3-a253-f01b06974fda admin admin] REQUEST_METHOD: `DELETE` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:09:02.430803 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9123bb99-6c7a-47d3-a253-f01b06974fda admin admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:09:02.431081 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9123bb99-6c7a-47d3-a253-f01b06974fda admin admin] PATH_INFO: `/v3/users/1808c319697549a292777f1b6a482f35` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:09:02.432248 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9123bb99-6c7a-47d3-a253-f01b06974fda admin admin] RBAC: Authorizing `identity:delete_user(user_id=1808c319697549a292777f1b6a482f35)` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:09:02.434198 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9123bb99-6c7a-47d3-a253-f01b06974fda admin admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:09:02.624688 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [None req-9123bb99-6c7a-47d3-a253-f01b06974fda admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '1808c319697549a292777f1b6a482f35'} {{(pid=92057) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:09:02.667435 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [None req-9123bb99-6c7a-47d3-a253-f01b06974fda admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '1808c319697549a292777f1b6a482f35'} {{(pid=92057) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:09:02.674824 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [None req-9123bb99-6c7a-47d3-a253-f01b06974fda admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '1808c319697549a292777f1b6a482f35'} {{(pid=92057) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:09:02.676482 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [None req-9123bb99-6c7a-47d3-a253-f01b06974fda admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '1808c319697549a292777f1b6a482f35'} {{(pid=92057) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:09:02.682224 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=91,keystone:DELETE=8,keystone:INSERT=1 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 13:09:02.690138 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [None req-9123bb99-6c7a-47d3-a253-f01b06974fda admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '1808c319697549a292777f1b6a482f35'} {{(pid=92057) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:09:02.696293 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 1271/2528] 10.222.0.30 () {58 vars in 1223 bytes} [Mon May 23 13:09:02 2022] DELETE /identity/v3/users/1808c319697549a292777f1b6a482f35 => generated 0 bytes in 391 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) May 23 13:09:02.700843 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1dded2c2-22c4-4bd3-b6db-b8043b975c35 None None] REQUEST_METHOD: `POST` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:09:02.701294 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1dded2c2-22c4-4bd3-b6db-b8043b975c35 None None] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:09:02.701951 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1dded2c2-22c4-4bd3-b6db-b8043b975c35 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:09:02.719685 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.auth.core [None req-1dded2c2-22c4-4bd3-b6db-b8043b975c35 None None] MFA Rules not processed for user `0f7119f8fa5c4445977671140ded91a4`. Rule list: `[]` (Enabled: `True`). {{(pid=92056) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 13:09:02.720881 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.fernet_utils [None req-1dded2c2-22c4-4bd3-b6db-b8043b975c35 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92056) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:09:02.785169 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 1258/2529] 10.222.0.30 () {58 vars in 957 bytes} [Mon May 23 13:09:02 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) May 23 13:09:02.794721 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9123bb99-6c7a-47d3-a253-f01b06974fda admin admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:09:02.796113 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.fernet_utils [None req-9123bb99-6c7a-47d3-a253-f01b06974fda admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92057) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:09:02.856086 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9123bb99-6c7a-47d3-a253-f01b06974fda admin admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:09:02.859762 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c09976e4-eee0-4da2-b97a-3308809ca911 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2eb83ca38e79449095f3a2f947bbb06a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9c389ae65aaa4542bb5a5c4fd0bacf5b', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:09:02.860401 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c09976e4-eee0-4da2-b97a-3308809ca911 service neutron] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:09:02.860682 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c09976e4-eee0-4da2-b97a-3308809ca911 service neutron] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:09:02.860955 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c09976e4-eee0-4da2-b97a-3308809ca911 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:09:02.870863 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c09976e4-eee0-4da2-b97a-3308809ca911 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:09:02.872322 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c09976e4-eee0-4da2-b97a-3308809ca911 service neutron] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:09:02.883010 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 1272/2530] 10.222.0.30 () {60 vars in 1365 bytes} [Mon May 23 13:09:02 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 89 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) May 23 13:09:03.309448 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1dded2c2-22c4-4bd3-b6db-b8043b975c35 None None] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:09:03.310878 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.fernet_utils [None req-1dded2c2-22c4-4bd3-b6db-b8043b975c35 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92056) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:09:03.325113 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1dded2c2-22c4-4bd3-b6db-b8043b975c35 None None] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:09:03.329972 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e74ac7d-14a2-47c0-a5d1-6cb4f7970462 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:09:03.331007 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e74ac7d-14a2-47c0-a5d1-6cb4f7970462 admin admin] REQUEST_METHOD: `DELETE` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:09:03.331150 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e74ac7d-14a2-47c0-a5d1-6cb4f7970462 admin admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:09:03.331332 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e74ac7d-14a2-47c0-a5d1-6cb4f7970462 admin admin] PATH_INFO: `/v3/projects/9cb259b3f8c240519ececd93edf16fc9` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:09:03.332706 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e74ac7d-14a2-47c0-a5d1-6cb4f7970462 admin admin] RBAC: Authorizing `identity:delete_project(project_id=9cb259b3f8c240519ececd93edf16fc9)` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:09:03.334676 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e74ac7d-14a2-47c0-a5d1-6cb4f7970462 admin admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:09:03.423516 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=88,keystone:INSERT=2,keystone:DELETE=10 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 13:09:03.430415 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-4e74ac7d-14a2-47c0-a5d1-6cb4f7970462 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '9cb259b3f8c240519ececd93edf16fc9', 'request_id': 'req-4e74ac7d-14a2-47c0-a5d1-6cb4f7970462'} {{(pid=92056) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:09:03.505776 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-4e74ac7d-14a2-47c0-a5d1-6cb4f7970462 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '9cb259b3f8c240519ececd93edf16fc9', 'request_id': 'req-4e74ac7d-14a2-47c0-a5d1-6cb4f7970462'} {{(pid=92056) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:09:03.512675 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-4e74ac7d-14a2-47c0-a5d1-6cb4f7970462 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '9cb259b3f8c240519ececd93edf16fc9', 'request_id': 'req-4e74ac7d-14a2-47c0-a5d1-6cb4f7970462'} {{(pid=92056) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:09:03.524902 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-4e74ac7d-14a2-47c0-a5d1-6cb4f7970462 admin admin] The token cache is being invalidate because project 9cb259b3f8c240519ececd93edf16fc9 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=92056) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} May 23 13:09:03.525190 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-4e74ac7d-14a2-47c0-a5d1-6cb4f7970462 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=92056) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:09:03.525851 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-4e74ac7d-14a2-47c0-a5d1-6cb4f7970462 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=92056) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:09:03.527512 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 1259/2531] 10.222.0.30 () {58 vars in 1232 bytes} [Mon May 23 13:09:03 2022] DELETE /identity/v3/projects/9cb259b3f8c240519ececd93edf16fc9 => generated 0 bytes in 219 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) May 23 13:09:03.531576 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c09976e4-eee0-4da2-b97a-3308809ca911 service neutron] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:09:03.543507 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c09976e4-eee0-4da2-b97a-3308809ca911 service neutron] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:09:03.606131 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3eb54bb6-46de-4c5a-be56-c24362718a0b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:09:03.606910 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3eb54bb6-46de-4c5a-be56-c24362718a0b admin admin] REQUEST_METHOD: `DELETE` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:09:03.607188 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3eb54bb6-46de-4c5a-be56-c24362718a0b admin admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:09:03.607492 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3eb54bb6-46de-4c5a-be56-c24362718a0b admin admin] PATH_INFO: `/v3/users/a38e93d4a5fd44dbbb7699d34dd99715` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:09:03.608721 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3eb54bb6-46de-4c5a-be56-c24362718a0b admin admin] RBAC: Authorizing `identity:delete_user(user_id=a38e93d4a5fd44dbbb7699d34dd99715)` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:09:03.610727 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3eb54bb6-46de-4c5a-be56-c24362718a0b admin admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:09:03.770994 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [None req-3eb54bb6-46de-4c5a-be56-c24362718a0b admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'a38e93d4a5fd44dbbb7699d34dd99715'} {{(pid=92057) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:09:03.831373 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [None req-3eb54bb6-46de-4c5a-be56-c24362718a0b admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'a38e93d4a5fd44dbbb7699d34dd99715'} {{(pid=92057) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:09:03.839367 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [None req-3eb54bb6-46de-4c5a-be56-c24362718a0b admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'a38e93d4a5fd44dbbb7699d34dd99715'} {{(pid=92057) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:09:03.840003 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [None req-3eb54bb6-46de-4c5a-be56-c24362718a0b admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'a38e93d4a5fd44dbbb7699d34dd99715'} {{(pid=92057) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:09:03.852409 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [None req-3eb54bb6-46de-4c5a-be56-c24362718a0b admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'a38e93d4a5fd44dbbb7699d34dd99715'} {{(pid=92057) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:09:03.858393 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 1273/2532] 10.222.0.30 () {58 vars in 1223 bytes} [Mon May 23 13:09:03 2022] DELETE /identity/v3/users/a38e93d4a5fd44dbbb7699d34dd99715 => generated 0 bytes in 327 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) May 23 13:09:04.526613 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e74ac7d-14a2-47c0-a5d1-6cb4f7970462 admin admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:09:04.528096 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.fernet_utils [None req-4e74ac7d-14a2-47c0-a5d1-6cb4f7970462 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92056) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:09:04.543187 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e74ac7d-14a2-47c0-a5d1-6cb4f7970462 admin admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:09:04.603250 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32c3d74e-a04f-4bc0-b692-1983891b8324 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:09:04.604110 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32c3d74e-a04f-4bc0-b692-1983891b8324 admin admin] REQUEST_METHOD: `DELETE` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:09:04.604414 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32c3d74e-a04f-4bc0-b692-1983891b8324 admin admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:09:04.604689 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32c3d74e-a04f-4bc0-b692-1983891b8324 admin admin] PATH_INFO: `/v3/projects/85f4c27d81d140c6b8cf6a674610697d` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:09:04.605862 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32c3d74e-a04f-4bc0-b692-1983891b8324 admin admin] RBAC: Authorizing `identity:delete_project(project_id=85f4c27d81d140c6b8cf6a674610697d)` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:09:04.607787 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32c3d74e-a04f-4bc0-b692-1983891b8324 admin admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:09:04.700029 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-32c3d74e-a04f-4bc0-b692-1983891b8324 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '85f4c27d81d140c6b8cf6a674610697d', 'request_id': 'req-32c3d74e-a04f-4bc0-b692-1983891b8324'} {{(pid=92056) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:09:04.768282 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-32c3d74e-a04f-4bc0-b692-1983891b8324 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '85f4c27d81d140c6b8cf6a674610697d', 'request_id': 'req-32c3d74e-a04f-4bc0-b692-1983891b8324'} {{(pid=92056) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:09:04.775830 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-32c3d74e-a04f-4bc0-b692-1983891b8324 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '85f4c27d81d140c6b8cf6a674610697d', 'request_id': 'req-32c3d74e-a04f-4bc0-b692-1983891b8324'} {{(pid=92056) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:09:04.787238 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-32c3d74e-a04f-4bc0-b692-1983891b8324 admin admin] The token cache is being invalidate because project 85f4c27d81d140c6b8cf6a674610697d was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=92056) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} May 23 13:09:04.787526 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-32c3d74e-a04f-4bc0-b692-1983891b8324 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=92056) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:09:04.793896 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-32c3d74e-a04f-4bc0-b692-1983891b8324 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=92056) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:09:04.795497 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 1260/2533] 10.222.0.30 () {58 vars in 1232 bytes} [Mon May 23 13:09:04 2022] DELETE /identity/v3/projects/85f4c27d81d140c6b8cf6a674610697d => generated 0 bytes in 269 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) May 23 13:09:04.800589 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d27f0fb-4e93-44d4-93eb-43bcffbdcf6b None None] REQUEST_METHOD: `POST` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:09:04.801710 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d27f0fb-4e93-44d4-93eb-43bcffbdcf6b None None] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:09:04.802132 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d27f0fb-4e93-44d4-93eb-43bcffbdcf6b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:09:04.814461 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:DELETE=12,keystone:SELECT=87,keystone:INSERT=1 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 13:09:04.820704 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.auth.core [None req-7d27f0fb-4e93-44d4-93eb-43bcffbdcf6b None None] MFA Rules not processed for user `0f7119f8fa5c4445977671140ded91a4`. Rule list: `[]` (Enabled: `True`). {{(pid=92057) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 13:09:04.821925 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.fernet_utils [None req-7d27f0fb-4e93-44d4-93eb-43bcffbdcf6b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92057) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:09:04.883225 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 1274/2534] 10.222.0.30 () {58 vars in 957 bytes} [Mon May 23 13:09:04 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 84 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) May 23 13:09:04.891004 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5c8e266-725a-492f-aa1b-c2d9c084f9e0 None None] REQUEST_METHOD: `POST` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:09:04.893252 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5c8e266-725a-492f-aa1b-c2d9c084f9e0 None None] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:09:04.893252 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5c8e266-725a-492f-aa1b-c2d9c084f9e0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:09:04.910617 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.auth.core [None req-d5c8e266-725a-492f-aa1b-c2d9c084f9e0 None None] MFA Rules not processed for user `0f7119f8fa5c4445977671140ded91a4`. Rule list: `[]` (Enabled: `True`). {{(pid=92056) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 13:09:04.911812 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.fernet_utils [None req-d5c8e266-725a-492f-aa1b-c2d9c084f9e0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92056) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:09:04.919709 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 1261/2535] 10.222.0.30 () {58 vars in 957 bytes} [Mon May 23 13:09:04 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 30 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) May 23 13:09:04.923893 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d27f0fb-4e93-44d4-93eb-43bcffbdcf6b None None] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:09:04.934776 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d27f0fb-4e93-44d4-93eb-43bcffbdcf6b None None] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:09:04.939341 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-caac4351-04cd-44b7-9989-d2b979d240df admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:09:04.939986 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-caac4351-04cd-44b7-9989-d2b979d240df admin admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:09:04.940267 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-caac4351-04cd-44b7-9989-d2b979d240df admin admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:09:04.940540 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-caac4351-04cd-44b7-9989-d2b979d240df admin admin] PATH_INFO: `/v3/domains` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:09:04.941251 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-caac4351-04cd-44b7-9989-d2b979d240df admin admin] RBAC: Adding query filter params (name=Default) {{(pid=92057) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} May 23 13:09:04.941556 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-caac4351-04cd-44b7-9989-d2b979d240df admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:09:04.943908 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-caac4351-04cd-44b7-9989-d2b979d240df admin admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:09:04.956425 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 1275/2536] 10.222.0.30 () {58 vars in 1152 bytes} [Mon May 23 13:09:04 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) May 23 13:09:04.961727 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5c8e266-725a-492f-aa1b-c2d9c084f9e0 None None] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:09:04.970962 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5c8e266-725a-492f-aa1b-c2d9c084f9e0 None None] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:09:04.976477 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0748b560-6ccf-476d-b663-8559d5f4a204 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:09:04.977119 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0748b560-6ccf-476d-b663-8559d5f4a204 admin admin] REQUEST_METHOD: `POST` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:09:04.977404 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0748b560-6ccf-476d-b663-8559d5f4a204 admin admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:09:04.977680 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0748b560-6ccf-476d-b663-8559d5f4a204 admin admin] PATH_INFO: `/v3/projects` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:09:04.978403 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0748b560-6ccf-476d-b663-8559d5f4a204 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:09:04.980308 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0748b560-6ccf-476d-b663-8559d5f4a204 admin admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:09:05.062664 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 1262/2537] 10.222.0.30 () {60 vars in 1152 bytes} [Mon May 23 13:09:04 2022] POST /identity/v3/projects => generated 393 bytes in 101 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) May 23 13:09:05.071624 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-caac4351-04cd-44b7-9989-d2b979d240df admin admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:09:05.081679 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-caac4351-04cd-44b7-9989-d2b979d240df admin admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:09:05.139728 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-410d8870-7e46-4a71-b7ed-8ad42e1adf6f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:09:05.140423 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-410d8870-7e46-4a71-b7ed-8ad42e1adf6f admin admin] REQUEST_METHOD: `POST` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:09:05.140698 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-410d8870-7e46-4a71-b7ed-8ad42e1adf6f admin admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:09:05.140969 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-410d8870-7e46-4a71-b7ed-8ad42e1adf6f admin admin] PATH_INFO: `/v3/users` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:09:05.141648 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-410d8870-7e46-4a71-b7ed-8ad42e1adf6f admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:09:05.143654 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-410d8870-7e46-4a71-b7ed-8ad42e1adf6f admin admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:09:05.250308 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 1276/2538] 10.222.0.30 () {60 vars in 1143 bytes} [Mon May 23 13:09:05 2022] POST /identity/v3/users => generated 297 bytes in 179 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) May 23 13:09:05.255060 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0748b560-6ccf-476d-b663-8559d5f4a204 admin admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:09:05.265539 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0748b560-6ccf-476d-b663-8559d5f4a204 admin admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:09:05.270996 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b55c519-ca4c-4429-aa42-abbc9798dcf2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:09:05.271693 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b55c519-ca4c-4429-aa42-abbc9798dcf2 admin admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:09:05.271971 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b55c519-ca4c-4429-aa42-abbc9798dcf2 admin admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:09:05.272240 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b55c519-ca4c-4429-aa42-abbc9798dcf2 admin admin] PATH_INFO: `/v3/roles` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:09:05.273274 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b55c519-ca4c-4429-aa42-abbc9798dcf2 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:09:05.275037 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b55c519-ca4c-4429-aa42-abbc9798dcf2 admin admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:09:05.285886 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 1263/2539] 10.222.0.30 () {58 vars in 1121 bytes} [Mon May 23 13:09:05 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) May 23 13:09:05.289791 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-410d8870-7e46-4a71-b7ed-8ad42e1adf6f admin admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:09:05.300380 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-410d8870-7e46-4a71-b7ed-8ad42e1adf6f admin admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:09:05.306344 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c63e4428-7648-44f1-81a8-1198a5d0fbbe admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:09:05.307031 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c63e4428-7648-44f1-81a8-1198a5d0fbbe admin admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:09:05.307343 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c63e4428-7648-44f1-81a8-1198a5d0fbbe admin admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:09:05.307614 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c63e4428-7648-44f1-81a8-1198a5d0fbbe admin admin] PATH_INFO: `/v3/roles` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:09:05.308331 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c63e4428-7648-44f1-81a8-1198a5d0fbbe admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:09:05.310152 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c63e4428-7648-44f1-81a8-1198a5d0fbbe admin admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:09:05.321193 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 1277/2540] 10.222.0.30 () {58 vars in 1121 bytes} [Mon May 23 13:09:05 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) May 23 13:09:05.325552 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b55c519-ca4c-4429-aa42-abbc9798dcf2 admin admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:09:05.336153 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b55c519-ca4c-4429-aa42-abbc9798dcf2 admin admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:09:05.340645 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9cc56c5c-cc55-4103-ae6f-9b26041c7c7b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:09:05.341516 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cc56c5c-cc55-4103-ae6f-9b26041c7c7b admin admin] REQUEST_METHOD: `PUT` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:09:05.341793 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cc56c5c-cc55-4103-ae6f-9b26041c7c7b admin admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:09:05.342112 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cc56c5c-cc55-4103-ae6f-9b26041c7c7b admin admin] PATH_INFO: `/v3/projects/9ced944af0d346c292fb8254d3649896/users/cae79ae1ae6b426c8b42a9d1aac1ccf3/roles/80fb22507c9a4ff09d44d8f12f6fff1a` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:09:05.356335 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9cc56c5c-cc55-4103-ae6f-9b26041c7c7b admin admin] RBAC: Authorizing `identity:create_grant(project_id=9ced944af0d346c292fb8254d3649896, user_id=cae79ae1ae6b426c8b42a9d1aac1ccf3, role_id=80fb22507c9a4ff09d44d8f12f6fff1a)` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:09:05.359026 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9cc56c5c-cc55-4103-ae6f-9b26041c7c7b admin admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:09:05.361175 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=86,keystone:DELETE=11,keystone:INSERT=3 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 13:09:05.451461 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 1264/2541] 10.222.0.30 () {60 vars in 1482 bytes} [Mon May 23 13:09:05 2022] PUT /identity/v3/projects/9ced944af0d346c292fb8254d3649896/users/cae79ae1ae6b426c8b42a9d1aac1ccf3/roles/80fb22507c9a4ff09d44d8f12f6fff1a => generated 0 bytes in 126 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) May 23 13:09:05.462102 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb72ccac-3109-4441-9fb7-679a8176853b None None] REQUEST_METHOD: `POST` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:09:05.462383 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb72ccac-3109-4441-9fb7-679a8176853b None None] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:09:05.462666 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb72ccac-3109-4441-9fb7-679a8176853b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:09:05.489414 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.auth.core [None req-bb72ccac-3109-4441-9fb7-679a8176853b None None] MFA Rules not processed for user `cae79ae1ae6b426c8b42a9d1aac1ccf3`. Rule list: `[]` (Enabled: `True`). {{(pid=92057) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 13:09:05.490765 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.fernet_utils [None req-bb72ccac-3109-4441-9fb7-679a8176853b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92057) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:09:05.563390 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 1278/2542] 10.222.0.30 () {58 vars in 957 bytes} [Mon May 23 13:09:05 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 102 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) May 23 13:09:05.569088 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9cc56c5c-cc55-4103-ae6f-9b26041c7c7b admin admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:09:05.642049 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9cc56c5c-cc55-4103-ae6f-9b26041c7c7b admin admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:09:05.647017 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0dfcdeb-c9e8-431c-b8bb-18f347c53d08 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:09:05.648012 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0dfcdeb-c9e8-431c-b8bb-18f347c53d08 admin admin] REQUEST_METHOD: `POST` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:09:05.648361 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0dfcdeb-c9e8-431c-b8bb-18f347c53d08 admin admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:09:05.648700 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0dfcdeb-c9e8-431c-b8bb-18f347c53d08 admin admin] PATH_INFO: `/v3/projects` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:09:05.649689 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0dfcdeb-c9e8-431c-b8bb-18f347c53d08 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:09:05.651706 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0dfcdeb-c9e8-431c-b8bb-18f347c53d08 admin admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:09:05.759688 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 1265/2543] 10.222.0.30 () {60 vars in 1152 bytes} [Mon May 23 13:09:05 2022] POST /identity/v3/projects => generated 391 bytes in 188 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) May 23 13:09:05.760845 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb72ccac-3109-4441-9fb7-679a8176853b None None] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:09:05.766741 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 13:09:05.772840 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb72ccac-3109-4441-9fb7-679a8176853b None None] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:09:05.828834 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-136dcfed-f694-44c4-9ae5-63dfc831b7b5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:09:05.829504 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-136dcfed-f694-44c4-9ae5-63dfc831b7b5 admin admin] REQUEST_METHOD: `POST` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:09:05.829754 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-136dcfed-f694-44c4-9ae5-63dfc831b7b5 admin admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:09:05.830069 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-136dcfed-f694-44c4-9ae5-63dfc831b7b5 admin admin] PATH_INFO: `/v3/users` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:09:05.830735 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-136dcfed-f694-44c4-9ae5-63dfc831b7b5 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:09:05.832785 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-136dcfed-f694-44c4-9ae5-63dfc831b7b5 admin admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:09:05.963431 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 1279/2544] 10.222.0.30 () {60 vars in 1143 bytes} [Mon May 23 13:09:05 2022] POST /identity/v3/users => generated 302 bytes in 203 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) May 23 13:09:05.967611 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0dfcdeb-c9e8-431c-b8bb-18f347c53d08 admin admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:09:05.978585 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0dfcdeb-c9e8-431c-b8bb-18f347c53d08 admin admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:09:05.983220 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46b202aa-b2e6-44fa-932c-8b5ab1c24829 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:09:05.983871 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46b202aa-b2e6-44fa-932c-8b5ab1c24829 admin admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:09:05.984152 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46b202aa-b2e6-44fa-932c-8b5ab1c24829 admin admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:09:05.984426 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46b202aa-b2e6-44fa-932c-8b5ab1c24829 admin admin] PATH_INFO: `/v3/roles` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:09:05.985145 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46b202aa-b2e6-44fa-932c-8b5ab1c24829 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:09:05.989972 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46b202aa-b2e6-44fa-932c-8b5ab1c24829 admin admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:09:06.001156 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 1266/2545] 10.222.0.30 () {58 vars in 1121 bytes} [Mon May 23 13:09:05 2022] GET /identity/v3/roles => generated 1420 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) May 23 13:09:06.005940 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-136dcfed-f694-44c4-9ae5-63dfc831b7b5 admin admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:09:06.016321 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-136dcfed-f694-44c4-9ae5-63dfc831b7b5 admin admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:09:06.021187 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8d37616-62be-4933-abc9-45ac71dfb1cc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:09:06.022169 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8d37616-62be-4933-abc9-45ac71dfb1cc admin admin] REQUEST_METHOD: `PUT` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:09:06.022457 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8d37616-62be-4933-abc9-45ac71dfb1cc admin admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:09:06.023002 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8d37616-62be-4933-abc9-45ac71dfb1cc admin admin] PATH_INFO: `/v3/projects/092cd574626948bba62da642631f8291/users/764e1a1b8fd64e51a84b7118c8ccdabb/roles/bc3c3c9d654940cbaa5177a6be23d1bc` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:09:06.039153 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8d37616-62be-4933-abc9-45ac71dfb1cc admin admin] RBAC: Authorizing `identity:create_grant(project_id=092cd574626948bba62da642631f8291, user_id=764e1a1b8fd64e51a84b7118c8ccdabb, role_id=bc3c3c9d654940cbaa5177a6be23d1bc)` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:09:06.041743 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8d37616-62be-4933-abc9-45ac71dfb1cc admin admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:09:06.096519 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 1280/2546] 10.222.0.30 () {60 vars in 1482 bytes} [Mon May 23 13:09:06 2022] PUT /identity/v3/projects/092cd574626948bba62da642631f8291/users/764e1a1b8fd64e51a84b7118c8ccdabb/roles/bc3c3c9d654940cbaa5177a6be23d1bc => generated 0 bytes in 91 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) May 23 13:09:06.101177 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46b202aa-b2e6-44fa-932c-8b5ab1c24829 admin admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:09:06.112494 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46b202aa-b2e6-44fa-932c-8b5ab1c24829 admin admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:09:06.171476 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85064257-95fb-4e86-9e1c-1b37aa0b1ff4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:09:06.172142 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85064257-95fb-4e86-9e1c-1b37aa0b1ff4 admin admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:09:06.172474 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85064257-95fb-4e86-9e1c-1b37aa0b1ff4 admin admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:09:06.172757 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85064257-95fb-4e86-9e1c-1b37aa0b1ff4 admin admin] PATH_INFO: `/v3/roles` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:09:06.173486 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85064257-95fb-4e86-9e1c-1b37aa0b1ff4 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:09:06.175139 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85064257-95fb-4e86-9e1c-1b37aa0b1ff4 admin admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:09:06.185649 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 1267/2547] 10.222.0.30 () {58 vars in 1121 bytes} [Mon May 23 13:09:06 2022] GET /identity/v3/roles => generated 1420 bytes in 85 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) May 23 13:09:06.189955 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8d37616-62be-4933-abc9-45ac71dfb1cc admin admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:09:06.200585 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8d37616-62be-4933-abc9-45ac71dfb1cc admin admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:09:06.205280 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8bb03a34-468a-4cb6-818e-2f38f78208a1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:09:06.206228 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8bb03a34-468a-4cb6-818e-2f38f78208a1 admin admin] REQUEST_METHOD: `PUT` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:09:06.206549 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8bb03a34-468a-4cb6-818e-2f38f78208a1 admin admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:09:06.206823 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8bb03a34-468a-4cb6-818e-2f38f78208a1 admin admin] PATH_INFO: `/v3/projects/092cd574626948bba62da642631f8291/users/764e1a1b8fd64e51a84b7118c8ccdabb/roles/bc3c3c9d654940cbaa5177a6be23d1bc` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:09:06.209338 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8bb03a34-468a-4cb6-818e-2f38f78208a1 admin admin] RBAC: Authorizing `identity:create_grant(project_id=092cd574626948bba62da642631f8291, user_id=764e1a1b8fd64e51a84b7118c8ccdabb, role_id=bc3c3c9d654940cbaa5177a6be23d1bc)` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:09:06.211999 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8bb03a34-468a-4cb6-818e-2f38f78208a1 admin admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:09:06.222053 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 1281/2548] 10.222.0.30 () {60 vars in 1482 bytes} [Mon May 23 13:09:06 2022] PUT /identity/v3/projects/092cd574626948bba62da642631f8291/users/764e1a1b8fd64e51a84b7118c8ccdabb/roles/bc3c3c9d654940cbaa5177a6be23d1bc => generated 0 bytes in 33 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) May 23 13:09:06.232356 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd59c803-c1a8-4529-aef6-adf2ca2ca845 None None] REQUEST_METHOD: `POST` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:09:06.232674 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd59c803-c1a8-4529-aef6-adf2ca2ca845 None None] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:09:06.232961 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd59c803-c1a8-4529-aef6-adf2ca2ca845 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:09:06.260253 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.auth.core [None req-bd59c803-c1a8-4529-aef6-adf2ca2ca845 None None] MFA Rules not processed for user `764e1a1b8fd64e51a84b7118c8ccdabb`. Rule list: `[]` (Enabled: `True`). {{(pid=92056) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 13:09:06.261447 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.fernet_utils [None req-bd59c803-c1a8-4529-aef6-adf2ca2ca845 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92056) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:09:06.311724 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:INSERT=2,keystone:SELECT=98 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 13:09:06.330751 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 1268/2549] 10.222.0.30 () {58 vars in 957 bytes} [Mon May 23 13:09:06 2022] POST /identity/v3/auth/tokens => generated 3325 bytes in 99 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) May 23 13:09:06.347155 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8bb03a34-468a-4cb6-818e-2f38f78208a1 admin admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:09:06.348939 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.fernet_utils [None req-8bb03a34-468a-4cb6-818e-2f38f78208a1 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92057) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:09:06.407051 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8bb03a34-468a-4cb6-818e-2f38f78208a1 admin admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:09:06.410928 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-046fc434-0f0d-4af3-9e8a-75880b95759f service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1666097f8ed545c6b0e4809c93e4775a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9c389ae65aaa4542bb5a5c4fd0bacf5b', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:09:06.411682 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-046fc434-0f0d-4af3-9e8a-75880b95759f service cinder] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:09:06.411976 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-046fc434-0f0d-4af3-9e8a-75880b95759f service cinder] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:09:06.412259 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-046fc434-0f0d-4af3-9e8a-75880b95759f service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:09:06.421348 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-046fc434-0f0d-4af3-9e8a-75880b95759f service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:09:06.422816 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-046fc434-0f0d-4af3-9e8a-75880b95759f service cinder] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:09:06.433052 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 1282/2550] 10.222.0.30 () {60 vars in 1365 bytes} [Mon May 23 13:09:06 2022] GET /identity/v3/auth/tokens => generated 3325 bytes in 86 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) May 23 13:09:06.479518 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd59c803-c1a8-4529-aef6-adf2ca2ca845 None None] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:09:06.480873 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.fernet_utils [None req-bd59c803-c1a8-4529-aef6-adf2ca2ca845 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92056) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:09:06.493201 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd59c803-c1a8-4529-aef6-adf2ca2ca845 None None] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:09:06.496766 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-685ab970-844e-472d-a510-c2d79d658445 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1666097f8ed545c6b0e4809c93e4775a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9c389ae65aaa4542bb5a5c4fd0bacf5b', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:09:06.497392 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-685ab970-844e-472d-a510-c2d79d658445 service cinder] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:09:06.497674 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-685ab970-844e-472d-a510-c2d79d658445 service cinder] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:09:06.497981 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-685ab970-844e-472d-a510-c2d79d658445 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:09:06.556313 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-685ab970-844e-472d-a510-c2d79d658445 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:09:06.557743 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-685ab970-844e-472d-a510-c2d79d658445 service cinder] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:09:06.566914 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 1269/2551] 10.222.0.30 () {60 vars in 1365 bytes} [Mon May 23 13:09:06 2022] GET /identity/v3/auth/tokens => generated 3260 bytes in 88 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) May 23 13:09:11.440859 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-046fc434-0f0d-4af3-9e8a-75880b95759f service cinder] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:09:11.499838 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 13:09:11.504503 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-046fc434-0f0d-4af3-9e8a-75880b95759f service cinder] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:09:11.509490 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95a52c34-79b6-435d-946b-8e9e628cd912 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:09:11.511538 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95a52c34-79b6-435d-946b-8e9e628cd912 admin admin] REQUEST_METHOD: `DELETE` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:09:11.511538 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95a52c34-79b6-435d-946b-8e9e628cd912 admin admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:09:11.511538 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95a52c34-79b6-435d-946b-8e9e628cd912 admin admin] PATH_INFO: `/v3/users/cae79ae1ae6b426c8b42a9d1aac1ccf3` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:09:11.512119 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95a52c34-79b6-435d-946b-8e9e628cd912 admin admin] RBAC: Authorizing `identity:delete_user(user_id=cae79ae1ae6b426c8b42a9d1aac1ccf3)` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:09:11.514089 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95a52c34-79b6-435d-946b-8e9e628cd912 admin admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:09:11.722208 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [None req-95a52c34-79b6-435d-946b-8e9e628cd912 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'cae79ae1ae6b426c8b42a9d1aac1ccf3'} {{(pid=92057) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:09:11.798353 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [None req-95a52c34-79b6-435d-946b-8e9e628cd912 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'cae79ae1ae6b426c8b42a9d1aac1ccf3'} {{(pid=92057) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:09:11.806952 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [None req-95a52c34-79b6-435d-946b-8e9e628cd912 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'cae79ae1ae6b426c8b42a9d1aac1ccf3'} {{(pid=92057) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:09:11.807786 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [None req-95a52c34-79b6-435d-946b-8e9e628cd912 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'cae79ae1ae6b426c8b42a9d1aac1ccf3'} {{(pid=92057) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:09:11.820676 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [None req-95a52c34-79b6-435d-946b-8e9e628cd912 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'cae79ae1ae6b426c8b42a9d1aac1ccf3'} {{(pid=92057) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:09:11.915512 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 1283/2552] 10.222.0.30 () {58 vars in 1223 bytes} [Mon May 23 13:09:11 2022] DELETE /identity/v3/users/cae79ae1ae6b426c8b42a9d1aac1ccf3 => generated 0 bytes in 389 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) May 23 13:09:11.915679 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-979e41e9-301f-48a7-af11-c0cf87b18284 None None] REQUEST_METHOD: `POST` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:09:11.915679 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-979e41e9-301f-48a7-af11-c0cf87b18284 None None] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:09:11.915679 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-979e41e9-301f-48a7-af11-c0cf87b18284 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:09:11.915679 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.auth.core [None req-979e41e9-301f-48a7-af11-c0cf87b18284 None None] MFA Rules not processed for user `0f7119f8fa5c4445977671140ded91a4`. Rule list: `[]` (Enabled: `True`). {{(pid=92056) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 13:09:11.915679 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.fernet_utils [None req-979e41e9-301f-48a7-af11-c0cf87b18284 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92056) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:09:11.917988 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 1270/2553] 10.222.0.30 () {58 vars in 957 bytes} [Mon May 23 13:09:11 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 84 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) May 23 13:09:11.928174 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95a52c34-79b6-435d-946b-8e9e628cd912 admin admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:09:11.929579 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.fernet_utils [None req-95a52c34-79b6-435d-946b-8e9e628cd912 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92057) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:09:11.986656 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95a52c34-79b6-435d-946b-8e9e628cd912 admin admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:09:11.990774 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1448d0a6-f477-4991-8407-a6ddde918bc5 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2eb83ca38e79449095f3a2f947bbb06a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9c389ae65aaa4542bb5a5c4fd0bacf5b', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:09:11.991427 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1448d0a6-f477-4991-8407-a6ddde918bc5 service neutron] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:09:11.991721 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1448d0a6-f477-4991-8407-a6ddde918bc5 service neutron] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:09:11.992445 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1448d0a6-f477-4991-8407-a6ddde918bc5 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:09:12.000967 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1448d0a6-f477-4991-8407-a6ddde918bc5 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:09:12.002465 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1448d0a6-f477-4991-8407-a6ddde918bc5 service neutron] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:09:12.012614 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 1284/2554] 10.222.0.30 () {60 vars in 1365 bytes} [Mon May 23 13:09:11 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 85 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) May 23 13:09:12.438427 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-979e41e9-301f-48a7-af11-c0cf87b18284 None None] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:09:12.439916 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.fernet_utils [None req-979e41e9-301f-48a7-af11-c0cf87b18284 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92056) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:09:12.453039 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-979e41e9-301f-48a7-af11-c0cf87b18284 None None] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:09:12.458300 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e31f6913-6405-4da4-809f-4bfcdbfb1327 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:09:12.459173 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e31f6913-6405-4da4-809f-4bfcdbfb1327 admin admin] REQUEST_METHOD: `DELETE` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:09:12.459453 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e31f6913-6405-4da4-809f-4bfcdbfb1327 admin admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:09:12.459727 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e31f6913-6405-4da4-809f-4bfcdbfb1327 admin admin] PATH_INFO: `/v3/projects/9ced944af0d346c292fb8254d3649896` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:09:12.460924 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e31f6913-6405-4da4-809f-4bfcdbfb1327 admin admin] RBAC: Authorizing `identity:delete_project(project_id=9ced944af0d346c292fb8254d3649896)` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:09:12.462998 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e31f6913-6405-4da4-809f-4bfcdbfb1327 admin admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:09:12.585522 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-e31f6913-6405-4da4-809f-4bfcdbfb1327 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '9ced944af0d346c292fb8254d3649896', 'request_id': 'req-e31f6913-6405-4da4-809f-4bfcdbfb1327'} {{(pid=92056) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:09:12.667407 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-e31f6913-6405-4da4-809f-4bfcdbfb1327 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '9ced944af0d346c292fb8254d3649896', 'request_id': 'req-e31f6913-6405-4da4-809f-4bfcdbfb1327'} {{(pid=92056) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:09:12.674117 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-e31f6913-6405-4da4-809f-4bfcdbfb1327 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '9ced944af0d346c292fb8254d3649896', 'request_id': 'req-e31f6913-6405-4da4-809f-4bfcdbfb1327'} {{(pid=92056) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:09:12.685446 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-e31f6913-6405-4da4-809f-4bfcdbfb1327 admin admin] The token cache is being invalidate because project 9ced944af0d346c292fb8254d3649896 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=92056) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} May 23 13:09:12.685754 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-e31f6913-6405-4da4-809f-4bfcdbfb1327 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=92056) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:09:12.686429 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-e31f6913-6405-4da4-809f-4bfcdbfb1327 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=92056) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:09:12.688597 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 1271/2555] 10.222.0.30 () {58 vars in 1232 bytes} [Mon May 23 13:09:12 2022] DELETE /identity/v3/projects/9ced944af0d346c292fb8254d3649896 => generated 0 bytes in 251 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) May 23 13:09:12.693106 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1448d0a6-f477-4991-8407-a6ddde918bc5 service neutron] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:09:12.702916 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1448d0a6-f477-4991-8407-a6ddde918bc5 service neutron] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:09:12.759148 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc9bbd10-0d6f-4c2d-8103-ae51987ea424 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:09:12.760010 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc9bbd10-0d6f-4c2d-8103-ae51987ea424 admin admin] REQUEST_METHOD: `DELETE` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:09:12.760295 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc9bbd10-0d6f-4c2d-8103-ae51987ea424 admin admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:09:12.760576 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc9bbd10-0d6f-4c2d-8103-ae51987ea424 admin admin] PATH_INFO: `/v3/users/764e1a1b8fd64e51a84b7118c8ccdabb` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:09:12.761749 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc9bbd10-0d6f-4c2d-8103-ae51987ea424 admin admin] RBAC: Authorizing `identity:delete_user(user_id=764e1a1b8fd64e51a84b7118c8ccdabb)` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:09:12.763708 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc9bbd10-0d6f-4c2d-8103-ae51987ea424 admin admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:09:12.784681 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=86,keystone:DELETE=13,keystone:INSERT=1 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 13:09:12.954858 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [None req-cc9bbd10-0d6f-4c2d-8103-ae51987ea424 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '764e1a1b8fd64e51a84b7118c8ccdabb'} {{(pid=92057) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:09:13.021718 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [None req-cc9bbd10-0d6f-4c2d-8103-ae51987ea424 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '764e1a1b8fd64e51a84b7118c8ccdabb'} {{(pid=92057) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:09:13.029536 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [None req-cc9bbd10-0d6f-4c2d-8103-ae51987ea424 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '764e1a1b8fd64e51a84b7118c8ccdabb'} {{(pid=92057) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:09:13.030202 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [None req-cc9bbd10-0d6f-4c2d-8103-ae51987ea424 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '764e1a1b8fd64e51a84b7118c8ccdabb'} {{(pid=92057) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:09:13.042213 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [None req-cc9bbd10-0d6f-4c2d-8103-ae51987ea424 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '764e1a1b8fd64e51a84b7118c8ccdabb'} {{(pid=92057) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:09:13.048095 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 1285/2556] 10.222.0.30 () {58 vars in 1223 bytes} [Mon May 23 13:09:12 2022] DELETE /identity/v3/users/764e1a1b8fd64e51a84b7118c8ccdabb => generated 0 bytes in 355 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) May 23 13:09:13.420733 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e31f6913-6405-4da4-809f-4bfcdbfb1327 admin admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:09:13.422285 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.fernet_utils [None req-e31f6913-6405-4da4-809f-4bfcdbfb1327 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92056) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:09:13.436600 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e31f6913-6405-4da4-809f-4bfcdbfb1327 admin admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:09:13.446027 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 13:09:13.495534 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78a8f566-00d6-437c-8757-9b8513cae66b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:09:13.496320 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78a8f566-00d6-437c-8757-9b8513cae66b admin admin] REQUEST_METHOD: `DELETE` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:09:13.496586 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78a8f566-00d6-437c-8757-9b8513cae66b admin admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:09:13.496858 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78a8f566-00d6-437c-8757-9b8513cae66b admin admin] PATH_INFO: `/v3/projects/092cd574626948bba62da642631f8291` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:09:13.498056 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-78a8f566-00d6-437c-8757-9b8513cae66b admin admin] RBAC: Authorizing `identity:delete_project(project_id=092cd574626948bba62da642631f8291)` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:09:13.500006 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-78a8f566-00d6-437c-8757-9b8513cae66b admin admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:09:13.621672 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-78a8f566-00d6-437c-8757-9b8513cae66b admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '092cd574626948bba62da642631f8291', 'request_id': 'req-78a8f566-00d6-437c-8757-9b8513cae66b'} {{(pid=92056) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:09:13.682890 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-78a8f566-00d6-437c-8757-9b8513cae66b admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '092cd574626948bba62da642631f8291', 'request_id': 'req-78a8f566-00d6-437c-8757-9b8513cae66b'} {{(pid=92056) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:09:13.689854 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-78a8f566-00d6-437c-8757-9b8513cae66b admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '092cd574626948bba62da642631f8291', 'request_id': 'req-78a8f566-00d6-437c-8757-9b8513cae66b'} {{(pid=92056) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:09:13.701777 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-78a8f566-00d6-437c-8757-9b8513cae66b admin admin] The token cache is being invalidate because project 092cd574626948bba62da642631f8291 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=92056) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} May 23 13:09:13.702097 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-78a8f566-00d6-437c-8757-9b8513cae66b admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=92056) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:09:13.702750 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-78a8f566-00d6-437c-8757-9b8513cae66b admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=92056) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:09:13.704481 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 1272/2557] 10.222.0.30 () {58 vars in 1232 bytes} [Mon May 23 13:09:13 2022] DELETE /identity/v3/projects/092cd574626948bba62da642631f8291 => generated 0 bytes in 284 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) May 23 13:09:13.710218 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0012e4ed-fcf4-4256-ba84-c80e3eac8635 None None] REQUEST_METHOD: `POST` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:09:13.710677 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0012e4ed-fcf4-4256-ba84-c80e3eac8635 None None] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:09:13.710789 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0012e4ed-fcf4-4256-ba84-c80e3eac8635 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:09:13.728255 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.auth.core [None req-0012e4ed-fcf4-4256-ba84-c80e3eac8635 None None] MFA Rules not processed for user `0f7119f8fa5c4445977671140ded91a4`. Rule list: `[]` (Enabled: `True`). {{(pid=92057) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 13:09:13.729450 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.fernet_utils [None req-0012e4ed-fcf4-4256-ba84-c80e3eac8635 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92057) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:09:13.789925 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 1286/2558] 10.222.0.30 () {58 vars in 957 bytes} [Mon May 23 13:09:13 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 81 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) May 23 13:09:13.798505 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dea80b93-5708-4991-b03b-c6441bce9bab None None] REQUEST_METHOD: `POST` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:09:13.798793 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dea80b93-5708-4991-b03b-c6441bce9bab None None] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:09:13.799072 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dea80b93-5708-4991-b03b-c6441bce9bab None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:09:13.817826 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.auth.core [None req-dea80b93-5708-4991-b03b-c6441bce9bab None None] MFA Rules not processed for user `0f7119f8fa5c4445977671140ded91a4`. Rule list: `[]` (Enabled: `True`). {{(pid=92056) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 13:09:13.819038 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.fernet_utils [None req-dea80b93-5708-4991-b03b-c6441bce9bab None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92056) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:09:13.826867 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 1273/2559] 10.222.0.30 () {58 vars in 957 bytes} [Mon May 23 13:09:13 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 29 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) May 23 13:09:13.831096 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0012e4ed-fcf4-4256-ba84-c80e3eac8635 None None] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:09:13.841053 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0012e4ed-fcf4-4256-ba84-c80e3eac8635 None None] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:09:13.845528 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89b6a93f-a97b-47a8-9a95-312a7696dca8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:09:13.846210 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89b6a93f-a97b-47a8-9a95-312a7696dca8 admin admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:09:13.846494 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89b6a93f-a97b-47a8-9a95-312a7696dca8 admin admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:09:13.846770 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89b6a93f-a97b-47a8-9a95-312a7696dca8 admin admin] PATH_INFO: `/v3/domains` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:09:13.847483 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89b6a93f-a97b-47a8-9a95-312a7696dca8 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=92057) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} May 23 13:09:13.847918 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89b6a93f-a97b-47a8-9a95-312a7696dca8 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:09:13.849484 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89b6a93f-a97b-47a8-9a95-312a7696dca8 admin admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:09:13.861119 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 1287/2560] 10.222.0.30 () {58 vars in 1152 bytes} [Mon May 23 13:09:13 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) May 23 13:09:13.865446 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dea80b93-5708-4991-b03b-c6441bce9bab None None] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:09:13.874431 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dea80b93-5708-4991-b03b-c6441bce9bab None None] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:09:13.879587 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6469c41-c87c-4132-95e6-3bec3f0c8eee admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:09:13.880284 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6469c41-c87c-4132-95e6-3bec3f0c8eee admin admin] REQUEST_METHOD: `POST` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:09:13.880566 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6469c41-c87c-4132-95e6-3bec3f0c8eee admin admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:09:13.880842 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6469c41-c87c-4132-95e6-3bec3f0c8eee admin admin] PATH_INFO: `/v3/projects` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:09:13.881516 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6469c41-c87c-4132-95e6-3bec3f0c8eee admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:09:13.883429 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6469c41-c87c-4132-95e6-3bec3f0c8eee admin admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:09:13.976919 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 1274/2561] 10.222.0.30 () {60 vars in 1152 bytes} [Mon May 23 13:09:13 2022] POST /identity/v3/projects => generated 383 bytes in 112 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) May 23 13:09:13.981060 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89b6a93f-a97b-47a8-9a95-312a7696dca8 admin admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:09:13.991646 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89b6a93f-a97b-47a8-9a95-312a7696dca8 admin admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:09:14.048214 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f2fe08d-9dea-4f7c-bfbc-5fe2ff3965f4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:09:14.048797 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f2fe08d-9dea-4f7c-bfbc-5fe2ff3965f4 admin admin] REQUEST_METHOD: `POST` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:09:14.049083 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f2fe08d-9dea-4f7c-bfbc-5fe2ff3965f4 admin admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:09:14.049361 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f2fe08d-9dea-4f7c-bfbc-5fe2ff3965f4 admin admin] PATH_INFO: `/v3/users` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:09:14.050071 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f2fe08d-9dea-4f7c-bfbc-5fe2ff3965f4 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:09:14.052504 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f2fe08d-9dea-4f7c-bfbc-5fe2ff3965f4 admin admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:09:14.170208 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 1288/2562] 10.222.0.30 () {60 vars in 1143 bytes} [Mon May 23 13:09:13 2022] POST /identity/v3/users => generated 292 bytes in 190 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) May 23 13:09:14.174505 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6469c41-c87c-4132-95e6-3bec3f0c8eee admin admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:09:14.185804 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6469c41-c87c-4132-95e6-3bec3f0c8eee admin admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:09:14.190383 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c9253bb-4557-4cd0-841c-b60f4649f15c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:09:14.191034 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c9253bb-4557-4cd0-841c-b60f4649f15c admin admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:09:14.191496 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c9253bb-4557-4cd0-841c-b60f4649f15c admin admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:09:14.192781 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c9253bb-4557-4cd0-841c-b60f4649f15c admin admin] PATH_INFO: `/v3/roles` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:09:14.193541 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c9253bb-4557-4cd0-841c-b60f4649f15c admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:09:14.195182 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c9253bb-4557-4cd0-841c-b60f4649f15c admin admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:09:14.205417 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 1275/2563] 10.222.0.30 () {58 vars in 1121 bytes} [Mon May 23 13:09:14 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) May 23 13:09:14.209350 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f2fe08d-9dea-4f7c-bfbc-5fe2ff3965f4 admin admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:09:14.219401 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f2fe08d-9dea-4f7c-bfbc-5fe2ff3965f4 admin admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:09:14.224079 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-713de72d-4fdd-4f04-8548-1ae2bcbbe5ca admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:09:14.224733 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-713de72d-4fdd-4f04-8548-1ae2bcbbe5ca admin admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:09:14.225019 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-713de72d-4fdd-4f04-8548-1ae2bcbbe5ca admin admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:09:14.225297 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-713de72d-4fdd-4f04-8548-1ae2bcbbe5ca admin admin] PATH_INFO: `/v3/roles` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:09:14.226050 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-713de72d-4fdd-4f04-8548-1ae2bcbbe5ca admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:09:14.227647 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-713de72d-4fdd-4f04-8548-1ae2bcbbe5ca admin admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:09:14.237509 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 1289/2564] 10.222.0.30 () {58 vars in 1121 bytes} [Mon May 23 13:09:14 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) May 23 13:09:14.242698 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c9253bb-4557-4cd0-841c-b60f4649f15c admin admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:09:14.252450 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c9253bb-4557-4cd0-841c-b60f4649f15c admin admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:09:14.258153 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae46f9ca-0602-41e8-800a-b2404c2407b8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:09:14.259020 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae46f9ca-0602-41e8-800a-b2404c2407b8 admin admin] REQUEST_METHOD: `PUT` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:09:14.259297 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae46f9ca-0602-41e8-800a-b2404c2407b8 admin admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:09:14.259564 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae46f9ca-0602-41e8-800a-b2404c2407b8 admin admin] PATH_INFO: `/v3/projects/8f189c4b75884eec83268238321a8a38/users/d02a9ba61c9446b9863951948f245e88/roles/80fb22507c9a4ff09d44d8f12f6fff1a` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:09:14.274577 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae46f9ca-0602-41e8-800a-b2404c2407b8 admin admin] RBAC: Authorizing `identity:create_grant(project_id=8f189c4b75884eec83268238321a8a38, user_id=d02a9ba61c9446b9863951948f245e88, role_id=80fb22507c9a4ff09d44d8f12f6fff1a)` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:09:14.280599 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae46f9ca-0602-41e8-800a-b2404c2407b8 admin admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:09:14.365255 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 1276/2565] 10.222.0.30 () {60 vars in 1482 bytes} [Mon May 23 13:09:14 2022] PUT /identity/v3/projects/8f189c4b75884eec83268238321a8a38/users/d02a9ba61c9446b9863951948f245e88/roles/80fb22507c9a4ff09d44d8f12f6fff1a => generated 0 bytes in 123 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) May 23 13:09:14.375598 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25146214-7d86-478c-a1c7-a5316c9d73c2 None None] REQUEST_METHOD: `POST` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:09:14.375932 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25146214-7d86-478c-a1c7-a5316c9d73c2 None None] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:09:14.376233 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25146214-7d86-478c-a1c7-a5316c9d73c2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:09:14.383056 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:DELETE=7,keystone:SELECT=89,keystone:INSERT=4 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 13:09:14.402988 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.auth.core [None req-25146214-7d86-478c-a1c7-a5316c9d73c2 None None] MFA Rules not processed for user `d02a9ba61c9446b9863951948f245e88`. Rule list: `[]` (Enabled: `True`). {{(pid=92057) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 13:09:14.404202 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.fernet_utils [None req-25146214-7d86-478c-a1c7-a5316c9d73c2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92057) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:09:14.466974 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 1290/2566] 10.222.0.30 () {58 vars in 957 bytes} [Mon May 23 13:09:14 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 92 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) May 23 13:09:14.471692 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae46f9ca-0602-41e8-800a-b2404c2407b8 admin admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:09:14.513367 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 13:09:14.533491 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae46f9ca-0602-41e8-800a-b2404c2407b8 admin admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:09:14.538717 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91099531-b253-4a01-8776-0585adb2a445 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:09:14.539370 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91099531-b253-4a01-8776-0585adb2a445 admin admin] REQUEST_METHOD: `POST` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:09:14.543823 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91099531-b253-4a01-8776-0585adb2a445 admin admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:09:14.543823 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91099531-b253-4a01-8776-0585adb2a445 admin admin] PATH_INFO: `/v3/projects` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:09:14.543823 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-91099531-b253-4a01-8776-0585adb2a445 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:09:14.543823 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-91099531-b253-4a01-8776-0585adb2a445 admin admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:09:14.638184 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 1277/2567] 10.222.0.30 () {60 vars in 1152 bytes} [Mon May 23 13:09:14 2022] POST /identity/v3/projects => generated 383 bytes in 167 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) May 23 13:09:14.642766 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25146214-7d86-478c-a1c7-a5316c9d73c2 None None] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:09:14.654274 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25146214-7d86-478c-a1c7-a5316c9d73c2 None None] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:09:14.711476 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74c59ada-0490-4d4d-9ae0-f692014166a4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:09:14.712152 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74c59ada-0490-4d4d-9ae0-f692014166a4 admin admin] REQUEST_METHOD: `POST` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:09:14.712436 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74c59ada-0490-4d4d-9ae0-f692014166a4 admin admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:09:14.712711 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74c59ada-0490-4d4d-9ae0-f692014166a4 admin admin] PATH_INFO: `/v3/users` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:09:14.713397 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74c59ada-0490-4d4d-9ae0-f692014166a4 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:09:14.715375 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74c59ada-0490-4d4d-9ae0-f692014166a4 admin admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:09:14.791316 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 1291/2568] 10.222.0.30 () {60 vars in 1143 bytes} [Mon May 23 13:09:14 2022] POST /identity/v3/users => generated 298 bytes in 149 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) May 23 13:09:14.795347 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91099531-b253-4a01-8776-0585adb2a445 admin admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:09:14.806013 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91099531-b253-4a01-8776-0585adb2a445 admin admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:09:14.811071 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-557d9df3-293c-42d1-9c79-9a838c181138 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:09:14.811728 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-557d9df3-293c-42d1-9c79-9a838c181138 admin admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:09:14.812019 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-557d9df3-293c-42d1-9c79-9a838c181138 admin admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:09:14.812386 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-557d9df3-293c-42d1-9c79-9a838c181138 admin admin] PATH_INFO: `/v3/roles` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:09:14.813122 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-557d9df3-293c-42d1-9c79-9a838c181138 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:09:14.814792 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-557d9df3-293c-42d1-9c79-9a838c181138 admin admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:09:14.826623 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 1278/2569] 10.222.0.30 () {58 vars in 1121 bytes} [Mon May 23 13:09:14 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) May 23 13:09:14.830834 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74c59ada-0490-4d4d-9ae0-f692014166a4 admin admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:09:14.841836 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74c59ada-0490-4d4d-9ae0-f692014166a4 admin admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:09:14.846484 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c41e9aa-2edc-489a-a43b-a1109d51236e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:09:14.847377 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c41e9aa-2edc-489a-a43b-a1109d51236e admin admin] REQUEST_METHOD: `PUT` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:09:14.848353 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c41e9aa-2edc-489a-a43b-a1109d51236e admin admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:09:14.848734 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c41e9aa-2edc-489a-a43b-a1109d51236e admin admin] PATH_INFO: `/v3/projects/1b8c7a71139541819c7530e5703e34a4/users/dbd7b842a9b745d7835adeca872d5a33/roles/bc3c3c9d654940cbaa5177a6be23d1bc` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:09:14.864394 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c41e9aa-2edc-489a-a43b-a1109d51236e admin admin] RBAC: Authorizing `identity:create_grant(project_id=1b8c7a71139541819c7530e5703e34a4, user_id=dbd7b842a9b745d7835adeca872d5a33, role_id=bc3c3c9d654940cbaa5177a6be23d1bc)` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:09:14.867082 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c41e9aa-2edc-489a-a43b-a1109d51236e admin admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:09:14.980066 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 1292/2570] 10.222.0.30 () {60 vars in 1482 bytes} [Mon May 23 13:09:14 2022] PUT /identity/v3/projects/1b8c7a71139541819c7530e5703e34a4/users/dbd7b842a9b745d7835adeca872d5a33/roles/bc3c3c9d654940cbaa5177a6be23d1bc => generated 0 bytes in 150 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) May 23 13:09:14.984187 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-557d9df3-293c-42d1-9c79-9a838c181138 admin admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:09:14.994471 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-557d9df3-293c-42d1-9c79-9a838c181138 admin admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:09:15.052945 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7cf23db8-ef8f-4aed-8f49-99f96a0c5522 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:09:15.053616 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7cf23db8-ef8f-4aed-8f49-99f96a0c5522 admin admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:09:15.053922 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7cf23db8-ef8f-4aed-8f49-99f96a0c5522 admin admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:09:15.054207 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7cf23db8-ef8f-4aed-8f49-99f96a0c5522 admin admin] PATH_INFO: `/v3/roles` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:09:15.054934 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7cf23db8-ef8f-4aed-8f49-99f96a0c5522 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:09:15.056541 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7cf23db8-ef8f-4aed-8f49-99f96a0c5522 admin admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:09:15.067137 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 1279/2571] 10.222.0.30 () {58 vars in 1121 bytes} [Mon May 23 13:09:14 2022] GET /identity/v3/roles => generated 1420 bytes in 84 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) May 23 13:09:15.071475 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c41e9aa-2edc-489a-a43b-a1109d51236e admin admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:09:15.081588 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c41e9aa-2edc-489a-a43b-a1109d51236e admin admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:09:15.086327 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7372ed74-5174-466c-8aab-76b0df1a75f9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:09:15.087246 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7372ed74-5174-466c-8aab-76b0df1a75f9 admin admin] REQUEST_METHOD: `PUT` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:09:15.087530 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7372ed74-5174-466c-8aab-76b0df1a75f9 admin admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:09:15.087805 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7372ed74-5174-466c-8aab-76b0df1a75f9 admin admin] PATH_INFO: `/v3/projects/1b8c7a71139541819c7530e5703e34a4/users/dbd7b842a9b745d7835adeca872d5a33/roles/bc3c3c9d654940cbaa5177a6be23d1bc` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:09:15.089724 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7372ed74-5174-466c-8aab-76b0df1a75f9 admin admin] RBAC: Authorizing `identity:create_grant(project_id=1b8c7a71139541819c7530e5703e34a4, user_id=dbd7b842a9b745d7835adeca872d5a33, role_id=bc3c3c9d654940cbaa5177a6be23d1bc)` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:09:15.092405 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7372ed74-5174-466c-8aab-76b0df1a75f9 admin admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:09:15.103085 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 1293/2572] 10.222.0.30 () {60 vars in 1482 bytes} [Mon May 23 13:09:15 2022] PUT /identity/v3/projects/1b8c7a71139541819c7530e5703e34a4/users/dbd7b842a9b745d7835adeca872d5a33/roles/bc3c3c9d654940cbaa5177a6be23d1bc => generated 0 bytes in 32 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) May 23 13:09:15.119688 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efb8b272-227a-4a28-aa38-149ed489c3c4 None None] REQUEST_METHOD: `POST` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:09:15.119981 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efb8b272-227a-4a28-aa38-149ed489c3c4 None None] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:09:15.120322 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efb8b272-227a-4a28-aa38-149ed489c3c4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:09:15.146961 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.auth.core [None req-efb8b272-227a-4a28-aa38-149ed489c3c4 None None] MFA Rules not processed for user `dbd7b842a9b745d7835adeca872d5a33`. Rule list: `[]` (Enabled: `True`). {{(pid=92056) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 13:09:15.148429 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.fernet_utils [None req-efb8b272-227a-4a28-aa38-149ed489c3c4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92056) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:09:15.216823 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 1280/2573] 10.222.0.30 () {58 vars in 957 bytes} [Mon May 23 13:09:15 2022] POST /identity/v3/auth/tokens => generated 3317 bytes in 98 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) May 23 13:09:15.232571 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7372ed74-5174-466c-8aab-76b0df1a75f9 admin admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:09:15.234257 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.fernet_utils [None req-7372ed74-5174-466c-8aab-76b0df1a75f9 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92057) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:09:15.286266 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 13:09:15.291044 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7372ed74-5174-466c-8aab-76b0df1a75f9 admin admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:09:15.294664 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9cace82a-3647-416e-bc61-ebe5a4852b1a service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1666097f8ed545c6b0e4809c93e4775a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9c389ae65aaa4542bb5a5c4fd0bacf5b', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:09:15.295293 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cace82a-3647-416e-bc61-ebe5a4852b1a service cinder] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:09:15.295573 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cace82a-3647-416e-bc61-ebe5a4852b1a service cinder] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:09:15.295846 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cace82a-3647-416e-bc61-ebe5a4852b1a service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:09:15.354416 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9cace82a-3647-416e-bc61-ebe5a4852b1a service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:09:15.355782 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9cace82a-3647-416e-bc61-ebe5a4852b1a service cinder] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:09:15.365208 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 1294/2574] 10.222.0.30 () {60 vars in 1365 bytes} [Mon May 23 13:09:15 2022] GET /identity/v3/auth/tokens => generated 3250 bytes in 133 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) May 23 13:09:20.318274 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-efb8b272-227a-4a28-aa38-149ed489c3c4 None None] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:09:20.319678 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.fernet_utils [None req-efb8b272-227a-4a28-aa38-149ed489c3c4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92056) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:09:20.332319 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-efb8b272-227a-4a28-aa38-149ed489c3c4 None None] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:09:20.336910 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5493bfb3-9836-4346-93a0-642c74e60140 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1666097f8ed545c6b0e4809c93e4775a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9c389ae65aaa4542bb5a5c4fd0bacf5b', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:09:20.337549 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5493bfb3-9836-4346-93a0-642c74e60140 service cinder] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:09:20.337866 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5493bfb3-9836-4346-93a0-642c74e60140 service cinder] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:09:20.338193 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5493bfb3-9836-4346-93a0-642c74e60140 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:09:20.347050 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5493bfb3-9836-4346-93a0-642c74e60140 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:09:20.348487 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5493bfb3-9836-4346-93a0-642c74e60140 service cinder] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:09:20.358166 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 1281/2575] 10.222.0.30 () {60 vars in 1365 bytes} [Mon May 23 13:09:20 2022] GET /identity/v3/auth/tokens => generated 3317 bytes in 40 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) May 23 13:09:25.360641 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=27 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 13:09:30.354047 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=90,keystone:INSERT=1 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 13:09:33.262828 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9cace82a-3647-416e-bc61-ebe5a4852b1a service cinder] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:09:33.325105 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9cace82a-3647-416e-bc61-ebe5a4852b1a service cinder] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:09:33.329559 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c31df80-6b9a-4288-9f26-6da366371fe9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:09:33.330457 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c31df80-6b9a-4288-9f26-6da366371fe9 admin admin] REQUEST_METHOD: `DELETE` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:09:33.330754 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c31df80-6b9a-4288-9f26-6da366371fe9 admin admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:09:33.331018 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c31df80-6b9a-4288-9f26-6da366371fe9 admin admin] PATH_INFO: `/v3/users/d02a9ba61c9446b9863951948f245e88` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:09:33.332188 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c31df80-6b9a-4288-9f26-6da366371fe9 admin admin] RBAC: Authorizing `identity:delete_user(user_id=d02a9ba61c9446b9863951948f245e88)` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:09:33.334152 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c31df80-6b9a-4288-9f26-6da366371fe9 admin admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:09:33.499729 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [None req-5c31df80-6b9a-4288-9f26-6da366371fe9 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'd02a9ba61c9446b9863951948f245e88'} {{(pid=92057) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:09:33.552219 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [None req-5c31df80-6b9a-4288-9f26-6da366371fe9 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'd02a9ba61c9446b9863951948f245e88'} {{(pid=92057) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:09:33.559552 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [None req-5c31df80-6b9a-4288-9f26-6da366371fe9 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'd02a9ba61c9446b9863951948f245e88'} {{(pid=92057) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:09:33.560195 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [None req-5c31df80-6b9a-4288-9f26-6da366371fe9 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'd02a9ba61c9446b9863951948f245e88'} {{(pid=92057) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:09:33.572631 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [None req-5c31df80-6b9a-4288-9f26-6da366371fe9 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'd02a9ba61c9446b9863951948f245e88'} {{(pid=92057) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:09:33.578638 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 1295/2576] 10.222.0.30 () {58 vars in 1223 bytes} [Mon May 23 13:09:33 2022] DELETE /identity/v3/users/d02a9ba61c9446b9863951948f245e88 => generated 0 bytes in 316 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) May 23 13:09:33.583117 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f8bf14b-2e07-40e4-a9db-b69280e094b3 None None] REQUEST_METHOD: `POST` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:09:33.583604 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f8bf14b-2e07-40e4-a9db-b69280e094b3 None None] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:09:33.583710 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f8bf14b-2e07-40e4-a9db-b69280e094b3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:09:33.601188 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.auth.core [None req-3f8bf14b-2e07-40e4-a9db-b69280e094b3 None None] MFA Rules not processed for user `0f7119f8fa5c4445977671140ded91a4`. Rule list: `[]` (Enabled: `True`). {{(pid=92056) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 13:09:33.602419 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.fernet_utils [None req-3f8bf14b-2e07-40e4-a9db-b69280e094b3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92056) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:09:33.660727 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 1282/2577] 10.222.0.30 () {58 vars in 957 bytes} [Mon May 23 13:09:33 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 79 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) May 23 13:09:33.670441 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c31df80-6b9a-4288-9f26-6da366371fe9 admin admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:09:33.672487 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.fernet_utils [None req-5c31df80-6b9a-4288-9f26-6da366371fe9 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92057) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:09:33.731872 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c31df80-6b9a-4288-9f26-6da366371fe9 admin admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:09:33.735675 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9560cc5b-dc30-45aa-a45a-ae726f400ea4 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2eb83ca38e79449095f3a2f947bbb06a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9c389ae65aaa4542bb5a5c4fd0bacf5b', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:09:33.736321 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9560cc5b-dc30-45aa-a45a-ae726f400ea4 service neutron] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:09:33.736629 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9560cc5b-dc30-45aa-a45a-ae726f400ea4 service neutron] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:09:33.736906 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9560cc5b-dc30-45aa-a45a-ae726f400ea4 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:09:33.746544 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9560cc5b-dc30-45aa-a45a-ae726f400ea4 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:09:33.747993 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9560cc5b-dc30-45aa-a45a-ae726f400ea4 service neutron] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:09:33.757600 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 1296/2578] 10.222.0.30 () {60 vars in 1365 bytes} [Mon May 23 13:09:33 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 88 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) May 23 13:09:33.905043 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc9a30fe-7344-4265-bae0-fa01a41dc3e7 None None] REQUEST_METHOD: `POST` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:09:33.909816 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc9a30fe-7344-4265-bae0-fa01a41dc3e7 None None] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:09:33.909816 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc9a30fe-7344-4265-bae0-fa01a41dc3e7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:09:33.922834 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.auth.core [None req-bc9a30fe-7344-4265-bae0-fa01a41dc3e7 None None] MFA Rules not processed for user `c6a6096760734ac49ce2f26c6065b701`. Rule list: `[]` (Enabled: `True`). {{(pid=92056) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 13:09:33.924019 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.fernet_utils [None req-bc9a30fe-7344-4265-bae0-fa01a41dc3e7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92056) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:09:33.983429 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 1283/2579] 10.222.0.30 () {58 vars in 957 bytes} [Mon May 23 13:09:33 2022] POST /identity/v3/auth/tokens => generated 3246 bytes in 79 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) May 23 13:09:34.171914 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9560cc5b-dc30-45aa-a45a-ae726f400ea4 service neutron] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:09:34.176440 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.fernet_utils [None req-9560cc5b-dc30-45aa-a45a-ae726f400ea4 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92057) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:09:34.197050 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9560cc5b-dc30-45aa-a45a-ae726f400ea4 service neutron] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:09:34.203320 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ee7c8de-782e-4e13-a03b-15d2c0745247 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:09:34.205084 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ee7c8de-782e-4e13-a03b-15d2c0745247 admin admin] REQUEST_METHOD: `DELETE` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:09:34.205084 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ee7c8de-782e-4e13-a03b-15d2c0745247 admin admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:09:34.205084 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ee7c8de-782e-4e13-a03b-15d2c0745247 admin admin] PATH_INFO: `/v3/projects/8f189c4b75884eec83268238321a8a38` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:09:34.205964 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ee7c8de-782e-4e13-a03b-15d2c0745247 admin admin] RBAC: Authorizing `identity:delete_project(project_id=8f189c4b75884eec83268238321a8a38)` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:09:34.207908 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ee7c8de-782e-4e13-a03b-15d2c0745247 admin admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:09:34.226288 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 13:09:34.383388 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [None req-2ee7c8de-782e-4e13-a03b-15d2c0745247 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '8f189c4b75884eec83268238321a8a38', 'request_id': 'req-2ee7c8de-782e-4e13-a03b-15d2c0745247'} {{(pid=92057) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:09:34.389297 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [None req-2ee7c8de-782e-4e13-a03b-15d2c0745247 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '8f189c4b75884eec83268238321a8a38', 'request_id': 'req-2ee7c8de-782e-4e13-a03b-15d2c0745247'} {{(pid=92057) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:09:34.461250 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [None req-2ee7c8de-782e-4e13-a03b-15d2c0745247 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '8f189c4b75884eec83268238321a8a38', 'request_id': 'req-2ee7c8de-782e-4e13-a03b-15d2c0745247'} {{(pid=92057) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:09:34.473700 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [None req-2ee7c8de-782e-4e13-a03b-15d2c0745247 admin admin] The token cache is being invalidate because project 8f189c4b75884eec83268238321a8a38 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=92057) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} May 23 13:09:34.474150 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [None req-2ee7c8de-782e-4e13-a03b-15d2c0745247 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=92057) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:09:34.474786 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [None req-2ee7c8de-782e-4e13-a03b-15d2c0745247 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=92057) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:09:34.476331 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 1297/2580] 10.222.0.30 () {58 vars in 1232 bytes} [Mon May 23 13:09:34 2022] DELETE /identity/v3/projects/8f189c4b75884eec83268238321a8a38 => generated 0 bytes in 305 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) May 23 13:09:34.480895 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc9a30fe-7344-4265-bae0-fa01a41dc3e7 None None] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:09:34.492131 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc9a30fe-7344-4265-bae0-fa01a41dc3e7 None None] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:09:34.554678 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eafe5319-be09-4959-a0ee-0dd2ed954b20 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:09:34.555557 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eafe5319-be09-4959-a0ee-0dd2ed954b20 admin admin] REQUEST_METHOD: `DELETE` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:09:34.555841 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eafe5319-be09-4959-a0ee-0dd2ed954b20 admin admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:09:34.556118 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eafe5319-be09-4959-a0ee-0dd2ed954b20 admin admin] PATH_INFO: `/v3/users/dbd7b842a9b745d7835adeca872d5a33` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:09:34.559325 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eafe5319-be09-4959-a0ee-0dd2ed954b20 admin admin] RBAC: Authorizing `identity:delete_user(user_id=dbd7b842a9b745d7835adeca872d5a33)` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:09:34.561254 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eafe5319-be09-4959-a0ee-0dd2ed954b20 admin admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:09:34.738181 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-eafe5319-be09-4959-a0ee-0dd2ed954b20 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'dbd7b842a9b745d7835adeca872d5a33'} {{(pid=92056) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:09:34.738828 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-eafe5319-be09-4959-a0ee-0dd2ed954b20 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'dbd7b842a9b745d7835adeca872d5a33'} {{(pid=92056) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:09:34.740544 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=93,keystone:DELETE=7 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 13:09:34.833113 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-eafe5319-be09-4959-a0ee-0dd2ed954b20 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'dbd7b842a9b745d7835adeca872d5a33'} {{(pid=92056) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:09:34.845954 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-eafe5319-be09-4959-a0ee-0dd2ed954b20 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'dbd7b842a9b745d7835adeca872d5a33'} {{(pid=92056) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:09:34.852745 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-eafe5319-be09-4959-a0ee-0dd2ed954b20 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'dbd7b842a9b745d7835adeca872d5a33'} {{(pid=92056) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:09:34.859956 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 1284/2581] 10.222.0.30 () {58 vars in 1223 bytes} [Mon May 23 13:09:34 2022] DELETE /identity/v3/users/dbd7b842a9b745d7835adeca872d5a33 => generated 0 bytes in 380 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) May 23 13:09:35.354561 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ee7c8de-782e-4e13-a03b-15d2c0745247 admin admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:09:35.356217 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.fernet_utils [None req-2ee7c8de-782e-4e13-a03b-15d2c0745247 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92057) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:09:35.373747 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ee7c8de-782e-4e13-a03b-15d2c0745247 admin admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:09:35.440896 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b0dbf8c-1e55-4d0e-a04f-d7d9b06971a6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:09:35.441743 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b0dbf8c-1e55-4d0e-a04f-d7d9b06971a6 admin admin] REQUEST_METHOD: `DELETE` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:09:35.442055 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b0dbf8c-1e55-4d0e-a04f-d7d9b06971a6 admin admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:09:35.442328 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b0dbf8c-1e55-4d0e-a04f-d7d9b06971a6 admin admin] PATH_INFO: `/v3/projects/1b8c7a71139541819c7530e5703e34a4` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:09:35.443513 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b0dbf8c-1e55-4d0e-a04f-d7d9b06971a6 admin admin] RBAC: Authorizing `identity:delete_project(project_id=1b8c7a71139541819c7530e5703e34a4)` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:09:35.445403 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b0dbf8c-1e55-4d0e-a04f-d7d9b06971a6 admin admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:09:35.551045 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [None req-5b0dbf8c-1e55-4d0e-a04f-d7d9b06971a6 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '1b8c7a71139541819c7530e5703e34a4', 'request_id': 'req-5b0dbf8c-1e55-4d0e-a04f-d7d9b06971a6'} {{(pid=92057) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:09:35.556426 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [None req-5b0dbf8c-1e55-4d0e-a04f-d7d9b06971a6 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '1b8c7a71139541819c7530e5703e34a4', 'request_id': 'req-5b0dbf8c-1e55-4d0e-a04f-d7d9b06971a6'} {{(pid=92057) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:09:35.615147 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [None req-5b0dbf8c-1e55-4d0e-a04f-d7d9b06971a6 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '1b8c7a71139541819c7530e5703e34a4', 'request_id': 'req-5b0dbf8c-1e55-4d0e-a04f-d7d9b06971a6'} {{(pid=92057) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:09:35.627207 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [None req-5b0dbf8c-1e55-4d0e-a04f-d7d9b06971a6 admin admin] The token cache is being invalidate because project 1b8c7a71139541819c7530e5703e34a4 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=92057) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} May 23 13:09:35.627494 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [None req-5b0dbf8c-1e55-4d0e-a04f-d7d9b06971a6 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=92057) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:09:35.628108 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [None req-5b0dbf8c-1e55-4d0e-a04f-d7d9b06971a6 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=92057) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:09:35.629639 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 1298/2582] 10.222.0.30 () {58 vars in 1232 bytes} [Mon May 23 13:09:35 2022] DELETE /identity/v3/projects/1b8c7a71139541819c7530e5703e34a4 => generated 0 bytes in 276 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) May 23 13:09:35.636260 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd4b6c00-8a0b-4cc6-959e-3a5a58b9ac51 None None] REQUEST_METHOD: `POST` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:09:35.636660 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd4b6c00-8a0b-4cc6-959e-3a5a58b9ac51 None None] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:09:35.636921 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd4b6c00-8a0b-4cc6-959e-3a5a58b9ac51 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:09:35.654971 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.auth.core [None req-dd4b6c00-8a0b-4cc6-959e-3a5a58b9ac51 None None] MFA Rules not processed for user `0f7119f8fa5c4445977671140ded91a4`. Rule list: `[]` (Enabled: `True`). {{(pid=92056) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 13:09:35.656148 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.fernet_utils [None req-dd4b6c00-8a0b-4cc6-959e-3a5a58b9ac51 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92056) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:09:35.716168 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 1285/2583] 10.222.0.30 () {58 vars in 957 bytes} [Mon May 23 13:09:35 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 82 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) May 23 13:09:35.724750 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ca3b50b-de53-4b66-ab3a-4ac5bfb4222b None None] REQUEST_METHOD: `POST` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:09:35.725042 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ca3b50b-de53-4b66-ab3a-4ac5bfb4222b None None] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:09:35.725319 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ca3b50b-de53-4b66-ab3a-4ac5bfb4222b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:09:35.743492 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.auth.core [None req-0ca3b50b-de53-4b66-ab3a-4ac5bfb4222b None None] MFA Rules not processed for user `0f7119f8fa5c4445977671140ded91a4`. Rule list: `[]` (Enabled: `True`). {{(pid=92057) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 13:09:35.745262 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.fernet_utils [None req-0ca3b50b-de53-4b66-ab3a-4ac5bfb4222b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92057) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:09:35.753194 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 1299/2584] 10.222.0.30 () {58 vars in 957 bytes} [Mon May 23 13:09:35 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 29 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) May 23 13:09:35.757327 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd4b6c00-8a0b-4cc6-959e-3a5a58b9ac51 None None] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:09:35.767553 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd4b6c00-8a0b-4cc6-959e-3a5a58b9ac51 None None] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:09:35.772203 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6f6789d-4203-4276-887d-261959a39449 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:09:35.773129 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6f6789d-4203-4276-887d-261959a39449 admin admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:09:35.773440 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6f6789d-4203-4276-887d-261959a39449 admin admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:09:35.773722 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6f6789d-4203-4276-887d-261959a39449 admin admin] PATH_INFO: `/v3/domains` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:09:35.774491 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6f6789d-4203-4276-887d-261959a39449 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=92056) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} May 23 13:09:35.774804 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6f6789d-4203-4276-887d-261959a39449 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:09:35.776886 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6f6789d-4203-4276-887d-261959a39449 admin admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:09:35.790156 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 1286/2585] 10.222.0.30 () {58 vars in 1152 bytes} [Mon May 23 13:09:35 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) May 23 13:09:35.796606 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ca3b50b-de53-4b66-ab3a-4ac5bfb4222b None None] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:09:35.805856 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ca3b50b-de53-4b66-ab3a-4ac5bfb4222b None None] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:09:35.811654 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b68b12f1-e9ee-4203-bd32-58e65e9bb035 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:09:35.812301 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b68b12f1-e9ee-4203-bd32-58e65e9bb035 admin admin] REQUEST_METHOD: `POST` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:09:35.812579 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b68b12f1-e9ee-4203-bd32-58e65e9bb035 admin admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:09:35.812848 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b68b12f1-e9ee-4203-bd32-58e65e9bb035 admin admin] PATH_INFO: `/v3/projects` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:09:35.813524 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b68b12f1-e9ee-4203-bd32-58e65e9bb035 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:09:35.815511 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b68b12f1-e9ee-4203-bd32-58e65e9bb035 admin admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:09:35.897032 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 1300/2586] 10.222.0.30 () {60 vars in 1152 bytes} [Mon May 23 13:09:35 2022] POST /identity/v3/projects => generated 371 bytes in 101 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) May 23 13:09:35.902045 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6f6789d-4203-4276-887d-261959a39449 admin admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:09:35.912455 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6f6789d-4203-4276-887d-261959a39449 admin admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:09:35.970176 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4050e983-7864-4131-8ece-98be4c01867b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:09:35.970841 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4050e983-7864-4131-8ece-98be4c01867b admin admin] REQUEST_METHOD: `POST` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:09:35.971133 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4050e983-7864-4131-8ece-98be4c01867b admin admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:09:35.971412 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4050e983-7864-4131-8ece-98be4c01867b admin admin] PATH_INFO: `/v3/users` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:09:35.972111 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4050e983-7864-4131-8ece-98be4c01867b admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:09:35.974137 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4050e983-7864-4131-8ece-98be4c01867b admin admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:09:36.044952 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 1287/2587] 10.222.0.30 () {60 vars in 1143 bytes} [Mon May 23 13:09:35 2022] POST /identity/v3/users => generated 286 bytes in 144 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) May 23 13:09:36.049190 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b68b12f1-e9ee-4203-bd32-58e65e9bb035 admin admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:09:36.060217 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b68b12f1-e9ee-4203-bd32-58e65e9bb035 admin admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:09:36.065051 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d599e1d-feb8-4e59-aaf7-694ee88d134e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:09:36.065748 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d599e1d-feb8-4e59-aaf7-694ee88d134e admin admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:09:36.066063 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d599e1d-feb8-4e59-aaf7-694ee88d134e admin admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:09:36.066344 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d599e1d-feb8-4e59-aaf7-694ee88d134e admin admin] PATH_INFO: `/v3/roles` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:09:36.067075 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d599e1d-feb8-4e59-aaf7-694ee88d134e admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:09:36.068685 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d599e1d-feb8-4e59-aaf7-694ee88d134e admin admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:09:36.070135 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=83,keystone:DELETE=14,keystone:INSERT=3 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 13:09:36.080461 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 1301/2588] 10.222.0.30 () {58 vars in 1121 bytes} [Mon May 23 13:09:36 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) May 23 13:09:36.084486 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4050e983-7864-4131-8ece-98be4c01867b admin admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:09:36.095021 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4050e983-7864-4131-8ece-98be4c01867b admin admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:09:36.099702 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46d03593-3228-4389-91f0-3cb67bae89d6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:09:36.100353 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46d03593-3228-4389-91f0-3cb67bae89d6 admin admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:09:36.100637 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46d03593-3228-4389-91f0-3cb67bae89d6 admin admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:09:36.100907 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46d03593-3228-4389-91f0-3cb67bae89d6 admin admin] PATH_INFO: `/v3/roles` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:09:36.101777 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46d03593-3228-4389-91f0-3cb67bae89d6 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:09:36.103403 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46d03593-3228-4389-91f0-3cb67bae89d6 admin admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:09:36.120616 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 1288/2589] 10.222.0.30 () {58 vars in 1121 bytes} [Mon May 23 13:09:36 2022] GET /identity/v3/roles => generated 1420 bytes in 37 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) May 23 13:09:36.125011 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d599e1d-feb8-4e59-aaf7-694ee88d134e admin admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:09:36.135522 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d599e1d-feb8-4e59-aaf7-694ee88d134e admin admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:09:36.144705 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e53ca548-d246-4587-b770-311be61ac0dd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:09:36.144705 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e53ca548-d246-4587-b770-311be61ac0dd admin admin] REQUEST_METHOD: `PUT` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:09:36.144705 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e53ca548-d246-4587-b770-311be61ac0dd admin admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:09:36.144705 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e53ca548-d246-4587-b770-311be61ac0dd admin admin] PATH_INFO: `/v3/projects/50473e6ceaae4c7e9051bb5afb6bbdb5/users/b000d815e7b64f37859b9c6a12194c96/roles/80fb22507c9a4ff09d44d8f12f6fff1a` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:09:36.158924 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e53ca548-d246-4587-b770-311be61ac0dd admin admin] RBAC: Authorizing `identity:create_grant(project_id=50473e6ceaae4c7e9051bb5afb6bbdb5, user_id=b000d815e7b64f37859b9c6a12194c96, role_id=80fb22507c9a4ff09d44d8f12f6fff1a)` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:09:36.161821 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e53ca548-d246-4587-b770-311be61ac0dd admin admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:09:36.251033 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 1302/2590] 10.222.0.30 () {60 vars in 1482 bytes} [Mon May 23 13:09:36 2022] PUT /identity/v3/projects/50473e6ceaae4c7e9051bb5afb6bbdb5/users/b000d815e7b64f37859b9c6a12194c96/roles/80fb22507c9a4ff09d44d8f12f6fff1a => generated 0 bytes in 126 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) May 23 13:09:36.263215 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36600d42-6789-4823-b18d-6eef587fd175 None None] REQUEST_METHOD: `POST` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:09:36.263506 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36600d42-6789-4823-b18d-6eef587fd175 None None] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:09:36.263801 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36600d42-6789-4823-b18d-6eef587fd175 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:09:36.289828 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.auth.core [None req-36600d42-6789-4823-b18d-6eef587fd175 None None] MFA Rules not processed for user `b000d815e7b64f37859b9c6a12194c96`. Rule list: `[]` (Enabled: `True`). {{(pid=92056) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 13:09:36.291065 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.fernet_utils [None req-36600d42-6789-4823-b18d-6eef587fd175 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92056) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:09:36.309932 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:INSERT=4,keystone:DELETE=3,keystone:SELECT=93 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 13:09:36.355976 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 1289/2591] 10.222.0.30 () {58 vars in 957 bytes} [Mon May 23 13:09:36 2022] POST /identity/v3/auth/tokens => generated 3238 bytes in 94 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) May 23 13:09:36.367705 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e53ca548-d246-4587-b770-311be61ac0dd admin admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:09:36.370421 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.fernet_utils [None req-e53ca548-d246-4587-b770-311be61ac0dd admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92057) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:09:36.428248 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e53ca548-d246-4587-b770-311be61ac0dd admin admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:09:36.431830 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-527b0671-8fb8-46e2-8261-9477bbbc5ddb service glance] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '5cea0feed4254ec085854986f99acf30', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9c389ae65aaa4542bb5a5c4fd0bacf5b', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:09:36.433009 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-527b0671-8fb8-46e2-8261-9477bbbc5ddb service glance] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:09:36.433283 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-527b0671-8fb8-46e2-8261-9477bbbc5ddb service glance] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:09:36.433538 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-527b0671-8fb8-46e2-8261-9477bbbc5ddb service glance] PATH_INFO: `/v3/auth/tokens` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:09:36.442333 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-527b0671-8fb8-46e2-8261-9477bbbc5ddb service glance] RBAC: Authorizing `identity:validate_token()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:09:36.445100 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-527b0671-8fb8-46e2-8261-9477bbbc5ddb service glance] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:09:36.453479 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 1303/2592] 10.222.0.30 () {60 vars in 1365 bytes} [Mon May 23 13:09:36 2022] GET /identity/v3/auth/tokens => generated 3238 bytes in 86 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) May 23 13:09:46.442400 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=18 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 13:09:46.449274 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=38,keystone:INSERT=1 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 13:09:55.312343 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36600d42-6789-4823-b18d-6eef587fd175 None None] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:09:55.313750 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.fernet_utils [None req-36600d42-6789-4823-b18d-6eef587fd175 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92056) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:09:55.370659 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36600d42-6789-4823-b18d-6eef587fd175 None None] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:09:55.374256 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26232996-c57d-4e65-9813-3fe08e31b656 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2eb83ca38e79449095f3a2f947bbb06a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9c389ae65aaa4542bb5a5c4fd0bacf5b', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:09:55.374886 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26232996-c57d-4e65-9813-3fe08e31b656 service neutron] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:09:55.375168 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26232996-c57d-4e65-9813-3fe08e31b656 service neutron] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:09:55.375443 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26232996-c57d-4e65-9813-3fe08e31b656 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:09:55.377211 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.fernet_utils [None req-26232996-c57d-4e65-9813-3fe08e31b656 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92056) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:09:55.438193 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26232996-c57d-4e65-9813-3fe08e31b656 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:09:55.439623 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26232996-c57d-4e65-9813-3fe08e31b656 service neutron] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:09:55.449067 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 1290/2593] 10.222.0.30 () {60 vars in 1365 bytes} [Mon May 23 13:09:55 2022] GET /identity/v3/auth/tokens => generated 3311 bytes in 137 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) May 23 13:09:58.354409 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-527b0671-8fb8-46e2-8261-9477bbbc5ddb service glance] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:09:58.369700 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-527b0671-8fb8-46e2-8261-9477bbbc5ddb service glance] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:09:58.373313 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d595d6c-634b-4dc9-be22-03fb96ba7b97 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2eb83ca38e79449095f3a2f947bbb06a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9c389ae65aaa4542bb5a5c4fd0bacf5b', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:09:58.375180 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d595d6c-634b-4dc9-be22-03fb96ba7b97 service neutron] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:09:58.375473 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d595d6c-634b-4dc9-be22-03fb96ba7b97 service neutron] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:09:58.375748 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d595d6c-634b-4dc9-be22-03fb96ba7b97 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:09:58.377575 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.fernet_utils [None req-2d595d6c-634b-4dc9-be22-03fb96ba7b97 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92057) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:09:58.467221 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d595d6c-634b-4dc9-be22-03fb96ba7b97 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:09:58.468733 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d595d6c-634b-4dc9-be22-03fb96ba7b97 service neutron] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:09:58.480425 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 1304/2594] 10.222.0.30 () {60 vars in 1365 bytes} [Mon May 23 13:09:58 2022] GET /identity/v3/auth/tokens => generated 3362 bytes in 127 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) May 23 13:09:59.658960 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26232996-c57d-4e65-9813-3fe08e31b656 service neutron] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:09:59.667463 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26232996-c57d-4e65-9813-3fe08e31b656 service neutron] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:09:59.671801 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8694006b-0df3-4b9d-b202-f642418eec26 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2eb83ca38e79449095f3a2f947bbb06a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9c389ae65aaa4542bb5a5c4fd0bacf5b', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:09:59.672432 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8694006b-0df3-4b9d-b202-f642418eec26 service neutron] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:09:59.672711 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8694006b-0df3-4b9d-b202-f642418eec26 service neutron] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:09:59.672985 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8694006b-0df3-4b9d-b202-f642418eec26 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:09:59.674717 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.fernet_utils [None req-8694006b-0df3-4b9d-b202-f642418eec26 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92056) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:09:59.688636 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8694006b-0df3-4b9d-b202-f642418eec26 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:09:59.690092 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8694006b-0df3-4b9d-b202-f642418eec26 service neutron] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:09:59.701218 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 1291/2595] 10.222.0.30 () {60 vars in 1365 bytes} [Mon May 23 13:09:59 2022] GET /identity/v3/auth/tokens => generated 3362 bytes in 43 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) May 23 13:10:08.475241 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=34 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 13:10:09.696228 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=56 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 13:10:12.284823 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98692847-3e66-4de6-aada-751b7d8fc28b None None] REQUEST_METHOD: `POST` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:10:12.285296 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98692847-3e66-4de6-aada-751b7d8fc28b None None] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:10:12.285401 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98692847-3e66-4de6-aada-751b7d8fc28b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:10:12.303449 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.auth.core [None req-98692847-3e66-4de6-aada-751b7d8fc28b None None] MFA Rules not processed for user `6d21a56126fd4ccd837c33ccc08ab609`. Rule list: `[]` (Enabled: `True`). {{(pid=92057) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 13:10:12.304676 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.fernet_utils [None req-98692847-3e66-4de6-aada-751b7d8fc28b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92057) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:10:12.360517 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 1305/2596] 10.222.0.30 () {58 vars in 1009 bytes} [Mon May 23 13:10:12 2022] POST /identity/v3/auth/tokens => generated 3185 bytes in 77 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) May 23 13:10:12.512813 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8694006b-0df3-4b9d-b202-f642418eec26 service neutron] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:10:12.516085 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.fernet_utils [None req-8694006b-0df3-4b9d-b202-f642418eec26 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92056) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:10:12.600503 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8694006b-0df3-4b9d-b202-f642418eec26 service neutron] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:10:12.605755 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75801b0e-3ce4-4c9f-8d02-8215d1e7c720 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '138d91edb99f450e846765f9eef00d62', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9c389ae65aaa4542bb5a5c4fd0bacf5b', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'service', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:10:12.606471 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75801b0e-3ce4-4c9f-8d02-8215d1e7c720 service swift] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:10:12.606753 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75801b0e-3ce4-4c9f-8d02-8215d1e7c720 service swift] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:10:12.607029 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75801b0e-3ce4-4c9f-8d02-8215d1e7c720 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:10:12.616276 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75801b0e-3ce4-4c9f-8d02-8215d1e7c720 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:10:12.617925 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75801b0e-3ce4-4c9f-8d02-8215d1e7c720 service swift] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:10:12.627091 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 1292/2597] 10.222.0.30 () {60 vars in 1384 bytes} [Mon May 23 13:10:12 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 117 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) May 23 13:10:17.570285 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98692847-3e66-4de6-aada-751b7d8fc28b None None] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:10:17.647068 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98692847-3e66-4de6-aada-751b7d8fc28b None None] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:10:17.652178 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a01b1a0b-15b4-4b64-9a0e-b3797a6c0a18 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:10:17.653213 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a01b1a0b-15b4-4b64-9a0e-b3797a6c0a18 admin admin] REQUEST_METHOD: `DELETE` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:10:17.653686 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a01b1a0b-15b4-4b64-9a0e-b3797a6c0a18 admin admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:10:17.653991 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a01b1a0b-15b4-4b64-9a0e-b3797a6c0a18 admin admin] PATH_INFO: `/v3/users/b000d815e7b64f37859b9c6a12194c96` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:10:17.656598 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a01b1a0b-15b4-4b64-9a0e-b3797a6c0a18 admin admin] RBAC: Authorizing `identity:delete_user(user_id=b000d815e7b64f37859b9c6a12194c96)` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:10:17.659596 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a01b1a0b-15b4-4b64-9a0e-b3797a6c0a18 admin admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:10:17.870141 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [None req-a01b1a0b-15b4-4b64-9a0e-b3797a6c0a18 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'b000d815e7b64f37859b9c6a12194c96'} {{(pid=92057) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:10:17.956234 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [None req-a01b1a0b-15b4-4b64-9a0e-b3797a6c0a18 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'b000d815e7b64f37859b9c6a12194c96'} {{(pid=92057) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:10:17.964859 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [None req-a01b1a0b-15b4-4b64-9a0e-b3797a6c0a18 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'b000d815e7b64f37859b9c6a12194c96'} {{(pid=92057) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:10:17.965528 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [None req-a01b1a0b-15b4-4b64-9a0e-b3797a6c0a18 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'b000d815e7b64f37859b9c6a12194c96'} {{(pid=92057) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:10:17.978685 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [None req-a01b1a0b-15b4-4b64-9a0e-b3797a6c0a18 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'b000d815e7b64f37859b9c6a12194c96'} {{(pid=92057) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:10:17.985541 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 1306/2598] 10.222.0.30 () {58 vars in 1223 bytes} [Mon May 23 13:10:17 2022] DELETE /identity/v3/users/b000d815e7b64f37859b9c6a12194c96 => generated 0 bytes in 416 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) May 23 13:10:17.990162 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52f9d14b-a849-4f97-9607-3bb2f6baf5c4 None None] REQUEST_METHOD: `POST` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:10:17.990654 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52f9d14b-a849-4f97-9607-3bb2f6baf5c4 None None] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:10:17.990776 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52f9d14b-a849-4f97-9607-3bb2f6baf5c4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:10:18.008283 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.auth.core [None req-52f9d14b-a849-4f97-9607-3bb2f6baf5c4 None None] MFA Rules not processed for user `0f7119f8fa5c4445977671140ded91a4`. Rule list: `[]` (Enabled: `True`). {{(pid=92056) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 13:10:18.009508 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.fernet_utils [None req-52f9d14b-a849-4f97-9607-3bb2f6baf5c4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92056) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:10:18.081432 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 1293/2599] 10.222.0.30 () {58 vars in 957 bytes} [Mon May 23 13:10:17 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 92 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) May 23 13:10:18.088044 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a01b1a0b-15b4-4b64-9a0e-b3797a6c0a18 admin admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:10:18.089749 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.fernet_utils [None req-a01b1a0b-15b4-4b64-9a0e-b3797a6c0a18 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92057) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:10:18.146920 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 13:10:18.181145 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a01b1a0b-15b4-4b64-9a0e-b3797a6c0a18 admin admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:10:18.185109 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67d4db64-ee4f-47c8-a443-fc1f028fbbaa service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2eb83ca38e79449095f3a2f947bbb06a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9c389ae65aaa4542bb5a5c4fd0bacf5b', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:10:18.185790 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67d4db64-ee4f-47c8-a443-fc1f028fbbaa service neutron] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:10:18.186101 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67d4db64-ee4f-47c8-a443-fc1f028fbbaa service neutron] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:10:18.186381 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67d4db64-ee4f-47c8-a443-fc1f028fbbaa service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:10:18.200198 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67d4db64-ee4f-47c8-a443-fc1f028fbbaa service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:10:18.201750 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67d4db64-ee4f-47c8-a443-fc1f028fbbaa service neutron] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:10:18.214443 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 1307/2600] 10.222.0.30 () {60 vars in 1365 bytes} [Mon May 23 13:10:18 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 127 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) May 23 13:10:19.032938 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52f9d14b-a849-4f97-9607-3bb2f6baf5c4 None None] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:10:19.035135 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.fernet_utils [None req-52f9d14b-a849-4f97-9607-3bb2f6baf5c4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92056) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:10:19.050533 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52f9d14b-a849-4f97-9607-3bb2f6baf5c4 None None] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:10:19.055485 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a124ca06-4e81-4a5f-b7bd-b1d8c18b3a28 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:10:19.056384 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a124ca06-4e81-4a5f-b7bd-b1d8c18b3a28 admin admin] REQUEST_METHOD: `DELETE` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:10:19.056700 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a124ca06-4e81-4a5f-b7bd-b1d8c18b3a28 admin admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:10:19.056983 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a124ca06-4e81-4a5f-b7bd-b1d8c18b3a28 admin admin] PATH_INFO: `/v3/projects/50473e6ceaae4c7e9051bb5afb6bbdb5` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:10:19.058458 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a124ca06-4e81-4a5f-b7bd-b1d8c18b3a28 admin admin] RBAC: Authorizing `identity:delete_project(project_id=50473e6ceaae4c7e9051bb5afb6bbdb5)` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:10:19.060443 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a124ca06-4e81-4a5f-b7bd-b1d8c18b3a28 admin admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:10:19.225549 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-a124ca06-4e81-4a5f-b7bd-b1d8c18b3a28 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '50473e6ceaae4c7e9051bb5afb6bbdb5', 'request_id': 'req-a124ca06-4e81-4a5f-b7bd-b1d8c18b3a28'} {{(pid=92056) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:10:19.294934 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-a124ca06-4e81-4a5f-b7bd-b1d8c18b3a28 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '50473e6ceaae4c7e9051bb5afb6bbdb5', 'request_id': 'req-a124ca06-4e81-4a5f-b7bd-b1d8c18b3a28'} {{(pid=92056) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:10:19.304158 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-a124ca06-4e81-4a5f-b7bd-b1d8c18b3a28 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '50473e6ceaae4c7e9051bb5afb6bbdb5', 'request_id': 'req-a124ca06-4e81-4a5f-b7bd-b1d8c18b3a28'} {{(pid=92056) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:10:19.316518 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-a124ca06-4e81-4a5f-b7bd-b1d8c18b3a28 admin admin] The token cache is being invalidate because project 50473e6ceaae4c7e9051bb5afb6bbdb5 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=92056) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} May 23 13:10:19.316853 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-a124ca06-4e81-4a5f-b7bd-b1d8c18b3a28 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=92056) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:10:19.317811 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-a124ca06-4e81-4a5f-b7bd-b1d8c18b3a28 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=92056) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:10:19.319551 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 1294/2601] 10.222.0.30 () {58 vars in 1232 bytes} [Mon May 23 13:10:19 2022] DELETE /identity/v3/projects/50473e6ceaae4c7e9051bb5afb6bbdb5 => generated 0 bytes in 287 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) May 23 13:10:19.324633 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0da0adb-d2b9-40c5-a277-916dcaeaf475 None None] REQUEST_METHOD: `POST` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:10:19.325136 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0da0adb-d2b9-40c5-a277-916dcaeaf475 None None] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:10:19.325262 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0da0adb-d2b9-40c5-a277-916dcaeaf475 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:10:19.345119 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.auth.core [None req-f0da0adb-d2b9-40c5-a277-916dcaeaf475 None None] MFA Rules not processed for user `0f7119f8fa5c4445977671140ded91a4`. Rule list: `[]` (Enabled: `True`). {{(pid=92057) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 13:10:19.346361 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.fernet_utils [None req-f0da0adb-d2b9-40c5-a277-916dcaeaf475 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92057) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:10:19.420187 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 1308/2602] 10.222.0.30 () {58 vars in 957 bytes} [Mon May 23 13:10:19 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 96 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) May 23 13:10:19.485270 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2a25385-d3b2-4471-8092-4cc4a19576ee None None] REQUEST_METHOD: `POST` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:10:19.485502 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2a25385-d3b2-4471-8092-4cc4a19576ee None None] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:10:19.485732 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2a25385-d3b2-4471-8092-4cc4a19576ee None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:10:19.518724 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.auth.core [None req-c2a25385-d3b2-4471-8092-4cc4a19576ee None None] MFA Rules not processed for user `0f7119f8fa5c4445977671140ded91a4`. Rule list: `[]` (Enabled: `True`). {{(pid=92056) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 13:10:19.519970 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.fernet_utils [None req-c2a25385-d3b2-4471-8092-4cc4a19576ee None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92056) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:10:19.528706 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 1295/2603] 10.222.0.30 () {58 vars in 957 bytes} [Mon May 23 13:10:19 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 45 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) May 23 13:10:19.533921 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0da0adb-d2b9-40c5-a277-916dcaeaf475 None None] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:10:19.543908 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0da0adb-d2b9-40c5-a277-916dcaeaf475 None None] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:10:19.549227 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a1fb373-c2ad-443f-b71d-5e6145a5241a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:10:19.549909 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a1fb373-c2ad-443f-b71d-5e6145a5241a admin admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:10:19.550199 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a1fb373-c2ad-443f-b71d-5e6145a5241a admin admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:10:19.550481 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a1fb373-c2ad-443f-b71d-5e6145a5241a admin admin] PATH_INFO: `/v3/domains` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:10:19.551201 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a1fb373-c2ad-443f-b71d-5e6145a5241a admin admin] RBAC: Adding query filter params (name=Default) {{(pid=92057) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} May 23 13:10:19.551506 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a1fb373-c2ad-443f-b71d-5e6145a5241a admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:10:19.553111 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a1fb373-c2ad-443f-b71d-5e6145a5241a admin admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:10:19.566933 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 1309/2604] 10.222.0.30 () {58 vars in 1152 bytes} [Mon May 23 13:10:19 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) May 23 13:10:19.571116 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2a25385-d3b2-4471-8092-4cc4a19576ee None None] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:10:19.580934 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2a25385-d3b2-4471-8092-4cc4a19576ee None None] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:10:19.585940 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa01882e-f0ec-48bf-9778-6cd77822b077 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:10:19.587251 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa01882e-f0ec-48bf-9778-6cd77822b077 admin admin] REQUEST_METHOD: `POST` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:10:19.587542 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa01882e-f0ec-48bf-9778-6cd77822b077 admin admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:10:19.587824 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa01882e-f0ec-48bf-9778-6cd77822b077 admin admin] PATH_INFO: `/v3/projects` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:10:19.588528 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa01882e-f0ec-48bf-9778-6cd77822b077 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:10:19.590451 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa01882e-f0ec-48bf-9778-6cd77822b077 admin admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:10:19.594794 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 13:10:19.692927 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 1296/2605] 10.222.0.30 () {60 vars in 1152 bytes} [Mon May 23 13:10:19 2022] POST /identity/v3/projects => generated 375 bytes in 122 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) May 23 13:10:19.699691 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a1fb373-c2ad-443f-b71d-5e6145a5241a admin admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:10:19.715645 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a1fb373-c2ad-443f-b71d-5e6145a5241a admin admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:10:19.785630 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b1fc5ce-083c-4be4-a953-b02bf9b109a3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:10:19.786361 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b1fc5ce-083c-4be4-a953-b02bf9b109a3 admin admin] REQUEST_METHOD: `POST` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:10:19.786651 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b1fc5ce-083c-4be4-a953-b02bf9b109a3 admin admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:10:19.786926 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b1fc5ce-083c-4be4-a953-b02bf9b109a3 admin admin] PATH_INFO: `/v3/users` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:10:19.787707 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b1fc5ce-083c-4be4-a953-b02bf9b109a3 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:10:19.789642 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b1fc5ce-083c-4be4-a953-b02bf9b109a3 admin admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:10:19.873669 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 1310/2606] 10.222.0.30 () {60 vars in 1143 bytes} [Mon May 23 13:10:19 2022] POST /identity/v3/users => generated 288 bytes in 175 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) May 23 13:10:19.878124 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa01882e-f0ec-48bf-9778-6cd77822b077 admin admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:10:19.893620 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa01882e-f0ec-48bf-9778-6cd77822b077 admin admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:10:19.901460 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d54c2ce2-a9f5-4301-addb-c55a83e4b42f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:10:19.902653 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d54c2ce2-a9f5-4301-addb-c55a83e4b42f admin admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:10:19.903155 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d54c2ce2-a9f5-4301-addb-c55a83e4b42f admin admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:10:19.903813 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d54c2ce2-a9f5-4301-addb-c55a83e4b42f admin admin] PATH_INFO: `/v3/roles` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:10:19.904757 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d54c2ce2-a9f5-4301-addb-c55a83e4b42f admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:10:19.906940 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d54c2ce2-a9f5-4301-addb-c55a83e4b42f admin admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:10:19.919806 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 1297/2607] 10.222.0.30 () {58 vars in 1121 bytes} [Mon May 23 13:10:19 2022] GET /identity/v3/roles => generated 1420 bytes in 42 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) May 23 13:10:19.923740 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b1fc5ce-083c-4be4-a953-b02bf9b109a3 admin admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:10:19.938580 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b1fc5ce-083c-4be4-a953-b02bf9b109a3 admin admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:10:19.944873 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24d19823-fa92-4a0f-806d-008cc1991aa9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:10:19.945813 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24d19823-fa92-4a0f-806d-008cc1991aa9 admin admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:10:19.946142 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24d19823-fa92-4a0f-806d-008cc1991aa9 admin admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:10:19.946424 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24d19823-fa92-4a0f-806d-008cc1991aa9 admin admin] PATH_INFO: `/v3/roles` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:10:19.947489 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24d19823-fa92-4a0f-806d-008cc1991aa9 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:10:19.948785 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24d19823-fa92-4a0f-806d-008cc1991aa9 admin admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:10:19.969240 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 1311/2608] 10.222.0.30 () {58 vars in 1121 bytes} [Mon May 23 13:10:19 2022] GET /identity/v3/roles => generated 1420 bytes in 43 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) May 23 13:10:19.970586 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d54c2ce2-a9f5-4301-addb-c55a83e4b42f admin admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:10:19.984818 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d54c2ce2-a9f5-4301-addb-c55a83e4b42f admin admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:10:19.992012 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f440e8da-5b24-434d-b5af-b65d30fc3179 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:10:19.993485 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f440e8da-5b24-434d-b5af-b65d30fc3179 admin admin] REQUEST_METHOD: `PUT` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:10:19.993776 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f440e8da-5b24-434d-b5af-b65d30fc3179 admin admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:10:19.994091 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f440e8da-5b24-434d-b5af-b65d30fc3179 admin admin] PATH_INFO: `/v3/projects/0ffdc054740c46f990ff4428f11cf07c/users/b19fc36cb5dc4aa9b684352cea0e6412/roles/80fb22507c9a4ff09d44d8f12f6fff1a` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:10:20.011834 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f440e8da-5b24-434d-b5af-b65d30fc3179 admin admin] RBAC: Authorizing `identity:create_grant(project_id=0ffdc054740c46f990ff4428f11cf07c, user_id=b19fc36cb5dc4aa9b684352cea0e6412, role_id=80fb22507c9a4ff09d44d8f12f6fff1a)` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:10:20.014580 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f440e8da-5b24-434d-b5af-b65d30fc3179 admin admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:10:20.124033 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 1298/2609] 10.222.0.30 () {60 vars in 1482 bytes} [Mon May 23 13:10:19 2022] PUT /identity/v3/projects/0ffdc054740c46f990ff4428f11cf07c/users/b19fc36cb5dc4aa9b684352cea0e6412/roles/80fb22507c9a4ff09d44d8f12f6fff1a => generated 0 bytes in 154 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) May 23 13:10:20.138190 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac7a1b03-e49f-4c4a-8eb2-e5ab3c7b0f56 None None] REQUEST_METHOD: `POST` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:10:20.138872 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac7a1b03-e49f-4c4a-8eb2-e5ab3c7b0f56 None None] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:10:20.139162 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac7a1b03-e49f-4c4a-8eb2-e5ab3c7b0f56 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:10:20.169369 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.auth.core [None req-ac7a1b03-e49f-4c4a-8eb2-e5ab3c7b0f56 None None] MFA Rules not processed for user `b19fc36cb5dc4aa9b684352cea0e6412`. Rule list: `[]` (Enabled: `True`). {{(pid=92057) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 13:10:20.170706 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.fernet_utils [None req-ac7a1b03-e49f-4c4a-8eb2-e5ab3c7b0f56 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92057) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:10:20.199140 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 13:10:20.259914 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 1312/2610] 10.222.0.30 () {58 vars in 957 bytes} [Mon May 23 13:10:20 2022] POST /identity/v3/auth/tokens => generated 3242 bytes in 126 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) May 23 13:10:20.270998 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f440e8da-5b24-434d-b5af-b65d30fc3179 admin admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:10:20.272965 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.fernet_utils [None req-f440e8da-5b24-434d-b5af-b65d30fc3179 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92056) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:10:20.357188 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f440e8da-5b24-434d-b5af-b65d30fc3179 admin admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:10:20.362202 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5b8cf05-666f-4dc2-b0e3-bf5fffafb0c9 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1666097f8ed545c6b0e4809c93e4775a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9c389ae65aaa4542bb5a5c4fd0bacf5b', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:10:20.362989 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5b8cf05-666f-4dc2-b0e3-bf5fffafb0c9 service cinder] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:10:20.363374 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5b8cf05-666f-4dc2-b0e3-bf5fffafb0c9 service cinder] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:10:20.363840 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5b8cf05-666f-4dc2-b0e3-bf5fffafb0c9 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:10:20.373558 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a5b8cf05-666f-4dc2-b0e3-bf5fffafb0c9 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:10:20.375359 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a5b8cf05-666f-4dc2-b0e3-bf5fffafb0c9 service cinder] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:10:20.385577 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 1299/2611] 10.222.0.30 () {60 vars in 1365 bytes} [Mon May 23 13:10:20 2022] GET /identity/v3/auth/tokens => generated 3242 bytes in 115 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) May 23 13:10:23.210607 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03cb0f95-2fd2-4181-b52d-2ded03231fa3 None None] REQUEST_METHOD: `POST` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:10:23.211142 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03cb0f95-2fd2-4181-b52d-2ded03231fa3 None None] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:10:23.211142 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03cb0f95-2fd2-4181-b52d-2ded03231fa3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:10:23.229408 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.auth.core [None req-03cb0f95-2fd2-4181-b52d-2ded03231fa3 None None] MFA Rules not processed for user `b19fc36cb5dc4aa9b684352cea0e6412`. Rule list: `[]` (Enabled: `True`). {{(pid=92057) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 13:10:23.230663 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.fernet_utils [None req-03cb0f95-2fd2-4181-b52d-2ded03231fa3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92057) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:10:23.238346 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 1313/2612] 10.222.0.30 () {58 vars in 957 bytes} [Mon May 23 13:10:23 2022] POST /identity/v3/auth/tokens => generated 3242 bytes in 29 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) May 23 13:10:30.380777 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=47,keystone:INSERT=2 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 13:10:30.885816 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a81955f-d489-4d77-a8ee-fd0e96631dfc None None] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:10:30.885816 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a81955f-d489-4d77-a8ee-fd0e96631dfc None None] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:10:30.885816 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a81955f-d489-4d77-a8ee-fd0e96631dfc None None] PATH_INFO: `/` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:10:30.886775 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 1300/2613] 10.222.0.30 () {54 vars in 921 bytes} [Mon May 23 13:10:30 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) May 23 13:10:30.893684 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4c569ba-ae60-47cb-905b-27ddfab3cb84 None None] REQUEST_METHOD: `POST` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:10:30.894240 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4c569ba-ae60-47cb-905b-27ddfab3cb84 None None] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:10:30.894327 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4c569ba-ae60-47cb-905b-27ddfab3cb84 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:10:30.913869 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.auth.core [None req-c4c569ba-ae60-47cb-905b-27ddfab3cb84 None None] MFA Rules not processed for user `cbe9931d94d44cc7ace0e83a7e5015fc`. Rule list: `[]` (Enabled: `True`). {{(pid=92057) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 13:10:30.915102 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.fernet_utils [None req-c4c569ba-ae60-47cb-905b-27ddfab3cb84 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92057) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:10:30.992563 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 1314/2614] 10.222.0.30 () {58 vars in 1018 bytes} [Mon May 23 13:10:30 2022] POST /identity/v3/auth/tokens => generated 3362 bytes in 99 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) May 23 13:10:31.004494 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a81955f-d489-4d77-a8ee-fd0e96631dfc None None] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:10:31.006718 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.fernet_utils [None req-9a81955f-d489-4d77-a8ee-fd0e96631dfc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92056) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:10:31.068101 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a81955f-d489-4d77-a8ee-fd0e96631dfc None None] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:10:31.071679 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff9000bb-528a-47e6-9312-43d1782d3b84 service glance] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '5cea0feed4254ec085854986f99acf30', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9c389ae65aaa4542bb5a5c4fd0bacf5b', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:10:31.072320 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff9000bb-528a-47e6-9312-43d1782d3b84 service glance] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:10:31.072605 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff9000bb-528a-47e6-9312-43d1782d3b84 service glance] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:10:31.072883 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff9000bb-528a-47e6-9312-43d1782d3b84 service glance] PATH_INFO: `/v3/auth/tokens` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:10:31.083997 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff9000bb-528a-47e6-9312-43d1782d3b84 service glance] RBAC: Authorizing `identity:validate_token()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:10:31.085424 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff9000bb-528a-47e6-9312-43d1782d3b84 service glance] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:10:31.096180 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 1301/2615] 10.222.0.30 () {60 vars in 1365 bytes} [Mon May 23 13:10:31 2022] GET /identity/v3/auth/tokens => generated 3362 bytes in 92 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) May 23 13:10:31.294068 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71457f41-2e8c-4046-9c85-5a341eb1697d None None] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:10:31.294429 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71457f41-2e8c-4046-9c85-5a341eb1697d None None] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:10:31.294739 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71457f41-2e8c-4046-9c85-5a341eb1697d None None] PATH_INFO: `/` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:10:31.296936 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 1315/2616] 10.222.0.30 () {54 vars in 921 bytes} [Mon May 23 13:10:31 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) May 23 13:10:31.302255 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f213ee0-368b-4c2f-9eed-724b878d86e3 None None] REQUEST_METHOD: `POST` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:10:31.302546 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f213ee0-368b-4c2f-9eed-724b878d86e3 None None] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:10:31.302847 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f213ee0-368b-4c2f-9eed-724b878d86e3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:10:31.339239 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.auth.core [None req-7f213ee0-368b-4c2f-9eed-724b878d86e3 None None] MFA Rules not processed for user `2eb83ca38e79449095f3a2f947bbb06a`. Rule list: `[]` (Enabled: `True`). {{(pid=92056) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 13:10:31.340421 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.fernet_utils [None req-7f213ee0-368b-4c2f-9eed-724b878d86e3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92056) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:10:31.391414 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 1302/2617] 10.222.0.30 () {58 vars in 1018 bytes} [Mon May 23 13:10:31 2022] POST /identity/v3/auth/tokens => generated 3111 bytes in 90 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) May 23 13:10:31.403032 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71457f41-2e8c-4046-9c85-5a341eb1697d None None] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:10:31.404668 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.fernet_utils [None req-71457f41-2e8c-4046-9c85-5a341eb1697d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92057) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:10:31.417914 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71457f41-2e8c-4046-9c85-5a341eb1697d None None] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:10:31.421889 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d39c36d-816e-4fa8-9ec7-ec155f34de71 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2eb83ca38e79449095f3a2f947bbb06a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9c389ae65aaa4542bb5a5c4fd0bacf5b', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:10:31.422530 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d39c36d-816e-4fa8-9ec7-ec155f34de71 service neutron] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:10:31.422810 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d39c36d-816e-4fa8-9ec7-ec155f34de71 service neutron] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:10:31.423089 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d39c36d-816e-4fa8-9ec7-ec155f34de71 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:10:31.429999 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d39c36d-816e-4fa8-9ec7-ec155f34de71 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:10:31.431415 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d39c36d-816e-4fa8-9ec7-ec155f34de71 service neutron] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:10:31.439730 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 1316/2618] 10.222.0.30 () {60 vars in 1365 bytes} [Mon May 23 13:10:31 2022] GET /identity/v3/auth/tokens => generated 3111 bytes in 37 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) May 23 13:10:31.521711 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7c4400f-0b21-4fcb-9c81-1ca4f4c7d195 None None] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:10:31.522035 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7c4400f-0b21-4fcb-9c81-1ca4f4c7d195 None None] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:10:31.522316 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7c4400f-0b21-4fcb-9c81-1ca4f4c7d195 None None] PATH_INFO: `/` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:10:31.523989 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 1303/2619] 10.222.0.30 () {54 vars in 921 bytes} [Mon May 23 13:10:31 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) May 23 13:10:31.529391 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-634cba8c-b921-493f-ae06-8d5f215c9700 None None] REQUEST_METHOD: `POST` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:10:31.529683 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-634cba8c-b921-493f-ae06-8d5f215c9700 None None] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:10:31.530123 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-634cba8c-b921-493f-ae06-8d5f215c9700 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:10:31.548673 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.auth.core [None req-634cba8c-b921-493f-ae06-8d5f215c9700 None None] MFA Rules not processed for user `cbe9931d94d44cc7ace0e83a7e5015fc`. Rule list: `[]` (Enabled: `True`). {{(pid=92057) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 13:10:31.549848 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.fernet_utils [None req-634cba8c-b921-493f-ae06-8d5f215c9700 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92057) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:10:31.558885 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 1317/2620] 10.222.0.30 () {58 vars in 1018 bytes} [Mon May 23 13:10:31 2022] POST /identity/v3/auth/tokens => generated 3362 bytes in 30 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) May 23 13:10:31.569688 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7c4400f-0b21-4fcb-9c81-1ca4f4c7d195 None None] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:10:31.580680 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7c4400f-0b21-4fcb-9c81-1ca4f4c7d195 None None] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:10:31.584240 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d26f4769-8103-4181-ba11-3142b3f65afe service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2eb83ca38e79449095f3a2f947bbb06a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9c389ae65aaa4542bb5a5c4fd0bacf5b', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:10:31.584899 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d26f4769-8103-4181-ba11-3142b3f65afe service neutron] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:10:31.585183 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d26f4769-8103-4181-ba11-3142b3f65afe service neutron] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:10:31.585464 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d26f4769-8103-4181-ba11-3142b3f65afe service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:10:31.596227 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d26f4769-8103-4181-ba11-3142b3f65afe service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:10:31.597654 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d26f4769-8103-4181-ba11-3142b3f65afe service neutron] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:10:31.609080 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 1304/2621] 10.222.0.30 () {60 vars in 1365 bytes} [Mon May 23 13:10:31 2022] GET /identity/v3/auth/tokens => generated 3362 bytes in 40 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) May 23 13:10:35.357519 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91028260-db46-4ed7-9b45-42a544755335 None None] REQUEST_METHOD: `POST` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:10:35.358794 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91028260-db46-4ed7-9b45-42a544755335 None None] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:10:35.358794 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91028260-db46-4ed7-9b45-42a544755335 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:10:35.377033 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.auth.core [None req-91028260-db46-4ed7-9b45-42a544755335 None None] MFA Rules not processed for user `6d21a56126fd4ccd837c33ccc08ab609`. Rule list: `[]` (Enabled: `True`). {{(pid=92057) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 13:10:35.378271 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.fernet_utils [None req-91028260-db46-4ed7-9b45-42a544755335 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92057) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:10:35.432802 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 1318/2622] 10.222.0.30 () {58 vars in 1009 bytes} [Mon May 23 13:10:35 2022] POST /identity/v3/auth/tokens => generated 3185 bytes in 76 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) May 23 13:10:35.444493 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d26f4769-8103-4181-ba11-3142b3f65afe service neutron] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:10:35.445950 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.fernet_utils [None req-d26f4769-8103-4181-ba11-3142b3f65afe service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92056) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:10:35.514645 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d26f4769-8103-4181-ba11-3142b3f65afe service neutron] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:10:35.519742 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f667ac47-c379-486b-99a4-9d48898dd1bb service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '138d91edb99f450e846765f9eef00d62', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9c389ae65aaa4542bb5a5c4fd0bacf5b', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'service', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:10:35.520381 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f667ac47-c379-486b-99a4-9d48898dd1bb service swift] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:10:35.520666 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f667ac47-c379-486b-99a4-9d48898dd1bb service swift] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:10:35.520945 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f667ac47-c379-486b-99a4-9d48898dd1bb service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:10:35.529170 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f667ac47-c379-486b-99a4-9d48898dd1bb service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:10:35.531463 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f667ac47-c379-486b-99a4-9d48898dd1bb service swift] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:10:35.539388 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 1305/2623] 10.222.0.30 () {60 vars in 1384 bytes} [Mon May 23 13:10:35 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 95 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) May 23 13:10:45.426331 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=85 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 13:10:45.536180 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=93 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 13:10:54.082379 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91028260-db46-4ed7-9b45-42a544755335 None None] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:10:54.083822 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.fernet_utils [None req-91028260-db46-4ed7-9b45-42a544755335 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92057) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:10:54.099652 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91028260-db46-4ed7-9b45-42a544755335 None None] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:10:54.105987 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de9b3cc0-6f3b-4e68-9b88-2af7d548afd4 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cbe9931d94d44cc7ace0e83a7e5015fc', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9c389ae65aaa4542bb5a5c4fd0bacf5b', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'member', 'reader', 'service', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:10:54.106538 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de9b3cc0-6f3b-4e68-9b88-2af7d548afd4 service nova] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:10:54.106821 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de9b3cc0-6f3b-4e68-9b88-2af7d548afd4 service nova] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:10:54.107096 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de9b3cc0-6f3b-4e68-9b88-2af7d548afd4 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:10:54.108917 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.fernet_utils [None req-de9b3cc0-6f3b-4e68-9b88-2af7d548afd4 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92057) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:10:54.122211 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de9b3cc0-6f3b-4e68-9b88-2af7d548afd4 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:10:54.123823 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de9b3cc0-6f3b-4e68-9b88-2af7d548afd4 service nova] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:10:54.135739 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 1319/2624] 10.222.0.30 () {60 vars in 1365 bytes} [Mon May 23 13:10:54 2022] GET /identity/v3/auth/tokens => generated 3362 bytes in 54 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) May 23 13:10:55.632970 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f667ac47-c379-486b-99a4-9d48898dd1bb service swift] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:10:55.634553 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.fernet_utils [None req-f667ac47-c379-486b-99a4-9d48898dd1bb service swift] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92056) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:10:55.651377 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f667ac47-c379-486b-99a4-9d48898dd1bb service swift] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:10:55.656903 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d12a0300-cb2f-4536-b976-ddece84150a0 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cbe9931d94d44cc7ace0e83a7e5015fc', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9c389ae65aaa4542bb5a5c4fd0bacf5b', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'member', 'reader', 'service', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:10:55.657633 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d12a0300-cb2f-4536-b976-ddece84150a0 service nova] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:10:55.657971 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d12a0300-cb2f-4536-b976-ddece84150a0 service nova] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:10:55.658251 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d12a0300-cb2f-4536-b976-ddece84150a0 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:10:55.660190 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.fernet_utils [None req-d12a0300-cb2f-4536-b976-ddece84150a0 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92056) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:10:55.674176 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d12a0300-cb2f-4536-b976-ddece84150a0 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:10:55.676116 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d12a0300-cb2f-4536-b976-ddece84150a0 service nova] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:10:55.687992 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 1306/2625] 10.222.0.30 () {60 vars in 1365 bytes} [Mon May 23 13:10:55 2022] GET /identity/v3/auth/tokens => generated 3362 bytes in 56 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) May 23 13:10:57.074118 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2d287f0-c23e-4a5a-a3e3-6697c5ebfe33 None None] REQUEST_METHOD: `POST` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:10:57.074118 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2d287f0-c23e-4a5a-a3e3-6697c5ebfe33 None None] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:10:57.074118 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2d287f0-c23e-4a5a-a3e3-6697c5ebfe33 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:10:57.083469 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.auth.core [None req-d2d287f0-c23e-4a5a-a3e3-6697c5ebfe33 None None] MFA Rules not processed for user `6d21a56126fd4ccd837c33ccc08ab609`. Rule list: `[]` (Enabled: `True`). {{(pid=92057) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 13:10:57.084665 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.fernet_utils [None req-d2d287f0-c23e-4a5a-a3e3-6697c5ebfe33 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92057) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:10:57.091967 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 1320/2626] 10.222.0.30 () {58 vars in 967 bytes} [Mon May 23 13:10:57 2022] POST /identity/v3/auth/tokens => generated 3185 bytes in 28 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) May 23 13:10:57.104922 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d12a0300-cb2f-4536-b976-ddece84150a0 service nova] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:10:57.117565 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d12a0300-cb2f-4536-b976-ddece84150a0 service nova] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:10:57.124177 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1348d5de-32e4-44d7-b0f1-b91b60753d4b service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '138d91edb99f450e846765f9eef00d62', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9c389ae65aaa4542bb5a5c4fd0bacf5b', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'service', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:10:57.124177 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1348d5de-32e4-44d7-b0f1-b91b60753d4b service swift] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:10:57.124177 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1348d5de-32e4-44d7-b0f1-b91b60753d4b service swift] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:10:57.124177 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1348d5de-32e4-44d7-b0f1-b91b60753d4b service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:10:57.132454 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1348d5de-32e4-44d7-b0f1-b91b60753d4b service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:10:57.134073 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1348d5de-32e4-44d7-b0f1-b91b60753d4b service swift] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:10:57.142462 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 1307/2627] 10.222.0.30 () {60 vars in 1384 bytes} [Mon May 23 13:10:57 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 38 msecs (HTTP/1.1 200) 6 headers in 379 bytes (2 switches on core 0) May 23 13:10:58.917080 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2dc96649-69f0-4c2c-85b2-91e1959687f6 None None] REQUEST_METHOD: `POST` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:10:58.917543 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2dc96649-69f0-4c2c-85b2-91e1959687f6 None None] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:10:58.917648 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2dc96649-69f0-4c2c-85b2-91e1959687f6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:10:58.935233 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.auth.core [None req-2dc96649-69f0-4c2c-85b2-91e1959687f6 None None] MFA Rules not processed for user `6d21a56126fd4ccd837c33ccc08ab609`. Rule list: `[]` (Enabled: `True`). {{(pid=92057) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 13:10:58.936409 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.fernet_utils [None req-2dc96649-69f0-4c2c-85b2-91e1959687f6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92057) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:10:58.944062 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 1321/2628] 10.222.0.30 () {58 vars in 1009 bytes} [Mon May 23 13:10:58 2022] POST /identity/v3/auth/tokens => generated 3185 bytes in 28 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) May 23 13:10:58.956987 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1348d5de-32e4-44d7-b0f1-b91b60753d4b service swift] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:10:58.976348 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1348d5de-32e4-44d7-b0f1-b91b60753d4b service swift] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:10:58.983419 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b372387c-5c04-463b-a11d-078a29640551 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '138d91edb99f450e846765f9eef00d62', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9c389ae65aaa4542bb5a5c4fd0bacf5b', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'service', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:10:58.984056 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b372387c-5c04-463b-a11d-078a29640551 service swift] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:10:58.984336 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b372387c-5c04-463b-a11d-078a29640551 service swift] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:10:58.984611 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b372387c-5c04-463b-a11d-078a29640551 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:10:58.992888 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b372387c-5c04-463b-a11d-078a29640551 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:10:58.994517 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b372387c-5c04-463b-a11d-078a29640551 service swift] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:10:59.002502 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 1308/2629] 10.222.0.30 () {60 vars in 1384 bytes} [Mon May 23 13:10:58 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 46 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) May 23 13:10:59.878919 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3dd4b67c-6cc5-4290-a50c-dd3d31bcc2f4 None None] REQUEST_METHOD: `POST` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:10:59.879248 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3dd4b67c-6cc5-4290-a50c-dd3d31bcc2f4 None None] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:10:59.879626 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3dd4b67c-6cc5-4290-a50c-dd3d31bcc2f4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:10:59.898184 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.auth.core [None req-3dd4b67c-6cc5-4290-a50c-dd3d31bcc2f4 None None] MFA Rules not processed for user `6d21a56126fd4ccd837c33ccc08ab609`. Rule list: `[]` (Enabled: `True`). {{(pid=92057) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 13:10:59.899357 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.fernet_utils [None req-3dd4b67c-6cc5-4290-a50c-dd3d31bcc2f4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92057) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:10:59.906410 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 1322/2630] 10.222.0.30 () {58 vars in 967 bytes} [Mon May 23 13:10:59 2022] POST /identity/v3/auth/tokens => generated 3185 bytes in 29 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) May 23 13:10:59.919135 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b372387c-5c04-463b-a11d-078a29640551 service swift] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:10:59.930290 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b372387c-5c04-463b-a11d-078a29640551 service swift] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:10:59.935365 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dae5bac6-3107-4295-b262-f6309a03cb33 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '138d91edb99f450e846765f9eef00d62', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9c389ae65aaa4542bb5a5c4fd0bacf5b', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'service', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:10:59.935998 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dae5bac6-3107-4295-b262-f6309a03cb33 service swift] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:10:59.936280 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dae5bac6-3107-4295-b262-f6309a03cb33 service swift] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:10:59.936555 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dae5bac6-3107-4295-b262-f6309a03cb33 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:10:59.945250 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dae5bac6-3107-4295-b262-f6309a03cb33 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:10:59.946853 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dae5bac6-3107-4295-b262-f6309a03cb33 service swift] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:10:59.955890 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 1309/2631] 10.222.0.30 () {60 vars in 1384 bytes} [Mon May 23 13:10:59 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 37 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) May 23 13:11:00.509700 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41fbe940-6844-4faa-bad2-534d906f116c None None] REQUEST_METHOD: `POST` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:11:00.510214 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41fbe940-6844-4faa-bad2-534d906f116c None None] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:11:00.510330 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41fbe940-6844-4faa-bad2-534d906f116c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:11:00.529751 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dae5bac6-3107-4295-b262-f6309a03cb33 service swift] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:11:00.531162 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.fernet_utils [None req-dae5bac6-3107-4295-b262-f6309a03cb33 service swift] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92056) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:11:00.535557 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.auth.core [None req-41fbe940-6844-4faa-bad2-534d906f116c None None] MFA Rules not processed for user `6d21a56126fd4ccd837c33ccc08ab609`. Rule list: `[]` (Enabled: `True`). {{(pid=92057) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 13:11:00.536722 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.fernet_utils [None req-41fbe940-6844-4faa-bad2-534d906f116c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92057) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:11:00.543880 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dae5bac6-3107-4295-b262-f6309a03cb33 service swift] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:11:00.544037 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 1323/2632] 10.222.0.30 () {58 vars in 1009 bytes} [Mon May 23 13:11:00 2022] POST /identity/v3/auth/tokens => generated 3185 bytes in 35 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) May 23 13:11:00.547449 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f317655-ef05-4edb-b67d-9355dedc50d6 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2eb83ca38e79449095f3a2f947bbb06a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9c389ae65aaa4542bb5a5c4fd0bacf5b', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:11:00.548124 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f317655-ef05-4edb-b67d-9355dedc50d6 service neutron] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:11:00.548373 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f317655-ef05-4edb-b67d-9355dedc50d6 service neutron] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:11:00.548655 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f317655-ef05-4edb-b67d-9355dedc50d6 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:11:00.550505 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.fernet_utils [None req-5f317655-ef05-4edb-b67d-9355dedc50d6 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92056) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:11:00.555913 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41fbe940-6844-4faa-bad2-534d906f116c None None] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:11:00.560427 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f317655-ef05-4edb-b67d-9355dedc50d6 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:11:00.561930 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f317655-ef05-4edb-b67d-9355dedc50d6 service neutron] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:11:00.567926 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41fbe940-6844-4faa-bad2-534d906f116c None None] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:11:00.570261 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 1310/2633] 10.222.0.30 () {60 vars in 1365 bytes} [Mon May 23 13:11:00 2022] GET /identity/v3/auth/tokens => generated 3111 bytes in 41 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) May 23 13:11:00.573066 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf2c8399-0e08-4a52-93f6-1e414b9c358f service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '138d91edb99f450e846765f9eef00d62', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9c389ae65aaa4542bb5a5c4fd0bacf5b', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'service', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:11:00.573702 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf2c8399-0e08-4a52-93f6-1e414b9c358f service swift] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:11:00.574044 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf2c8399-0e08-4a52-93f6-1e414b9c358f service swift] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:11:00.574316 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf2c8399-0e08-4a52-93f6-1e414b9c358f service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:11:00.583400 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf2c8399-0e08-4a52-93f6-1e414b9c358f service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:11:00.584994 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf2c8399-0e08-4a52-93f6-1e414b9c358f service swift] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:11:00.593348 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 1324/2634] 10.222.0.30 () {60 vars in 1384 bytes} [Mon May 23 13:11:00 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 38 msecs (HTTP/1.1 200) 6 headers in 379 bytes (2 switches on core 0) May 23 13:11:01.139964 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fff51bbf-d571-482b-a3f8-1c8a70a9633c None None] REQUEST_METHOD: `POST` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:11:01.139964 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fff51bbf-d571-482b-a3f8-1c8a70a9633c None None] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:11:01.139964 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fff51bbf-d571-482b-a3f8-1c8a70a9633c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:11:01.156098 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.auth.core [None req-fff51bbf-d571-482b-a3f8-1c8a70a9633c None None] MFA Rules not processed for user `6d21a56126fd4ccd837c33ccc08ab609`. Rule list: `[]` (Enabled: `True`). {{(pid=92056) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 13:11:01.157297 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.fernet_utils [None req-fff51bbf-d571-482b-a3f8-1c8a70a9633c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92056) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:11:01.166443 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 1311/2635] 10.222.0.30 () {58 vars in 967 bytes} [Mon May 23 13:11:01 2022] POST /identity/v3/auth/tokens => generated 3185 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) May 23 13:11:01.179904 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf2c8399-0e08-4a52-93f6-1e414b9c358f service swift] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:11:01.191196 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf2c8399-0e08-4a52-93f6-1e414b9c358f service swift] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:11:01.196267 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ac3d1b7-f540-418d-8403-e3a82ebdab80 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '138d91edb99f450e846765f9eef00d62', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9c389ae65aaa4542bb5a5c4fd0bacf5b', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'service', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:11:01.196911 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ac3d1b7-f540-418d-8403-e3a82ebdab80 service swift] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:11:01.197197 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ac3d1b7-f540-418d-8403-e3a82ebdab80 service swift] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:11:01.197480 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ac3d1b7-f540-418d-8403-e3a82ebdab80 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:11:01.206309 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ac3d1b7-f540-418d-8403-e3a82ebdab80 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:11:01.207901 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ac3d1b7-f540-418d-8403-e3a82ebdab80 service swift] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:11:01.217045 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 1325/2636] 10.222.0.30 () {60 vars in 1384 bytes} [Mon May 23 13:11:01 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 38 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) May 23 13:11:03.841248 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fff51bbf-d571-482b-a3f8-1c8a70a9633c None None] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:11:03.903418 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fff51bbf-d571-482b-a3f8-1c8a70a9633c None None] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:11:03.908003 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aac086b4-9a62-4d5e-8253-0e5f58798d4b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:11:03.908860 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aac086b4-9a62-4d5e-8253-0e5f58798d4b admin admin] REQUEST_METHOD: `DELETE` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:11:03.909141 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aac086b4-9a62-4d5e-8253-0e5f58798d4b admin admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:11:03.909417 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aac086b4-9a62-4d5e-8253-0e5f58798d4b admin admin] PATH_INFO: `/v3/users/b19fc36cb5dc4aa9b684352cea0e6412` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:11:03.910834 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aac086b4-9a62-4d5e-8253-0e5f58798d4b admin admin] RBAC: Authorizing `identity:delete_user(user_id=b19fc36cb5dc4aa9b684352cea0e6412)` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:11:03.912747 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aac086b4-9a62-4d5e-8253-0e5f58798d4b admin admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:11:04.072969 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-aac086b4-9a62-4d5e-8253-0e5f58798d4b admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'b19fc36cb5dc4aa9b684352cea0e6412'} {{(pid=92056) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:11:04.073582 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-aac086b4-9a62-4d5e-8253-0e5f58798d4b admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'b19fc36cb5dc4aa9b684352cea0e6412'} {{(pid=92056) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:11:04.121883 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-aac086b4-9a62-4d5e-8253-0e5f58798d4b admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'b19fc36cb5dc4aa9b684352cea0e6412'} {{(pid=92056) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:11:04.135190 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-aac086b4-9a62-4d5e-8253-0e5f58798d4b admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'b19fc36cb5dc4aa9b684352cea0e6412'} {{(pid=92056) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:11:04.142083 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-aac086b4-9a62-4d5e-8253-0e5f58798d4b admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'b19fc36cb5dc4aa9b684352cea0e6412'} {{(pid=92056) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:11:04.148857 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 1312/2637] 10.222.0.30 () {58 vars in 1223 bytes} [Mon May 23 13:11:03 2022] DELETE /identity/v3/users/b19fc36cb5dc4aa9b684352cea0e6412 => generated 0 bytes in 308 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) May 23 13:11:04.153511 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49bd199d-13bc-4b18-a33c-b2da497488e7 None None] REQUEST_METHOD: `POST` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:11:04.154045 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49bd199d-13bc-4b18-a33c-b2da497488e7 None None] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:11:04.154153 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49bd199d-13bc-4b18-a33c-b2da497488e7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:11:04.171398 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.auth.core [None req-49bd199d-13bc-4b18-a33c-b2da497488e7 None None] MFA Rules not processed for user `0f7119f8fa5c4445977671140ded91a4`. Rule list: `[]` (Enabled: `True`). {{(pid=92057) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 13:11:04.172583 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.fernet_utils [None req-49bd199d-13bc-4b18-a33c-b2da497488e7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92057) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:11:04.230407 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 1326/2638] 10.222.0.30 () {58 vars in 957 bytes} [Mon May 23 13:11:04 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 78 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) May 23 13:11:04.240568 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aac086b4-9a62-4d5e-8253-0e5f58798d4b admin admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:11:04.242005 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.fernet_utils [None req-aac086b4-9a62-4d5e-8253-0e5f58798d4b admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92056) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:11:04.252482 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 13:11:04.301402 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aac086b4-9a62-4d5e-8253-0e5f58798d4b admin admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:11:04.305626 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95a583bf-06ed-4d0e-a0c8-9fb2a1002687 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2eb83ca38e79449095f3a2f947bbb06a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9c389ae65aaa4542bb5a5c4fd0bacf5b', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:11:04.306333 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95a583bf-06ed-4d0e-a0c8-9fb2a1002687 service neutron] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:11:04.306610 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95a583bf-06ed-4d0e-a0c8-9fb2a1002687 service neutron] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:11:04.306884 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95a583bf-06ed-4d0e-a0c8-9fb2a1002687 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:11:04.317452 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95a583bf-06ed-4d0e-a0c8-9fb2a1002687 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:11:04.318952 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95a583bf-06ed-4d0e-a0c8-9fb2a1002687 service neutron] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:11:04.329061 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 1313/2639] 10.222.0.30 () {60 vars in 1365 bytes} [Mon May 23 13:11:04 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 89 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) May 23 13:11:04.552677 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49bd199d-13bc-4b18-a33c-b2da497488e7 None None] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:11:04.554219 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.fernet_utils [None req-49bd199d-13bc-4b18-a33c-b2da497488e7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92057) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:11:04.567032 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49bd199d-13bc-4b18-a33c-b2da497488e7 None None] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:11:04.571681 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37d93111-548e-4737-bac7-bbb4daf4a256 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:11:04.573120 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37d93111-548e-4737-bac7-bbb4daf4a256 admin admin] REQUEST_METHOD: `DELETE` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:11:04.573444 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37d93111-548e-4737-bac7-bbb4daf4a256 admin admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:11:04.573909 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37d93111-548e-4737-bac7-bbb4daf4a256 admin admin] PATH_INFO: `/v3/projects/0ffdc054740c46f990ff4428f11cf07c` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:11:04.575132 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37d93111-548e-4737-bac7-bbb4daf4a256 admin admin] RBAC: Authorizing `identity:delete_project(project_id=0ffdc054740c46f990ff4428f11cf07c)` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:11:04.576996 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37d93111-548e-4737-bac7-bbb4daf4a256 admin admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:11:04.717322 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [None req-37d93111-548e-4737-bac7-bbb4daf4a256 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '0ffdc054740c46f990ff4428f11cf07c', 'request_id': 'req-37d93111-548e-4737-bac7-bbb4daf4a256'} {{(pid=92057) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:11:04.723506 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [None req-37d93111-548e-4737-bac7-bbb4daf4a256 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '0ffdc054740c46f990ff4428f11cf07c', 'request_id': 'req-37d93111-548e-4737-bac7-bbb4daf4a256'} {{(pid=92057) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:11:04.819065 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [None req-37d93111-548e-4737-bac7-bbb4daf4a256 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '0ffdc054740c46f990ff4428f11cf07c', 'request_id': 'req-37d93111-548e-4737-bac7-bbb4daf4a256'} {{(pid=92057) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:11:04.831188 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [None req-37d93111-548e-4737-bac7-bbb4daf4a256 admin admin] The token cache is being invalidate because project 0ffdc054740c46f990ff4428f11cf07c was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=92057) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} May 23 13:11:04.831412 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [None req-37d93111-548e-4737-bac7-bbb4daf4a256 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=92057) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:11:04.832077 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [None req-37d93111-548e-4737-bac7-bbb4daf4a256 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=92057) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:11:04.833636 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 1327/2640] 10.222.0.30 () {58 vars in 1232 bytes} [Mon May 23 13:11:04 2022] DELETE /identity/v3/projects/0ffdc054740c46f990ff4428f11cf07c => generated 0 bytes in 282 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) May 23 13:11:04.838887 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c68b31a6-c1ea-44e5-b00b-25e9b9f9b780 None None] REQUEST_METHOD: `POST` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:11:04.839178 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c68b31a6-c1ea-44e5-b00b-25e9b9f9b780 None None] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:11:04.839460 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c68b31a6-c1ea-44e5-b00b-25e9b9f9b780 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:11:04.858234 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.auth.core [None req-c68b31a6-c1ea-44e5-b00b-25e9b9f9b780 None None] MFA Rules not processed for user `0f7119f8fa5c4445977671140ded91a4`. Rule list: `[]` (Enabled: `True`). {{(pid=92056) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 13:11:04.859417 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.fernet_utils [None req-c68b31a6-c1ea-44e5-b00b-25e9b9f9b780 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92056) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:11:04.919639 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 1314/2641] 10.222.0.30 () {58 vars in 957 bytes} [Mon May 23 13:11:04 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 82 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) May 23 13:11:04.927899 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7eedda6-94ec-466d-ae4a-46220bb2c1b7 None None] REQUEST_METHOD: `POST` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:11:04.928161 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7eedda6-94ec-466d-ae4a-46220bb2c1b7 None None] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:11:04.928446 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7eedda6-94ec-466d-ae4a-46220bb2c1b7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:11:04.948335 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.auth.core [None req-e7eedda6-94ec-466d-ae4a-46220bb2c1b7 None None] MFA Rules not processed for user `0f7119f8fa5c4445977671140ded91a4`. Rule list: `[]` (Enabled: `True`). {{(pid=92057) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 13:11:04.949533 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.fernet_utils [None req-e7eedda6-94ec-466d-ae4a-46220bb2c1b7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92057) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:11:04.958147 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 1328/2642] 10.222.0.30 () {58 vars in 957 bytes} [Mon May 23 13:11:04 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) May 23 13:11:04.962716 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c68b31a6-c1ea-44e5-b00b-25e9b9f9b780 None None] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:11:04.972929 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c68b31a6-c1ea-44e5-b00b-25e9b9f9b780 None None] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:11:04.977541 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c16c722-544d-458d-ae14-87097e2b80cf admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:11:04.978223 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c16c722-544d-458d-ae14-87097e2b80cf admin admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:11:04.978502 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c16c722-544d-458d-ae14-87097e2b80cf admin admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:11:04.978829 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c16c722-544d-458d-ae14-87097e2b80cf admin admin] PATH_INFO: `/v3/domains` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:11:04.979494 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c16c722-544d-458d-ae14-87097e2b80cf admin admin] RBAC: Adding query filter params (name=Default) {{(pid=92056) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} May 23 13:11:04.979799 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c16c722-544d-458d-ae14-87097e2b80cf admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:11:04.981395 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c16c722-544d-458d-ae14-87097e2b80cf admin admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:11:04.994039 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 1315/2643] 10.222.0.30 () {58 vars in 1152 bytes} [Mon May 23 13:11:04 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) May 23 13:11:04.998225 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7eedda6-94ec-466d-ae4a-46220bb2c1b7 None None] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:11:05.008405 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7eedda6-94ec-466d-ae4a-46220bb2c1b7 None None] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:11:05.013134 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac48f4bf-6372-4b57-8ed7-515f1d8acc01 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:11:05.013849 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac48f4bf-6372-4b57-8ed7-515f1d8acc01 admin admin] REQUEST_METHOD: `POST` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:11:05.014231 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac48f4bf-6372-4b57-8ed7-515f1d8acc01 admin admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:11:05.014566 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac48f4bf-6372-4b57-8ed7-515f1d8acc01 admin admin] PATH_INFO: `/v3/projects` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:11:05.015323 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac48f4bf-6372-4b57-8ed7-515f1d8acc01 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:11:05.017349 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac48f4bf-6372-4b57-8ed7-515f1d8acc01 admin admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:11:05.025904 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 13:11:05.140736 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 1329/2644] 10.222.0.30 () {60 vars in 1152 bytes} [Mon May 23 13:11:04 2022] POST /identity/v3/projects => generated 383 bytes in 143 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) May 23 13:11:05.145151 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c16c722-544d-458d-ae14-87097e2b80cf admin admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:11:05.155278 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c16c722-544d-458d-ae14-87097e2b80cf admin admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:11:05.221357 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0a0fc0b-f939-48df-8428-146d09360684 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:11:05.222055 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0a0fc0b-f939-48df-8428-146d09360684 admin admin] REQUEST_METHOD: `POST` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:11:05.222342 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0a0fc0b-f939-48df-8428-146d09360684 admin admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:11:05.222620 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0a0fc0b-f939-48df-8428-146d09360684 admin admin] PATH_INFO: `/v3/users` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:11:05.223323 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0a0fc0b-f939-48df-8428-146d09360684 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:11:05.225320 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0a0fc0b-f939-48df-8428-146d09360684 admin admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:11:05.310987 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 1316/2645] 10.222.0.30 () {60 vars in 1143 bytes} [Mon May 23 13:11:05 2022] POST /identity/v3/users => generated 292 bytes in 166 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) May 23 13:11:05.315878 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac48f4bf-6372-4b57-8ed7-515f1d8acc01 admin admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:11:05.326165 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac48f4bf-6372-4b57-8ed7-515f1d8acc01 admin admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:11:05.331609 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c733633-d9b2-4243-a15a-d23de6cb5c89 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:11:05.332117 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c733633-d9b2-4243-a15a-d23de6cb5c89 admin admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:11:05.332403 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c733633-d9b2-4243-a15a-d23de6cb5c89 admin admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:11:05.332677 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c733633-d9b2-4243-a15a-d23de6cb5c89 admin admin] PATH_INFO: `/v3/roles` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:11:05.333389 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c733633-d9b2-4243-a15a-d23de6cb5c89 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:11:05.335039 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c733633-d9b2-4243-a15a-d23de6cb5c89 admin admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:11:05.345119 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 1330/2646] 10.222.0.30 () {58 vars in 1121 bytes} [Mon May 23 13:11:05 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) May 23 13:11:05.349294 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0a0fc0b-f939-48df-8428-146d09360684 admin admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:11:05.359974 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0a0fc0b-f939-48df-8428-146d09360684 admin admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:11:05.365226 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5002084c-b0a2-4748-b7d9-7843b1e5ae2d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:11:05.365899 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5002084c-b0a2-4748-b7d9-7843b1e5ae2d admin admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:11:05.366218 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5002084c-b0a2-4748-b7d9-7843b1e5ae2d admin admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:11:05.366475 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5002084c-b0a2-4748-b7d9-7843b1e5ae2d admin admin] PATH_INFO: `/v3/roles` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:11:05.367204 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5002084c-b0a2-4748-b7d9-7843b1e5ae2d admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:11:05.368822 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5002084c-b0a2-4748-b7d9-7843b1e5ae2d admin admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:11:05.385556 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 1317/2647] 10.222.0.30 () {58 vars in 1121 bytes} [Mon May 23 13:11:05 2022] GET /identity/v3/roles => generated 1420 bytes in 37 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) May 23 13:11:05.389931 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c733633-d9b2-4243-a15a-d23de6cb5c89 admin admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:11:05.401055 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c733633-d9b2-4243-a15a-d23de6cb5c89 admin admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:11:05.405686 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-112e33e9-6b24-4bcf-8c87-0580fb0a58e7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:11:05.406603 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-112e33e9-6b24-4bcf-8c87-0580fb0a58e7 admin admin] REQUEST_METHOD: `PUT` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:11:05.406892 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-112e33e9-6b24-4bcf-8c87-0580fb0a58e7 admin admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:11:05.407168 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-112e33e9-6b24-4bcf-8c87-0580fb0a58e7 admin admin] PATH_INFO: `/v3/projects/1f2fed134f274a6ebe5ffe183205e269/users/20d9973f32f847a6a54a60405beabdbf/roles/80fb22507c9a4ff09d44d8f12f6fff1a` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:11:05.422529 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-112e33e9-6b24-4bcf-8c87-0580fb0a58e7 admin admin] RBAC: Authorizing `identity:create_grant(project_id=1f2fed134f274a6ebe5ffe183205e269, user_id=20d9973f32f847a6a54a60405beabdbf, role_id=80fb22507c9a4ff09d44d8f12f6fff1a)` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:11:05.425451 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-112e33e9-6b24-4bcf-8c87-0580fb0a58e7 admin admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:11:05.522136 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 1331/2648] 10.222.0.30 () {60 vars in 1482 bytes} [Mon May 23 13:11:05 2022] PUT /identity/v3/projects/1f2fed134f274a6ebe5ffe183205e269/users/20d9973f32f847a6a54a60405beabdbf/roles/80fb22507c9a4ff09d44d8f12f6fff1a => generated 0 bytes in 133 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) May 23 13:11:05.527457 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9eb72195-7a45-4f75-8870-69c42361c59c None None] REQUEST_METHOD: `POST` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:11:05.527780 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9eb72195-7a45-4f75-8870-69c42361c59c None None] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:11:05.528110 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9eb72195-7a45-4f75-8870-69c42361c59c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:11:05.546095 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.auth.core [None req-9eb72195-7a45-4f75-8870-69c42361c59c None None] MFA Rules not processed for user `0f7119f8fa5c4445977671140ded91a4`. Rule list: `[]` (Enabled: `True`). {{(pid=92056) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 13:11:05.547290 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.fernet_utils [None req-9eb72195-7a45-4f75-8870-69c42361c59c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92056) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:11:05.556943 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 13:11:05.607182 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 1318/2649] 10.222.0.30 () {58 vars in 957 bytes} [Mon May 23 13:11:05 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 81 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) May 23 13:11:05.617631 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-112e33e9-6b24-4bcf-8c87-0580fb0a58e7 admin admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:11:05.619331 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.fernet_utils [None req-112e33e9-6b24-4bcf-8c87-0580fb0a58e7 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92057) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:11:05.676937 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-112e33e9-6b24-4bcf-8c87-0580fb0a58e7 admin admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:11:05.680636 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63900d48-8b1b-438e-a8e1-23f7124a2c46 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2eb83ca38e79449095f3a2f947bbb06a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9c389ae65aaa4542bb5a5c4fd0bacf5b', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:11:05.681292 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63900d48-8b1b-438e-a8e1-23f7124a2c46 service neutron] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:11:05.681577 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63900d48-8b1b-438e-a8e1-23f7124a2c46 service neutron] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:11:05.681853 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63900d48-8b1b-438e-a8e1-23f7124a2c46 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:11:05.697872 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63900d48-8b1b-438e-a8e1-23f7124a2c46 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:11:05.699367 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63900d48-8b1b-438e-a8e1-23f7124a2c46 service neutron] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:11:05.709760 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 1332/2650] 10.222.0.30 () {60 vars in 1365 bytes} [Mon May 23 13:11:05 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 93 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) May 23 13:11:10.903539 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dee03c29-14f7-4a75-84a4-885635cc3478 None None] REQUEST_METHOD: `POST` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:11:10.904025 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dee03c29-14f7-4a75-84a4-885635cc3478 None None] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:11:10.904132 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dee03c29-14f7-4a75-84a4-885635cc3478 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:11:10.933756 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.auth.core [None req-dee03c29-14f7-4a75-84a4-885635cc3478 None None] MFA Rules not processed for user `20d9973f32f847a6a54a60405beabdbf`. Rule list: `[]` (Enabled: `True`). {{(pid=92056) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 13:11:10.935022 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.fernet_utils [None req-dee03c29-14f7-4a75-84a4-885635cc3478 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92056) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:11:11.011164 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 1319/2651] 10.222.0.30 () {58 vars in 957 bytes} [Mon May 23 13:11:10 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 109 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) May 23 13:11:11.015318 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63900d48-8b1b-438e-a8e1-23f7124a2c46 service neutron] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:11:11.026829 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63900d48-8b1b-438e-a8e1-23f7124a2c46 service neutron] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:11:11.031562 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8b5354d-3c6f-430b-a185-937850990661 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:11:11.032407 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8b5354d-3c6f-430b-a185-937850990661 admin admin] REQUEST_METHOD: `POST` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:11:11.032825 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8b5354d-3c6f-430b-a185-937850990661 admin admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:11:11.033117 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8b5354d-3c6f-430b-a185-937850990661 admin admin] PATH_INFO: `/v3/projects` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:11:11.033985 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8b5354d-3c6f-430b-a185-937850990661 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:11:11.035908 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8b5354d-3c6f-430b-a185-937850990661 admin admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:11:11.134281 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 1333/2652] 10.222.0.30 () {60 vars in 1152 bytes} [Mon May 23 13:11:11 2022] POST /identity/v3/projects => generated 383 bytes in 119 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) May 23 13:11:11.138468 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dee03c29-14f7-4a75-84a4-885635cc3478 None None] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:11:11.149865 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dee03c29-14f7-4a75-84a4-885635cc3478 None None] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:11:11.221910 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f0acc8f-f090-43b2-b02e-ceac4e50a695 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:11:11.230410 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f0acc8f-f090-43b2-b02e-ceac4e50a695 admin admin] REQUEST_METHOD: `POST` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:11:11.230410 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f0acc8f-f090-43b2-b02e-ceac4e50a695 admin admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:11:11.230410 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f0acc8f-f090-43b2-b02e-ceac4e50a695 admin admin] PATH_INFO: `/v3/users` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:11:11.230410 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f0acc8f-f090-43b2-b02e-ceac4e50a695 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:11:11.230410 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f0acc8f-f090-43b2-b02e-ceac4e50a695 admin admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:11:11.344121 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 1320/2653] 10.222.0.30 () {60 vars in 1143 bytes} [Mon May 23 13:11:11 2022] POST /identity/v3/users => generated 298 bytes in 206 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) May 23 13:11:11.348293 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8b5354d-3c6f-430b-a185-937850990661 admin admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:11:11.358790 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8b5354d-3c6f-430b-a185-937850990661 admin admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:11:11.363791 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1788b52-dc1a-49c1-a067-b7ad49ebc449 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:11:11.364445 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1788b52-dc1a-49c1-a067-b7ad49ebc449 admin admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:11:11.364735 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1788b52-dc1a-49c1-a067-b7ad49ebc449 admin admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:11:11.365017 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1788b52-dc1a-49c1-a067-b7ad49ebc449 admin admin] PATH_INFO: `/v3/roles` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:11:11.365736 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1788b52-dc1a-49c1-a067-b7ad49ebc449 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:11:11.367455 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1788b52-dc1a-49c1-a067-b7ad49ebc449 admin admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:11:11.379045 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 1334/2654] 10.222.0.30 () {58 vars in 1121 bytes} [Mon May 23 13:11:11 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) May 23 13:11:11.383215 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f0acc8f-f090-43b2-b02e-ceac4e50a695 admin admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:11:11.393815 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f0acc8f-f090-43b2-b02e-ceac4e50a695 admin admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:11:11.398816 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-277828e2-dadc-460f-a2d9-40294584a358 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:11:11.399707 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-277828e2-dadc-460f-a2d9-40294584a358 admin admin] REQUEST_METHOD: `PUT` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:11:11.400018 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-277828e2-dadc-460f-a2d9-40294584a358 admin admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:11:11.400295 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-277828e2-dadc-460f-a2d9-40294584a358 admin admin] PATH_INFO: `/v3/projects/5ec3d4c508314b668b685921169854d6/users/03422526f5b843d49823f4f2316f7f56/roles/bc3c3c9d654940cbaa5177a6be23d1bc` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:11:11.417222 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-277828e2-dadc-460f-a2d9-40294584a358 admin admin] RBAC: Authorizing `identity:create_grant(project_id=5ec3d4c508314b668b685921169854d6, user_id=03422526f5b843d49823f4f2316f7f56, role_id=bc3c3c9d654940cbaa5177a6be23d1bc)` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:11:11.419968 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-277828e2-dadc-460f-a2d9-40294584a358 admin admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:11:11.528955 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 1321/2655] 10.222.0.30 () {60 vars in 1482 bytes} [Mon May 23 13:11:11 2022] PUT /identity/v3/projects/5ec3d4c508314b668b685921169854d6/users/03422526f5b843d49823f4f2316f7f56/roles/bc3c3c9d654940cbaa5177a6be23d1bc => generated 0 bytes in 145 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) May 23 13:11:11.531318 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1788b52-dc1a-49c1-a067-b7ad49ebc449 admin admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:11:11.541629 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1788b52-dc1a-49c1-a067-b7ad49ebc449 admin admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:11:11.602758 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74daba22-5387-4820-98f7-40972b2f43e6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:11:11.603557 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74daba22-5387-4820-98f7-40972b2f43e6 admin admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:11:11.603849 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74daba22-5387-4820-98f7-40972b2f43e6 admin admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:11:11.604245 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74daba22-5387-4820-98f7-40972b2f43e6 admin admin] PATH_INFO: `/v3/roles` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:11:11.605156 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74daba22-5387-4820-98f7-40972b2f43e6 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:11:11.606882 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74daba22-5387-4820-98f7-40972b2f43e6 admin admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:11:11.618227 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 1335/2656] 10.222.0.30 () {58 vars in 1121 bytes} [Mon May 23 13:11:11 2022] GET /identity/v3/roles => generated 1420 bytes in 87 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) May 23 13:11:11.622290 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-277828e2-dadc-460f-a2d9-40294584a358 admin admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:11:11.634028 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-277828e2-dadc-460f-a2d9-40294584a358 admin admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:11:11.638918 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46fc88dc-70d9-4f63-98f3-fcb644a6d094 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:11:11.639876 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46fc88dc-70d9-4f63-98f3-fcb644a6d094 admin admin] REQUEST_METHOD: `PUT` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:11:11.640326 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46fc88dc-70d9-4f63-98f3-fcb644a6d094 admin admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:11:11.643405 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46fc88dc-70d9-4f63-98f3-fcb644a6d094 admin admin] PATH_INFO: `/v3/projects/5ec3d4c508314b668b685921169854d6/users/03422526f5b843d49823f4f2316f7f56/roles/bc3c3c9d654940cbaa5177a6be23d1bc` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:11:11.643405 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46fc88dc-70d9-4f63-98f3-fcb644a6d094 admin admin] RBAC: Authorizing `identity:create_grant(project_id=5ec3d4c508314b668b685921169854d6, user_id=03422526f5b843d49823f4f2316f7f56, role_id=bc3c3c9d654940cbaa5177a6be23d1bc)` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:11:11.645399 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46fc88dc-70d9-4f63-98f3-fcb644a6d094 admin admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:11:11.651540 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 13:11:11.657012 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 1322/2657] 10.222.0.30 () {60 vars in 1482 bytes} [Mon May 23 13:11:11 2022] PUT /identity/v3/projects/5ec3d4c508314b668b685921169854d6/users/03422526f5b843d49823f4f2316f7f56/roles/bc3c3c9d654940cbaa5177a6be23d1bc => generated 0 bytes in 35 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) May 23 13:11:17.318154 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b693880-252b-4f9c-b553-967e885fbaeb None None] REQUEST_METHOD: `POST` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:11:17.318732 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b693880-252b-4f9c-b553-967e885fbaeb None None] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:11:17.318732 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b693880-252b-4f9c-b553-967e885fbaeb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:11:17.348154 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.auth.core [None req-0b693880-252b-4f9c-b553-967e885fbaeb None None] MFA Rules not processed for user `03422526f5b843d49823f4f2316f7f56`. Rule list: `[]` (Enabled: `True`). {{(pid=92057) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 13:11:17.349528 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.fernet_utils [None req-0b693880-252b-4f9c-b553-967e885fbaeb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92057) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:11:17.358605 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 13:11:17.436423 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 1336/2658] 10.222.0.30 () {58 vars in 957 bytes} [Mon May 23 13:11:17 2022] POST /identity/v3/auth/tokens => generated 3317 bytes in 119 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) May 23 13:11:17.449775 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46fc88dc-70d9-4f63-98f3-fcb644a6d094 admin admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:11:17.451639 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.fernet_utils [None req-46fc88dc-70d9-4f63-98f3-fcb644a6d094 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92056) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:11:17.515870 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46fc88dc-70d9-4f63-98f3-fcb644a6d094 admin admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:11:17.520487 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20239e25-5142-441d-8005-065286bd8bc1 service glance] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '5cea0feed4254ec085854986f99acf30', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9c389ae65aaa4542bb5a5c4fd0bacf5b', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:11:17.521304 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20239e25-5142-441d-8005-065286bd8bc1 service glance] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:11:17.521436 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20239e25-5142-441d-8005-065286bd8bc1 service glance] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:11:17.521681 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20239e25-5142-441d-8005-065286bd8bc1 service glance] PATH_INFO: `/v3/auth/tokens` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:11:17.586295 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20239e25-5142-441d-8005-065286bd8bc1 service glance] RBAC: Authorizing `identity:validate_token()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:11:17.587742 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20239e25-5142-441d-8005-065286bd8bc1 service glance] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:11:17.597541 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 1323/2659] 10.222.0.30 () {60 vars in 1365 bytes} [Mon May 23 13:11:17 2022] GET /identity/v3/auth/tokens => generated 3250 bytes in 148 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) May 23 13:11:17.801123 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30a99a49-d874-4f01-9ee1-16133da8d4e2 None None] REQUEST_METHOD: `POST` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:11:17.807109 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30a99a49-d874-4f01-9ee1-16133da8d4e2 None None] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:11:17.807109 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30a99a49-d874-4f01-9ee1-16133da8d4e2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:11:17.822055 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.auth.core [None req-30a99a49-d874-4f01-9ee1-16133da8d4e2 None None] MFA Rules not processed for user `6d21a56126fd4ccd837c33ccc08ab609`. Rule list: `[]` (Enabled: `True`). {{(pid=92057) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 13:11:17.823291 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.fernet_utils [None req-30a99a49-d874-4f01-9ee1-16133da8d4e2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92057) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:11:17.881085 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 1337/2660] 10.222.0.30 () {58 vars in 1009 bytes} [Mon May 23 13:11:17 2022] POST /identity/v3/auth/tokens => generated 3185 bytes in 81 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) May 23 13:11:17.902752 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20239e25-5142-441d-8005-065286bd8bc1 service glance] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:11:17.904142 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.fernet_utils [None req-20239e25-5142-441d-8005-065286bd8bc1 service glance] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92056) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:11:17.973422 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20239e25-5142-441d-8005-065286bd8bc1 service glance] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:11:17.978320 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31613cc6-59ef-40f9-b909-664e1a778a1c service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '138d91edb99f450e846765f9eef00d62', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9c389ae65aaa4542bb5a5c4fd0bacf5b', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'service', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:11:17.978961 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31613cc6-59ef-40f9-b909-664e1a778a1c service swift] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:11:17.979247 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31613cc6-59ef-40f9-b909-664e1a778a1c service swift] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:11:17.979518 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31613cc6-59ef-40f9-b909-664e1a778a1c service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:11:17.988491 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31613cc6-59ef-40f9-b909-664e1a778a1c service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:11:17.990105 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31613cc6-59ef-40f9-b909-664e1a778a1c service swift] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:11:17.998066 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 1324/2661] 10.222.0.30 () {60 vars in 1384 bytes} [Mon May 23 13:11:17 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 96 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) May 23 13:11:27.386420 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30a99a49-d874-4f01-9ee1-16133da8d4e2 None None] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:11:27.387828 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.fernet_utils [None req-30a99a49-d874-4f01-9ee1-16133da8d4e2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92057) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:11:27.455557 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30a99a49-d874-4f01-9ee1-16133da8d4e2 None None] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:11:27.459334 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95da9ce8-5a6a-4f24-b324-2322ea624867 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1666097f8ed545c6b0e4809c93e4775a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9c389ae65aaa4542bb5a5c4fd0bacf5b', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:11:27.460012 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95da9ce8-5a6a-4f24-b324-2322ea624867 service cinder] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:11:27.460295 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95da9ce8-5a6a-4f24-b324-2322ea624867 service cinder] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:11:27.460572 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95da9ce8-5a6a-4f24-b324-2322ea624867 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:11:27.462537 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.fernet_utils [None req-95da9ce8-5a6a-4f24-b324-2322ea624867 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92057) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:11:27.537354 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=100 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 13:11:27.552571 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95da9ce8-5a6a-4f24-b324-2322ea624867 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:11:27.554292 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95da9ce8-5a6a-4f24-b324-2322ea624867 service cinder] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:11:27.577011 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 1338/2662] 10.222.0.30 () {60 vars in 1365 bytes} [Mon May 23 13:11:27 2022] GET /identity/v3/auth/tokens => generated 3252 bytes in 191 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) May 23 13:11:27.994862 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=81 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 13:11:28.599564 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea4c8f6e-d284-4bba-b5b2-9f218143a788 None None] REQUEST_METHOD: `POST` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:11:28.599849 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea4c8f6e-d284-4bba-b5b2-9f218143a788 None None] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:11:28.600128 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea4c8f6e-d284-4bba-b5b2-9f218143a788 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:11:28.618040 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.auth.core [None req-ea4c8f6e-d284-4bba-b5b2-9f218143a788 None None] MFA Rules not processed for user `20d9973f32f847a6a54a60405beabdbf`. Rule list: `[]` (Enabled: `True`). {{(pid=92056) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 13:11:28.619236 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.fernet_utils [None req-ea4c8f6e-d284-4bba-b5b2-9f218143a788 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92056) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:11:28.626541 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 1325/2663] 10.222.0.30 () {58 vars in 957 bytes} [Mon May 23 13:11:28 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 28 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) May 23 13:11:29.327572 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95da9ce8-5a6a-4f24-b324-2322ea624867 service cinder] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:11:29.330441 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.fernet_utils [None req-95da9ce8-5a6a-4f24-b324-2322ea624867 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92057) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:11:29.416853 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95da9ce8-5a6a-4f24-b324-2322ea624867 service cinder] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:11:29.422277 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68f75b8d-50b1-4786-9bd4-e73c7dce3edb service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd5fb51f8d0fa46439cb29666970b885b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9c389ae65aaa4542bb5a5c4fd0bacf5b', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'service', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:11:29.423549 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68f75b8d-50b1-4786-9bd4-e73c7dce3edb service placement] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:11:29.423833 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68f75b8d-50b1-4786-9bd4-e73c7dce3edb service placement] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:11:29.424113 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68f75b8d-50b1-4786-9bd4-e73c7dce3edb service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:11:29.425948 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.fernet_utils [None req-68f75b8d-50b1-4786-9bd4-e73c7dce3edb service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92057) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:11:29.437437 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68f75b8d-50b1-4786-9bd4-e73c7dce3edb service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:11:29.440012 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68f75b8d-50b1-4786-9bd4-e73c7dce3edb service placement] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:11:29.449955 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 1339/2664] 10.222.0.30 () {60 vars in 1365 bytes} [Mon May 23 13:11:29 2022] GET /identity/v3/auth/tokens => generated 3298 bytes in 123 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) May 23 13:11:30.487038 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea4c8f6e-d284-4bba-b5b2-9f218143a788 None None] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:11:30.488476 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.fernet_utils [None req-ea4c8f6e-d284-4bba-b5b2-9f218143a788 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92056) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:11:30.550409 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea4c8f6e-d284-4bba-b5b2-9f218143a788 None None] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:11:30.554102 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3effcab1-d5a1-4c3b-8f88-64e0988cc859 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2eb83ca38e79449095f3a2f947bbb06a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9c389ae65aaa4542bb5a5c4fd0bacf5b', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:11:30.554780 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3effcab1-d5a1-4c3b-8f88-64e0988cc859 service neutron] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:11:30.555067 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3effcab1-d5a1-4c3b-8f88-64e0988cc859 service neutron] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:11:30.555348 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3effcab1-d5a1-4c3b-8f88-64e0988cc859 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:11:30.557130 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.fernet_utils [None req-3effcab1-d5a1-4c3b-8f88-64e0988cc859 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92056) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:11:30.623324 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3effcab1-d5a1-4c3b-8f88-64e0988cc859 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:11:30.625143 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3effcab1-d5a1-4c3b-8f88-64e0988cc859 service neutron] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:11:30.635225 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 1326/2665] 10.222.0.30 () {60 vars in 1365 bytes} [Mon May 23 13:11:30 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 149 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) May 23 13:11:32.277168 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f8f456d-0aef-4b00-96e2-714d7a61b113 None None] REQUEST_METHOD: `POST` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:11:32.277638 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f8f456d-0aef-4b00-96e2-714d7a61b113 None None] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:11:32.277741 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f8f456d-0aef-4b00-96e2-714d7a61b113 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:11:32.296598 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.auth.core [None req-5f8f456d-0aef-4b00-96e2-714d7a61b113 None None] MFA Rules not processed for user `6d21a56126fd4ccd837c33ccc08ab609`. Rule list: `[]` (Enabled: `True`). {{(pid=92057) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 13:11:32.298498 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.fernet_utils [None req-5f8f456d-0aef-4b00-96e2-714d7a61b113 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92057) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:11:32.306209 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 1340/2666] 10.222.0.30 () {58 vars in 1009 bytes} [Mon May 23 13:11:32 2022] POST /identity/v3/auth/tokens => generated 3185 bytes in 30 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) May 23 13:11:32.319969 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3effcab1-d5a1-4c3b-8f88-64e0988cc859 service neutron] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:11:32.332812 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3effcab1-d5a1-4c3b-8f88-64e0988cc859 service neutron] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:11:32.337927 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51a79bda-30d2-4631-8c26-5459cedfcf76 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '138d91edb99f450e846765f9eef00d62', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9c389ae65aaa4542bb5a5c4fd0bacf5b', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'service', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:11:32.338761 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51a79bda-30d2-4631-8c26-5459cedfcf76 service swift] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:11:32.339042 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51a79bda-30d2-4631-8c26-5459cedfcf76 service swift] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:11:32.339487 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51a79bda-30d2-4631-8c26-5459cedfcf76 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:11:32.348262 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51a79bda-30d2-4631-8c26-5459cedfcf76 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:11:32.349822 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51a79bda-30d2-4631-8c26-5459cedfcf76 service swift] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:11:32.358661 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 1327/2667] 10.222.0.30 () {60 vars in 1384 bytes} [Mon May 23 13:11:32 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 39 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) May 23 13:11:34.838894 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f8f456d-0aef-4b00-96e2-714d7a61b113 None None] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:11:34.852653 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f8f456d-0aef-4b00-96e2-714d7a61b113 None None] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:11:34.858111 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-447cdd1d-07b0-4eff-bcdf-eebc6e7ad645 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd5fb51f8d0fa46439cb29666970b885b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9c389ae65aaa4542bb5a5c4fd0bacf5b', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'service', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:11:34.858886 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-447cdd1d-07b0-4eff-bcdf-eebc6e7ad645 service placement] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:11:34.859979 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-447cdd1d-07b0-4eff-bcdf-eebc6e7ad645 service placement] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:11:34.860492 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-447cdd1d-07b0-4eff-bcdf-eebc6e7ad645 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:11:34.862372 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.fernet_utils [None req-447cdd1d-07b0-4eff-bcdf-eebc6e7ad645 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92057) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:11:34.883460 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-447cdd1d-07b0-4eff-bcdf-eebc6e7ad645 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:11:34.885146 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-447cdd1d-07b0-4eff-bcdf-eebc6e7ad645 service placement] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:11:34.896374 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 1341/2668] 10.222.0.30 () {60 vars in 1365 bytes} [Mon May 23 13:11:34 2022] GET /identity/v3/auth/tokens => generated 3298 bytes in 58 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) May 23 13:11:34.979176 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51a79bda-30d2-4631-8c26-5459cedfcf76 service swift] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:11:34.981576 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.fernet_utils [None req-51a79bda-30d2-4631-8c26-5459cedfcf76 service swift] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92056) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:11:34.996470 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51a79bda-30d2-4631-8c26-5459cedfcf76 service swift] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:11:35.010362 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5144d218-61ac-481e-b1c6-f7f71c64d00f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:11:35.011258 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5144d218-61ac-481e-b1c6-f7f71c64d00f admin admin] REQUEST_METHOD: `DELETE` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:11:35.011564 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5144d218-61ac-481e-b1c6-f7f71c64d00f admin admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:11:35.012076 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5144d218-61ac-481e-b1c6-f7f71c64d00f admin admin] PATH_INFO: `/v3/users/257251830f5148e5b9e435c2c13828e8` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:11:35.013479 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5144d218-61ac-481e-b1c6-f7f71c64d00f admin admin] RBAC: Authorizing `identity:delete_user(user_id=257251830f5148e5b9e435c2c13828e8)` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:11:35.016433 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5144d218-61ac-481e-b1c6-f7f71c64d00f admin admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:11:35.220651 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-5144d218-61ac-481e-b1c6-f7f71c64d00f admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '257251830f5148e5b9e435c2c13828e8'} {{(pid=92056) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:11:35.221507 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-5144d218-61ac-481e-b1c6-f7f71c64d00f admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '257251830f5148e5b9e435c2c13828e8'} {{(pid=92056) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:11:35.294464 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-5144d218-61ac-481e-b1c6-f7f71c64d00f admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '257251830f5148e5b9e435c2c13828e8'} {{(pid=92056) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:11:35.308414 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-5144d218-61ac-481e-b1c6-f7f71c64d00f admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '257251830f5148e5b9e435c2c13828e8'} {{(pid=92056) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:11:35.316269 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-5144d218-61ac-481e-b1c6-f7f71c64d00f admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '257251830f5148e5b9e435c2c13828e8'} {{(pid=92056) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:11:35.324742 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 1328/2669] 10.222.0.30 () {58 vars in 1223 bytes} [Mon May 23 13:11:34 2022] DELETE /identity/v3/users/257251830f5148e5b9e435c2c13828e8 => generated 0 bytes in 346 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) May 23 13:11:35.623132 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-447cdd1d-07b0-4eff-bcdf-eebc6e7ad645 service placement] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:11:35.701245 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-447cdd1d-07b0-4eff-bcdf-eebc6e7ad645 service placement] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:11:35.783735 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c98e194-6d16-4a09-b3ae-738e6193f41c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:11:35.784644 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c98e194-6d16-4a09-b3ae-738e6193f41c admin admin] REQUEST_METHOD: `DELETE` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:11:35.784861 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c98e194-6d16-4a09-b3ae-738e6193f41c admin admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:11:35.785207 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c98e194-6d16-4a09-b3ae-738e6193f41c admin admin] PATH_INFO: `/v3/projects/0ae347c79fe74082906f0d8eb9dc6677` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:11:35.786360 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c98e194-6d16-4a09-b3ae-738e6193f41c admin admin] RBAC: Authorizing `identity:delete_project(project_id=0ae347c79fe74082906f0d8eb9dc6677)` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:11:35.788261 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c98e194-6d16-4a09-b3ae-738e6193f41c admin admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:11:35.798431 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=100 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 13:11:35.929859 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [None req-8c98e194-6d16-4a09-b3ae-738e6193f41c admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '0ae347c79fe74082906f0d8eb9dc6677', 'request_id': 'req-8c98e194-6d16-4a09-b3ae-738e6193f41c'} {{(pid=92057) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:11:35.936199 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [None req-8c98e194-6d16-4a09-b3ae-738e6193f41c admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '0ae347c79fe74082906f0d8eb9dc6677', 'request_id': 'req-8c98e194-6d16-4a09-b3ae-738e6193f41c'} {{(pid=92057) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:11:36.006966 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [None req-8c98e194-6d16-4a09-b3ae-738e6193f41c admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '0ae347c79fe74082906f0d8eb9dc6677', 'request_id': 'req-8c98e194-6d16-4a09-b3ae-738e6193f41c'} {{(pid=92057) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:11:36.035366 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [None req-8c98e194-6d16-4a09-b3ae-738e6193f41c admin admin] The token cache is being invalidate because project 0ae347c79fe74082906f0d8eb9dc6677 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=92057) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} May 23 13:11:36.035610 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [None req-8c98e194-6d16-4a09-b3ae-738e6193f41c admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=92057) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:11:36.036212 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [None req-8c98e194-6d16-4a09-b3ae-738e6193f41c admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=92057) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:11:36.038005 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 1342/2670] 10.222.0.30 () {58 vars in 1232 bytes} [Mon May 23 13:11:35 2022] DELETE /identity/v3/projects/0ae347c79fe74082906f0d8eb9dc6677 => generated 0 bytes in 415 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) May 23 13:11:36.042862 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bad800a-4aeb-4e86-9b65-4c4b1ab4deb3 None None] REQUEST_METHOD: `POST` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:11:36.043390 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bad800a-4aeb-4e86-9b65-4c4b1ab4deb3 None None] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:11:36.043519 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bad800a-4aeb-4e86-9b65-4c4b1ab4deb3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:11:36.048787 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 13:11:36.082809 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.auth.core [None req-3bad800a-4aeb-4e86-9b65-4c4b1ab4deb3 None None] MFA Rules not processed for user `0f7119f8fa5c4445977671140ded91a4`. Rule list: `[]` (Enabled: `True`). {{(pid=92056) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 13:11:36.084317 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.fernet_utils [None req-3bad800a-4aeb-4e86-9b65-4c4b1ab4deb3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92056) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:11:36.146305 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 1329/2671] 10.222.0.30 () {58 vars in 957 bytes} [Mon May 23 13:11:36 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 104 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) May 23 13:11:36.154647 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-faf1e9f6-dddc-48a9-923f-df62b29586ae None None] REQUEST_METHOD: `POST` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:11:36.154932 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-faf1e9f6-dddc-48a9-923f-df62b29586ae None None] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:11:36.155209 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-faf1e9f6-dddc-48a9-923f-df62b29586ae None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:11:36.174405 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.auth.core [None req-faf1e9f6-dddc-48a9-923f-df62b29586ae None None] MFA Rules not processed for user `0f7119f8fa5c4445977671140ded91a4`. Rule list: `[]` (Enabled: `True`). {{(pid=92057) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 13:11:36.175607 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.fernet_utils [None req-faf1e9f6-dddc-48a9-923f-df62b29586ae None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92057) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:11:36.183804 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 1343/2672] 10.222.0.30 () {58 vars in 957 bytes} [Mon May 23 13:11:36 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 30 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) May 23 13:11:36.188610 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3bad800a-4aeb-4e86-9b65-4c4b1ab4deb3 None None] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:11:36.198780 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3bad800a-4aeb-4e86-9b65-4c4b1ab4deb3 None None] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:11:36.204159 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca1fd4e3-2dca-4112-be23-2391a2f837e0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:11:36.204827 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca1fd4e3-2dca-4112-be23-2391a2f837e0 admin admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:11:36.205224 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca1fd4e3-2dca-4112-be23-2391a2f837e0 admin admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:11:36.205535 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca1fd4e3-2dca-4112-be23-2391a2f837e0 admin admin] PATH_INFO: `/v3/domains` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:11:36.206315 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca1fd4e3-2dca-4112-be23-2391a2f837e0 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=92056) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} May 23 13:11:36.206654 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca1fd4e3-2dca-4112-be23-2391a2f837e0 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:11:36.208311 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca1fd4e3-2dca-4112-be23-2391a2f837e0 admin admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:11:36.220461 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 1330/2673] 10.222.0.30 () {58 vars in 1152 bytes} [Mon May 23 13:11:36 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) May 23 13:11:36.224810 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-faf1e9f6-dddc-48a9-923f-df62b29586ae None None] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:11:36.236900 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-faf1e9f6-dddc-48a9-923f-df62b29586ae None None] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:11:36.241499 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0bc78a0-b614-4c57-ba69-47f7158d1bb4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:11:36.242184 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0bc78a0-b614-4c57-ba69-47f7158d1bb4 admin admin] REQUEST_METHOD: `POST` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:11:36.242469 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0bc78a0-b614-4c57-ba69-47f7158d1bb4 admin admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:11:36.242743 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0bc78a0-b614-4c57-ba69-47f7158d1bb4 admin admin] PATH_INFO: `/v3/projects` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:11:36.243430 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0bc78a0-b614-4c57-ba69-47f7158d1bb4 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:11:36.245336 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0bc78a0-b614-4c57-ba69-47f7158d1bb4 admin admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:11:36.327826 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 1344/2674] 10.222.0.30 () {60 vars in 1152 bytes} [Mon May 23 13:11:36 2022] POST /identity/v3/projects => generated 357 bytes in 103 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) May 23 13:11:36.332659 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca1fd4e3-2dca-4112-be23-2391a2f837e0 admin admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:11:36.342726 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca1fd4e3-2dca-4112-be23-2391a2f837e0 admin admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:11:36.410449 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e745eb61-0696-4a9f-a29c-ad26b6a3fc1d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:11:36.411131 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e745eb61-0696-4a9f-a29c-ad26b6a3fc1d admin admin] REQUEST_METHOD: `POST` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:11:36.411412 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e745eb61-0696-4a9f-a29c-ad26b6a3fc1d admin admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:11:36.411687 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e745eb61-0696-4a9f-a29c-ad26b6a3fc1d admin admin] PATH_INFO: `/v3/users` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:11:36.412397 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e745eb61-0696-4a9f-a29c-ad26b6a3fc1d admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:11:36.414363 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e745eb61-0696-4a9f-a29c-ad26b6a3fc1d admin admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:11:36.493360 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 1331/2675] 10.222.0.30 () {60 vars in 1143 bytes} [Mon May 23 13:11:36 2022] POST /identity/v3/users => generated 279 bytes in 161 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) May 23 13:11:36.497753 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0bc78a0-b614-4c57-ba69-47f7158d1bb4 admin admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:11:36.509256 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0bc78a0-b614-4c57-ba69-47f7158d1bb4 admin admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:11:36.513920 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0c9c648-7369-41e0-813f-00c1ae6f4b60 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:11:36.514649 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0c9c648-7369-41e0-813f-00c1ae6f4b60 admin admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:11:36.514940 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0c9c648-7369-41e0-813f-00c1ae6f4b60 admin admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:11:36.515220 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0c9c648-7369-41e0-813f-00c1ae6f4b60 admin admin] PATH_INFO: `/v3/roles` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:11:36.516122 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0c9c648-7369-41e0-813f-00c1ae6f4b60 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:11:36.517755 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0c9c648-7369-41e0-813f-00c1ae6f4b60 admin admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:11:36.528531 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 1345/2676] 10.222.0.30 () {58 vars in 1121 bytes} [Mon May 23 13:11:36 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) May 23 13:11:36.532852 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e745eb61-0696-4a9f-a29c-ad26b6a3fc1d admin admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:11:36.543512 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e745eb61-0696-4a9f-a29c-ad26b6a3fc1d admin admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:11:36.549165 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-126bc8d9-470a-428e-8486-988575c9c267 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:11:36.549977 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-126bc8d9-470a-428e-8486-988575c9c267 admin admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:11:36.550419 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-126bc8d9-470a-428e-8486-988575c9c267 admin admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:11:36.550759 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-126bc8d9-470a-428e-8486-988575c9c267 admin admin] PATH_INFO: `/v3/roles` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:11:36.551602 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-126bc8d9-470a-428e-8486-988575c9c267 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:11:36.553250 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-126bc8d9-470a-428e-8486-988575c9c267 admin admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:11:36.568754 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 1332/2677] 10.222.0.30 () {58 vars in 1121 bytes} [Mon May 23 13:11:36 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) May 23 13:11:36.570004 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0c9c648-7369-41e0-813f-00c1ae6f4b60 admin admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:11:36.580382 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0c9c648-7369-41e0-813f-00c1ae6f4b60 admin admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:11:36.585853 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f88d4d2-9aad-41ed-873a-ed3a7b2328a4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:11:36.586690 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f88d4d2-9aad-41ed-873a-ed3a7b2328a4 admin admin] REQUEST_METHOD: `PUT` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:11:36.586979 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f88d4d2-9aad-41ed-873a-ed3a7b2328a4 admin admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:11:36.587259 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f88d4d2-9aad-41ed-873a-ed3a7b2328a4 admin admin] PATH_INFO: `/v3/projects/2ffb230cff3247c79c7a3b8a03ed7fbe/users/8fd11f00514f42009e72bfb0454309aa/roles/80fb22507c9a4ff09d44d8f12f6fff1a` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:11:36.605899 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f88d4d2-9aad-41ed-873a-ed3a7b2328a4 admin admin] RBAC: Authorizing `identity:create_grant(project_id=2ffb230cff3247c79c7a3b8a03ed7fbe, user_id=8fd11f00514f42009e72bfb0454309aa, role_id=80fb22507c9a4ff09d44d8f12f6fff1a)` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:11:36.610909 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f88d4d2-9aad-41ed-873a-ed3a7b2328a4 admin admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:11:36.677903 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 1346/2678] 10.222.0.30 () {60 vars in 1482 bytes} [Mon May 23 13:11:36 2022] PUT /identity/v3/projects/2ffb230cff3247c79c7a3b8a03ed7fbe/users/8fd11f00514f42009e72bfb0454309aa/roles/80fb22507c9a4ff09d44d8f12f6fff1a => generated 0 bytes in 108 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) May 23 13:11:36.689047 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6d08524-f3ce-4453-a704-bc0582831aae None None] REQUEST_METHOD: `POST` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:11:36.690257 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6d08524-f3ce-4453-a704-bc0582831aae None None] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:11:36.690713 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6d08524-f3ce-4453-a704-bc0582831aae None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:11:36.736143 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.auth.core [None req-c6d08524-f3ce-4453-a704-bc0582831aae None None] MFA Rules not processed for user `8fd11f00514f42009e72bfb0454309aa`. Rule list: `[]` (Enabled: `True`). {{(pid=92056) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 13:11:36.737609 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.fernet_utils [None req-c6d08524-f3ce-4453-a704-bc0582831aae None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92056) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:11:36.792918 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 13:11:36.826505 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 1333/2679] 10.222.0.30 () {58 vars in 957 bytes} [Mon May 23 13:11:36 2022] POST /identity/v3/auth/tokens => generated 3224 bytes in 139 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) May 23 13:11:36.827164 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f88d4d2-9aad-41ed-873a-ed3a7b2328a4 admin admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:11:36.920892 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f88d4d2-9aad-41ed-873a-ed3a7b2328a4 admin admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:11:36.927140 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2ee4887-86ae-4496-95c7-77161a3e9dc7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:11:36.927846 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2ee4887-86ae-4496-95c7-77161a3e9dc7 admin admin] REQUEST_METHOD: `POST` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:11:36.928131 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2ee4887-86ae-4496-95c7-77161a3e9dc7 admin admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:11:36.928409 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2ee4887-86ae-4496-95c7-77161a3e9dc7 admin admin] PATH_INFO: `/v3/projects` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:11:36.929114 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2ee4887-86ae-4496-95c7-77161a3e9dc7 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:11:36.931098 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2ee4887-86ae-4496-95c7-77161a3e9dc7 admin admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:11:37.061599 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 1347/2680] 10.222.0.30 () {60 vars in 1152 bytes} [Mon May 23 13:11:36 2022] POST /identity/v3/projects => generated 357 bytes in 235 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) May 23 13:11:37.068591 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6d08524-f3ce-4453-a704-bc0582831aae None None] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:11:37.079616 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6d08524-f3ce-4453-a704-bc0582831aae None None] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:11:37.141592 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10abd52c-341a-4e29-abb3-6ee8857e0e9b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:11:37.142290 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10abd52c-341a-4e29-abb3-6ee8857e0e9b admin admin] REQUEST_METHOD: `POST` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:11:37.142574 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10abd52c-341a-4e29-abb3-6ee8857e0e9b admin admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:11:37.142849 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10abd52c-341a-4e29-abb3-6ee8857e0e9b admin admin] PATH_INFO: `/v3/users` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:11:37.143539 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10abd52c-341a-4e29-abb3-6ee8857e0e9b admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:11:37.145509 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10abd52c-341a-4e29-abb3-6ee8857e0e9b admin admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:11:37.254094 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 1334/2681] 10.222.0.30 () {60 vars in 1143 bytes} [Mon May 23 13:11:37 2022] POST /identity/v3/users => generated 285 bytes in 186 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) May 23 13:11:37.257903 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2ee4887-86ae-4496-95c7-77161a3e9dc7 admin admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:11:37.269458 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2ee4887-86ae-4496-95c7-77161a3e9dc7 admin admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:11:37.274324 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53cb20f0-9a8a-4ee6-8552-7185da4b69e9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:11:37.275110 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53cb20f0-9a8a-4ee6-8552-7185da4b69e9 admin admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:11:37.275404 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53cb20f0-9a8a-4ee6-8552-7185da4b69e9 admin admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:11:37.275682 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53cb20f0-9a8a-4ee6-8552-7185da4b69e9 admin admin] PATH_INFO: `/v3/roles` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:11:37.276406 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53cb20f0-9a8a-4ee6-8552-7185da4b69e9 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:11:37.278043 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53cb20f0-9a8a-4ee6-8552-7185da4b69e9 admin admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:11:37.279554 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=89,keystone:DELETE=7,keystone:INSERT=4 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 13:11:37.290494 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 1348/2682] 10.222.0.30 () {58 vars in 1121 bytes} [Mon May 23 13:11:37 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) May 23 13:11:37.294658 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10abd52c-341a-4e29-abb3-6ee8857e0e9b admin admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:11:37.305741 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10abd52c-341a-4e29-abb3-6ee8857e0e9b admin admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:11:37.310407 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f11db750-baf6-4fa9-8486-131ea4fca690 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:11:37.311276 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f11db750-baf6-4fa9-8486-131ea4fca690 admin admin] REQUEST_METHOD: `PUT` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:11:37.311650 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f11db750-baf6-4fa9-8486-131ea4fca690 admin admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:11:37.311922 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f11db750-baf6-4fa9-8486-131ea4fca690 admin admin] PATH_INFO: `/v3/projects/80f999adea41454198f57871becedc41/users/944a40695a1b4aae934b1248524d47f0/roles/bc3c3c9d654940cbaa5177a6be23d1bc` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:11:37.327705 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f11db750-baf6-4fa9-8486-131ea4fca690 admin admin] RBAC: Authorizing `identity:create_grant(project_id=80f999adea41454198f57871becedc41, user_id=944a40695a1b4aae934b1248524d47f0, role_id=bc3c3c9d654940cbaa5177a6be23d1bc)` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:11:37.331264 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f11db750-baf6-4fa9-8486-131ea4fca690 admin admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:11:37.419560 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 1335/2683] 10.222.0.30 () {60 vars in 1482 bytes} [Mon May 23 13:11:37 2022] PUT /identity/v3/projects/80f999adea41454198f57871becedc41/users/944a40695a1b4aae934b1248524d47f0/roles/bc3c3c9d654940cbaa5177a6be23d1bc => generated 0 bytes in 125 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) May 23 13:11:37.423892 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53cb20f0-9a8a-4ee6-8552-7185da4b69e9 admin admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:11:37.434061 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53cb20f0-9a8a-4ee6-8552-7185da4b69e9 admin admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:11:37.493769 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf71656e-c153-4a2d-88bb-dd3dc8094575 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:11:37.494470 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf71656e-c153-4a2d-88bb-dd3dc8094575 admin admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:11:37.494758 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf71656e-c153-4a2d-88bb-dd3dc8094575 admin admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:11:37.495038 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf71656e-c153-4a2d-88bb-dd3dc8094575 admin admin] PATH_INFO: `/v3/roles` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:11:37.495763 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf71656e-c153-4a2d-88bb-dd3dc8094575 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:11:37.497387 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf71656e-c153-4a2d-88bb-dd3dc8094575 admin admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:11:37.508970 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 1349/2684] 10.222.0.30 () {58 vars in 1121 bytes} [Mon May 23 13:11:37 2022] GET /identity/v3/roles => generated 1420 bytes in 86 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) May 23 13:11:37.513063 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f11db750-baf6-4fa9-8486-131ea4fca690 admin admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:11:37.528272 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f11db750-baf6-4fa9-8486-131ea4fca690 admin admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:11:37.533270 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-436cd20a-13b5-40f4-bd82-054564f69402 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:11:37.534181 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-436cd20a-13b5-40f4-bd82-054564f69402 admin admin] REQUEST_METHOD: `PUT` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:11:37.534463 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-436cd20a-13b5-40f4-bd82-054564f69402 admin admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:11:37.534737 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-436cd20a-13b5-40f4-bd82-054564f69402 admin admin] PATH_INFO: `/v3/projects/80f999adea41454198f57871becedc41/users/944a40695a1b4aae934b1248524d47f0/roles/bc3c3c9d654940cbaa5177a6be23d1bc` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:11:37.536696 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-436cd20a-13b5-40f4-bd82-054564f69402 admin admin] RBAC: Authorizing `identity:create_grant(project_id=80f999adea41454198f57871becedc41, user_id=944a40695a1b4aae934b1248524d47f0, role_id=bc3c3c9d654940cbaa5177a6be23d1bc)` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:11:37.539407 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-436cd20a-13b5-40f4-bd82-054564f69402 admin admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:11:37.550829 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 1336/2685] 10.222.0.30 () {60 vars in 1482 bytes} [Mon May 23 13:11:37 2022] PUT /identity/v3/projects/80f999adea41454198f57871becedc41/users/944a40695a1b4aae934b1248524d47f0/roles/bc3c3c9d654940cbaa5177a6be23d1bc => generated 0 bytes in 38 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) May 23 13:11:37.561754 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc30d3da-e95f-4ba3-b48f-711c7a08f2f1 None None] REQUEST_METHOD: `POST` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:11:37.562079 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc30d3da-e95f-4ba3-b48f-711c7a08f2f1 None None] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:11:37.562884 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc30d3da-e95f-4ba3-b48f-711c7a08f2f1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:11:37.590754 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.auth.core [None req-fc30d3da-e95f-4ba3-b48f-711c7a08f2f1 None None] MFA Rules not processed for user `944a40695a1b4aae934b1248524d47f0`. Rule list: `[]` (Enabled: `True`). {{(pid=92057) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 13:11:37.591956 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.fernet_utils [None req-fc30d3da-e95f-4ba3-b48f-711c7a08f2f1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92057) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:11:37.664488 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 1350/2686] 10.222.0.30 () {58 vars in 957 bytes} [Mon May 23 13:11:37 2022] POST /identity/v3/auth/tokens => generated 3291 bytes in 104 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) May 23 13:11:37.684707 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-436cd20a-13b5-40f4-bd82-054564f69402 admin admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:11:37.686549 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.fernet_utils [None req-436cd20a-13b5-40f4-bd82-054564f69402 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92056) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:11:37.765362 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-436cd20a-13b5-40f4-bd82-054564f69402 admin admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:11:37.769186 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b6b5aa9-6db1-487b-a6ef-79ebb5e9bf67 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1666097f8ed545c6b0e4809c93e4775a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9c389ae65aaa4542bb5a5c4fd0bacf5b', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:11:37.769839 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b6b5aa9-6db1-487b-a6ef-79ebb5e9bf67 service cinder] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:11:37.770154 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b6b5aa9-6db1-487b-a6ef-79ebb5e9bf67 service cinder] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:11:37.770430 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b6b5aa9-6db1-487b-a6ef-79ebb5e9bf67 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:11:37.780310 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b6b5aa9-6db1-487b-a6ef-79ebb5e9bf67 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:11:37.782568 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b6b5aa9-6db1-487b-a6ef-79ebb5e9bf67 service cinder] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:11:37.792800 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 1337/2687] 10.222.0.30 () {60 vars in 1365 bytes} [Mon May 23 13:11:37 2022] GET /identity/v3/auth/tokens => generated 3291 bytes in 109 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) May 23 13:11:37.996631 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc30d3da-e95f-4ba3-b48f-711c7a08f2f1 None None] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:11:38.007260 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc30d3da-e95f-4ba3-b48f-711c7a08f2f1 None None] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:11:38.010952 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3efbf8d-b50d-4657-bf53-47beb10781d8 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1666097f8ed545c6b0e4809c93e4775a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9c389ae65aaa4542bb5a5c4fd0bacf5b', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:11:38.011602 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3efbf8d-b50d-4657-bf53-47beb10781d8 service cinder] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:11:38.011901 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3efbf8d-b50d-4657-bf53-47beb10781d8 service cinder] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:11:38.017464 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3efbf8d-b50d-4657-bf53-47beb10781d8 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:11:38.092107 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3efbf8d-b50d-4657-bf53-47beb10781d8 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:11:38.094442 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3efbf8d-b50d-4657-bf53-47beb10781d8 service cinder] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:11:38.105019 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 1351/2688] 10.222.0.30 () {60 vars in 1365 bytes} [Mon May 23 13:11:37 2022] GET /identity/v3/auth/tokens => generated 3224 bytes in 109 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) May 23 13:11:47.788251 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=85,keystone:INSERT=5 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 13:11:48.100790 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=93 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 13:11:59.473316 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b6b5aa9-6db1-487b-a6ef-79ebb5e9bf67 service cinder] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:11:59.545360 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b6b5aa9-6db1-487b-a6ef-79ebb5e9bf67 service cinder] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:11:59.550720 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54d96a1d-11fa-48a9-9981-e79718045dc8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:11:59.551588 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54d96a1d-11fa-48a9-9981-e79718045dc8 admin admin] REQUEST_METHOD: `DELETE` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:11:59.551866 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54d96a1d-11fa-48a9-9981-e79718045dc8 admin admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:11:59.552142 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54d96a1d-11fa-48a9-9981-e79718045dc8 admin admin] PATH_INFO: `/v3/users/8fd11f00514f42009e72bfb0454309aa` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:11:59.553336 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54d96a1d-11fa-48a9-9981-e79718045dc8 admin admin] RBAC: Authorizing `identity:delete_user(user_id=8fd11f00514f42009e72bfb0454309aa)` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:11:59.555423 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54d96a1d-11fa-48a9-9981-e79718045dc8 admin admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:11:59.696995 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-54d96a1d-11fa-48a9-9981-e79718045dc8 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '8fd11f00514f42009e72bfb0454309aa'} {{(pid=92056) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:11:59.697663 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-54d96a1d-11fa-48a9-9981-e79718045dc8 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '8fd11f00514f42009e72bfb0454309aa'} {{(pid=92056) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:11:59.760721 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-54d96a1d-11fa-48a9-9981-e79718045dc8 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '8fd11f00514f42009e72bfb0454309aa'} {{(pid=92056) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:11:59.773765 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-54d96a1d-11fa-48a9-9981-e79718045dc8 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '8fd11f00514f42009e72bfb0454309aa'} {{(pid=92056) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:11:59.780497 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-54d96a1d-11fa-48a9-9981-e79718045dc8 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '8fd11f00514f42009e72bfb0454309aa'} {{(pid=92056) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:11:59.786528 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 1338/2689] 10.222.0.30 () {58 vars in 1223 bytes} [Mon May 23 13:11:59 2022] DELETE /identity/v3/users/8fd11f00514f42009e72bfb0454309aa => generated 0 bytes in 314 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) May 23 13:11:59.791384 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fbb0aff-a0ba-402e-a1a8-f86780d3df2e None None] REQUEST_METHOD: `POST` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:11:59.794748 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fbb0aff-a0ba-402e-a1a8-f86780d3df2e None None] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:11:59.794748 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fbb0aff-a0ba-402e-a1a8-f86780d3df2e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:11:59.809906 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.auth.core [None req-2fbb0aff-a0ba-402e-a1a8-f86780d3df2e None None] MFA Rules not processed for user `0f7119f8fa5c4445977671140ded91a4`. Rule list: `[]` (Enabled: `True`). {{(pid=92057) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 13:11:59.811096 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.fernet_utils [None req-2fbb0aff-a0ba-402e-a1a8-f86780d3df2e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92057) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:11:59.876513 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 1352/2690] 10.222.0.30 () {58 vars in 957 bytes} [Mon May 23 13:11:59 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) May 23 13:11:59.886917 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54d96a1d-11fa-48a9-9981-e79718045dc8 admin admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:11:59.888317 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.fernet_utils [None req-54d96a1d-11fa-48a9-9981-e79718045dc8 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92056) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:11:59.968031 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54d96a1d-11fa-48a9-9981-e79718045dc8 admin admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:11:59.972710 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-836f5c18-f1d6-425f-a8f9-76d8f725f32b service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2eb83ca38e79449095f3a2f947bbb06a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9c389ae65aaa4542bb5a5c4fd0bacf5b', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:11:59.973352 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-836f5c18-f1d6-425f-a8f9-76d8f725f32b service neutron] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:11:59.973625 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-836f5c18-f1d6-425f-a8f9-76d8f725f32b service neutron] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:11:59.973936 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-836f5c18-f1d6-425f-a8f9-76d8f725f32b service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:11:59.983590 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-836f5c18-f1d6-425f-a8f9-76d8f725f32b service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:11:59.985069 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-836f5c18-f1d6-425f-a8f9-76d8f725f32b service neutron] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:11:59.995430 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 1339/2691] 10.222.0.30 () {60 vars in 1365 bytes} [Mon May 23 13:11:59 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 109 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) May 23 13:12:00.402635 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2fbb0aff-a0ba-402e-a1a8-f86780d3df2e None None] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:12:00.404066 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.fernet_utils [None req-2fbb0aff-a0ba-402e-a1a8-f86780d3df2e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92057) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:12:00.418269 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2fbb0aff-a0ba-402e-a1a8-f86780d3df2e None None] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:12:00.435821 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b7d2636-0878-4381-b243-b425103737b5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:12:00.436785 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b7d2636-0878-4381-b243-b425103737b5 admin admin] REQUEST_METHOD: `DELETE` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:12:00.436980 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b7d2636-0878-4381-b243-b425103737b5 admin admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:12:00.437260 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b7d2636-0878-4381-b243-b425103737b5 admin admin] PATH_INFO: `/v3/projects/2ffb230cff3247c79c7a3b8a03ed7fbe` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:12:00.438618 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b7d2636-0878-4381-b243-b425103737b5 admin admin] RBAC: Authorizing `identity:delete_project(project_id=2ffb230cff3247c79c7a3b8a03ed7fbe)` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:12:00.440524 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b7d2636-0878-4381-b243-b425103737b5 admin admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:12:00.544856 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [None req-4b7d2636-0878-4381-b243-b425103737b5 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '2ffb230cff3247c79c7a3b8a03ed7fbe', 'request_id': 'req-4b7d2636-0878-4381-b243-b425103737b5'} {{(pid=92057) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:12:00.550367 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [None req-4b7d2636-0878-4381-b243-b425103737b5 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '2ffb230cff3247c79c7a3b8a03ed7fbe', 'request_id': 'req-4b7d2636-0878-4381-b243-b425103737b5'} {{(pid=92057) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:12:00.633602 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [None req-4b7d2636-0878-4381-b243-b425103737b5 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '2ffb230cff3247c79c7a3b8a03ed7fbe', 'request_id': 'req-4b7d2636-0878-4381-b243-b425103737b5'} {{(pid=92057) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:12:00.645637 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [None req-4b7d2636-0878-4381-b243-b425103737b5 admin admin] The token cache is being invalidate because project 2ffb230cff3247c79c7a3b8a03ed7fbe was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=92057) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} May 23 13:12:00.645953 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [None req-4b7d2636-0878-4381-b243-b425103737b5 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=92057) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:12:00.646571 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [None req-4b7d2636-0878-4381-b243-b425103737b5 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=92057) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:12:00.648364 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 1353/2692] 10.222.0.30 () {58 vars in 1232 bytes} [Mon May 23 13:12:00 2022] DELETE /identity/v3/projects/2ffb230cff3247c79c7a3b8a03ed7fbe => generated 0 bytes in 246 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) May 23 13:12:00.652348 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-836f5c18-f1d6-425f-a8f9-76d8f725f32b service neutron] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:12:00.657302 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 13:12:00.663242 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-836f5c18-f1d6-425f-a8f9-76d8f725f32b service neutron] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:12:00.721540 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9101f5c1-769f-4260-b128-66f5f4521b2c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:12:00.722384 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9101f5c1-769f-4260-b128-66f5f4521b2c admin admin] REQUEST_METHOD: `DELETE` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:12:00.722671 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9101f5c1-769f-4260-b128-66f5f4521b2c admin admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:12:00.722947 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9101f5c1-769f-4260-b128-66f5f4521b2c admin admin] PATH_INFO: `/v3/users/944a40695a1b4aae934b1248524d47f0` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:12:00.724127 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9101f5c1-769f-4260-b128-66f5f4521b2c admin admin] RBAC: Authorizing `identity:delete_user(user_id=944a40695a1b4aae934b1248524d47f0)` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:12:00.726092 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9101f5c1-769f-4260-b128-66f5f4521b2c admin admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:12:00.922665 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-9101f5c1-769f-4260-b128-66f5f4521b2c admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '944a40695a1b4aae934b1248524d47f0'} {{(pid=92056) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:12:00.923314 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-9101f5c1-769f-4260-b128-66f5f4521b2c admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '944a40695a1b4aae934b1248524d47f0'} {{(pid=92056) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:12:00.977689 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-9101f5c1-769f-4260-b128-66f5f4521b2c admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '944a40695a1b4aae934b1248524d47f0'} {{(pid=92056) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:12:00.989929 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-9101f5c1-769f-4260-b128-66f5f4521b2c admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '944a40695a1b4aae934b1248524d47f0'} {{(pid=92056) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:12:00.996852 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-9101f5c1-769f-4260-b128-66f5f4521b2c admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '944a40695a1b4aae934b1248524d47f0'} {{(pid=92056) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:12:01.003486 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 1340/2693] 10.222.0.30 () {58 vars in 1223 bytes} [Mon May 23 13:12:00 2022] DELETE /identity/v3/users/944a40695a1b4aae934b1248524d47f0 => generated 0 bytes in 352 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) May 23 13:12:01.414488 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b7d2636-0878-4381-b243-b425103737b5 admin admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:12:01.415934 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.fernet_utils [None req-4b7d2636-0878-4381-b243-b425103737b5 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92057) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:12:01.432247 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b7d2636-0878-4381-b243-b425103737b5 admin admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:12:01.490949 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55bd935f-fbb2-4c15-bd77-48989a5d3464 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:12:01.491807 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55bd935f-fbb2-4c15-bd77-48989a5d3464 admin admin] REQUEST_METHOD: `DELETE` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:12:01.492090 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55bd935f-fbb2-4c15-bd77-48989a5d3464 admin admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:12:01.492370 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55bd935f-fbb2-4c15-bd77-48989a5d3464 admin admin] PATH_INFO: `/v3/projects/80f999adea41454198f57871becedc41` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:12:01.493551 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55bd935f-fbb2-4c15-bd77-48989a5d3464 admin admin] RBAC: Authorizing `identity:delete_project(project_id=80f999adea41454198f57871becedc41)` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:12:01.495524 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55bd935f-fbb2-4c15-bd77-48989a5d3464 admin admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:12:01.578828 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=91,keystone:DELETE=8,keystone:INSERT=1 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 13:12:01.595517 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [None req-55bd935f-fbb2-4c15-bd77-48989a5d3464 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '80f999adea41454198f57871becedc41', 'request_id': 'req-55bd935f-fbb2-4c15-bd77-48989a5d3464'} {{(pid=92057) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:12:01.601074 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [None req-55bd935f-fbb2-4c15-bd77-48989a5d3464 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '80f999adea41454198f57871becedc41', 'request_id': 'req-55bd935f-fbb2-4c15-bd77-48989a5d3464'} {{(pid=92057) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:12:01.640000 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [None req-55bd935f-fbb2-4c15-bd77-48989a5d3464 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '80f999adea41454198f57871becedc41', 'request_id': 'req-55bd935f-fbb2-4c15-bd77-48989a5d3464'} {{(pid=92057) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:12:01.654551 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [None req-55bd935f-fbb2-4c15-bd77-48989a5d3464 admin admin] The token cache is being invalidate because project 80f999adea41454198f57871becedc41 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=92057) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} May 23 13:12:01.654821 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [None req-55bd935f-fbb2-4c15-bd77-48989a5d3464 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=92057) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:12:01.658825 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [None req-55bd935f-fbb2-4c15-bd77-48989a5d3464 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=92057) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:12:01.658825 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 1354/2694] 10.222.0.30 () {58 vars in 1232 bytes} [Mon May 23 13:12:01 2022] DELETE /identity/v3/projects/80f999adea41454198f57871becedc41 => generated 0 bytes in 243 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) May 23 13:12:01.662184 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c631a338-5d1d-4e0b-a400-0ec402ce4622 None None] REQUEST_METHOD: `POST` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:12:01.662698 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c631a338-5d1d-4e0b-a400-0ec402ce4622 None None] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:12:01.662810 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c631a338-5d1d-4e0b-a400-0ec402ce4622 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:12:01.714341 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.auth.core [None req-c631a338-5d1d-4e0b-a400-0ec402ce4622 None None] MFA Rules not processed for user `0f7119f8fa5c4445977671140ded91a4`. Rule list: `[]` (Enabled: `True`). {{(pid=92056) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 13:12:01.715538 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.fernet_utils [None req-c631a338-5d1d-4e0b-a400-0ec402ce4622 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92056) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:12:01.764933 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 13:12:01.775007 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 1341/2695] 10.222.0.30 () {58 vars in 957 bytes} [Mon May 23 13:12:01 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 114 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) May 23 13:12:01.782902 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2e6df01-8e59-4d7a-8b85-a76d7a40612b None None] REQUEST_METHOD: `POST` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:12:01.783195 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2e6df01-8e59-4d7a-8b85-a76d7a40612b None None] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:12:01.783475 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2e6df01-8e59-4d7a-8b85-a76d7a40612b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:12:01.801895 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.auth.core [None req-f2e6df01-8e59-4d7a-8b85-a76d7a40612b None None] MFA Rules not processed for user `0f7119f8fa5c4445977671140ded91a4`. Rule list: `[]` (Enabled: `True`). {{(pid=92057) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 13:12:01.803092 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.fernet_utils [None req-f2e6df01-8e59-4d7a-8b85-a76d7a40612b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92057) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:12:01.811090 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 1355/2696] 10.222.0.30 () {58 vars in 957 bytes} [Mon May 23 13:12:01 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 29 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) May 23 13:12:01.816141 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c631a338-5d1d-4e0b-a400-0ec402ce4622 None None] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:12:01.826198 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c631a338-5d1d-4e0b-a400-0ec402ce4622 None None] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:12:01.831096 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1de085ac-cce4-4305-8cf9-5251cef07954 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:12:01.831768 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1de085ac-cce4-4305-8cf9-5251cef07954 admin admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:12:01.832075 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1de085ac-cce4-4305-8cf9-5251cef07954 admin admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:12:01.832352 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1de085ac-cce4-4305-8cf9-5251cef07954 admin admin] PATH_INFO: `/v3/domains` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:12:01.833071 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1de085ac-cce4-4305-8cf9-5251cef07954 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=92056) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} May 23 13:12:01.833386 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1de085ac-cce4-4305-8cf9-5251cef07954 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:12:01.835008 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1de085ac-cce4-4305-8cf9-5251cef07954 admin admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:12:01.847062 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 1342/2697] 10.222.0.30 () {58 vars in 1152 bytes} [Mon May 23 13:12:01 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) May 23 13:12:01.850956 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2e6df01-8e59-4d7a-8b85-a76d7a40612b None None] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:12:01.860032 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2e6df01-8e59-4d7a-8b85-a76d7a40612b None None] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:12:01.866561 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-debfa89d-2930-486f-8e0e-da5eefd47703 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:12:01.867647 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-debfa89d-2930-486f-8e0e-da5eefd47703 admin admin] REQUEST_METHOD: `POST` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:12:01.868137 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-debfa89d-2930-486f-8e0e-da5eefd47703 admin admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:12:01.868618 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-debfa89d-2930-486f-8e0e-da5eefd47703 admin admin] PATH_INFO: `/v3/projects` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:12:01.869910 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-debfa89d-2930-486f-8e0e-da5eefd47703 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:12:01.873903 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-debfa89d-2930-486f-8e0e-da5eefd47703 admin admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:12:01.972318 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 1356/2698] 10.222.0.30 () {60 vars in 1152 bytes} [Mon May 23 13:12:01 2022] POST /identity/v3/projects => generated 377 bytes in 122 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) May 23 13:12:01.976447 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1de085ac-cce4-4305-8cf9-5251cef07954 admin admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:12:01.986957 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1de085ac-cce4-4305-8cf9-5251cef07954 admin admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:12:02.043102 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-455930e3-3e90-4ef5-9b47-8d00bd088748 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:12:02.043760 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-455930e3-3e90-4ef5-9b47-8d00bd088748 admin admin] REQUEST_METHOD: `POST` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:12:02.044041 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-455930e3-3e90-4ef5-9b47-8d00bd088748 admin admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:12:02.044347 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-455930e3-3e90-4ef5-9b47-8d00bd088748 admin admin] PATH_INFO: `/v3/users` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:12:02.045038 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-455930e3-3e90-4ef5-9b47-8d00bd088748 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:12:02.047145 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-455930e3-3e90-4ef5-9b47-8d00bd088748 admin admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:12:02.119121 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 1343/2699] 10.222.0.30 () {60 vars in 1143 bytes} [Mon May 23 13:12:01 2022] POST /identity/v3/users => generated 289 bytes in 143 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) May 23 13:12:02.123204 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-debfa89d-2930-486f-8e0e-da5eefd47703 admin admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:12:02.134398 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-debfa89d-2930-486f-8e0e-da5eefd47703 admin admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:12:02.139056 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e5ec8a0-9cce-4c26-9798-c1705c60cbce admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:12:02.139710 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e5ec8a0-9cce-4c26-9798-c1705c60cbce admin admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:12:02.139998 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e5ec8a0-9cce-4c26-9798-c1705c60cbce admin admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:12:02.140277 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e5ec8a0-9cce-4c26-9798-c1705c60cbce admin admin] PATH_INFO: `/v3/roles` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:12:02.140999 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e5ec8a0-9cce-4c26-9798-c1705c60cbce admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:12:02.142779 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e5ec8a0-9cce-4c26-9798-c1705c60cbce admin admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:12:02.154583 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 1357/2700] 10.222.0.30 () {58 vars in 1121 bytes} [Mon May 23 13:12:02 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) May 23 13:12:02.161421 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-455930e3-3e90-4ef5-9b47-8d00bd088748 admin admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:12:02.171409 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-455930e3-3e90-4ef5-9b47-8d00bd088748 admin admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:12:02.176760 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c4400b8-52de-4b5a-a300-0908d9b26b5b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:12:02.177403 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c4400b8-52de-4b5a-a300-0908d9b26b5b admin admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:12:02.177686 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c4400b8-52de-4b5a-a300-0908d9b26b5b admin admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:12:02.177991 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c4400b8-52de-4b5a-a300-0908d9b26b5b admin admin] PATH_INFO: `/v3/roles` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:12:02.178718 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c4400b8-52de-4b5a-a300-0908d9b26b5b admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:12:02.180299 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c4400b8-52de-4b5a-a300-0908d9b26b5b admin admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:12:02.190419 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 1344/2701] 10.222.0.30 () {58 vars in 1121 bytes} [Mon May 23 13:12:02 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) May 23 13:12:02.194697 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e5ec8a0-9cce-4c26-9798-c1705c60cbce admin admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:12:02.204862 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e5ec8a0-9cce-4c26-9798-c1705c60cbce admin admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:12:02.209584 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42210518-9fbf-405e-83af-3f165a64aa82 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:12:02.210489 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42210518-9fbf-405e-83af-3f165a64aa82 admin admin] REQUEST_METHOD: `PUT` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:12:02.210777 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42210518-9fbf-405e-83af-3f165a64aa82 admin admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:12:02.211106 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42210518-9fbf-405e-83af-3f165a64aa82 admin admin] PATH_INFO: `/v3/projects/6d45f708c06b46b29ec24bf5076a8662/users/466293b17aa2478daafee579b9b0a69d/roles/80fb22507c9a4ff09d44d8f12f6fff1a` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:12:02.226315 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-42210518-9fbf-405e-83af-3f165a64aa82 admin admin] RBAC: Authorizing `identity:create_grant(project_id=6d45f708c06b46b29ec24bf5076a8662, user_id=466293b17aa2478daafee579b9b0a69d, role_id=80fb22507c9a4ff09d44d8f12f6fff1a)` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:12:02.228947 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-42210518-9fbf-405e-83af-3f165a64aa82 admin admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:12:02.314883 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 1358/2702] 10.222.0.30 () {60 vars in 1482 bytes} [Mon May 23 13:12:02 2022] PUT /identity/v3/projects/6d45f708c06b46b29ec24bf5076a8662/users/466293b17aa2478daafee579b9b0a69d/roles/80fb22507c9a4ff09d44d8f12f6fff1a => generated 0 bytes in 121 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) May 23 13:12:02.326472 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b620b19-fe03-4e40-8a5a-a021d0628673 None None] REQUEST_METHOD: `POST` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:12:02.326757 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b620b19-fe03-4e40-8a5a-a021d0628673 None None] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:12:02.327059 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b620b19-fe03-4e40-8a5a-a021d0628673 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:12:02.354216 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.auth.core [None req-7b620b19-fe03-4e40-8a5a-a021d0628673 None None] MFA Rules not processed for user `466293b17aa2478daafee579b9b0a69d`. Rule list: `[]` (Enabled: `True`). {{(pid=92056) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 13:12:02.355492 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.fernet_utils [None req-7b620b19-fe03-4e40-8a5a-a021d0628673 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92056) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:12:02.418819 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 1345/2703] 10.222.0.30 () {58 vars in 957 bytes} [Mon May 23 13:12:02 2022] POST /identity/v3/auth/tokens => generated 3244 bytes in 94 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) May 23 13:12:02.424049 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42210518-9fbf-405e-83af-3f165a64aa82 admin admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:12:02.484069 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42210518-9fbf-405e-83af-3f165a64aa82 admin admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:12:02.489522 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3773acd9-e6f7-415d-9bb8-0272fdaa2f96 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:12:02.490203 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3773acd9-e6f7-415d-9bb8-0272fdaa2f96 admin admin] REQUEST_METHOD: `POST` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:12:02.490488 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3773acd9-e6f7-415d-9bb8-0272fdaa2f96 admin admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:12:02.490763 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3773acd9-e6f7-415d-9bb8-0272fdaa2f96 admin admin] PATH_INFO: `/v3/projects` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:12:02.491448 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3773acd9-e6f7-415d-9bb8-0272fdaa2f96 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:12:02.493339 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3773acd9-e6f7-415d-9bb8-0272fdaa2f96 admin admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:12:02.588982 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 1359/2704] 10.222.0.30 () {60 vars in 1152 bytes} [Mon May 23 13:12:02 2022] POST /identity/v3/projects => generated 375 bytes in 163 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) May 23 13:12:02.590546 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b620b19-fe03-4e40-8a5a-a021d0628673 None None] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:12:02.601687 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b620b19-fe03-4e40-8a5a-a021d0628673 None None] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:12:02.648419 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 13:12:02.657093 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a7a14cc-3a63-4cae-ab9d-cae7392fabd0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:12:02.658404 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a7a14cc-3a63-4cae-ab9d-cae7392fabd0 admin admin] REQUEST_METHOD: `POST` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:12:02.662623 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a7a14cc-3a63-4cae-ab9d-cae7392fabd0 admin admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:12:02.662623 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a7a14cc-3a63-4cae-ab9d-cae7392fabd0 admin admin] PATH_INFO: `/v3/users` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:12:02.662623 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a7a14cc-3a63-4cae-ab9d-cae7392fabd0 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:12:02.662623 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a7a14cc-3a63-4cae-ab9d-cae7392fabd0 admin admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:12:02.765427 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 1346/2705] 10.222.0.30 () {60 vars in 1143 bytes} [Mon May 23 13:12:02 2022] POST /identity/v3/users => generated 294 bytes in 175 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) May 23 13:12:02.769566 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3773acd9-e6f7-415d-9bb8-0272fdaa2f96 admin admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:12:02.779989 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3773acd9-e6f7-415d-9bb8-0272fdaa2f96 admin admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:12:02.784905 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0c84c16-390e-4285-9b15-1325de9ce6ed admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:12:02.788824 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0c84c16-390e-4285-9b15-1325de9ce6ed admin admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:12:02.788824 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0c84c16-390e-4285-9b15-1325de9ce6ed admin admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:12:02.788824 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0c84c16-390e-4285-9b15-1325de9ce6ed admin admin] PATH_INFO: `/v3/roles` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:12:02.788824 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f0c84c16-390e-4285-9b15-1325de9ce6ed admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:12:02.788824 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f0c84c16-390e-4285-9b15-1325de9ce6ed admin admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:12:02.800017 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 1360/2706] 10.222.0.30 () {58 vars in 1121 bytes} [Mon May 23 13:12:02 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) May 23 13:12:02.804322 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a7a14cc-3a63-4cae-ab9d-cae7392fabd0 admin admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:12:02.815468 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a7a14cc-3a63-4cae-ab9d-cae7392fabd0 admin admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:12:02.820437 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-309d85c6-a963-4123-8772-780a6bd52fe2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:12:02.821403 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-309d85c6-a963-4123-8772-780a6bd52fe2 admin admin] REQUEST_METHOD: `PUT` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:12:02.821750 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-309d85c6-a963-4123-8772-780a6bd52fe2 admin admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:12:02.822131 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-309d85c6-a963-4123-8772-780a6bd52fe2 admin admin] PATH_INFO: `/v3/projects/7be608a3c1f64d75b6c67258bce03b80/users/3aa50e67b22947488f2b8998762b369b/roles/bc3c3c9d654940cbaa5177a6be23d1bc` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:12:02.837559 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-309d85c6-a963-4123-8772-780a6bd52fe2 admin admin] RBAC: Authorizing `identity:create_grant(project_id=7be608a3c1f64d75b6c67258bce03b80, user_id=3aa50e67b22947488f2b8998762b369b, role_id=bc3c3c9d654940cbaa5177a6be23d1bc)` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:12:02.840334 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-309d85c6-a963-4123-8772-780a6bd52fe2 admin admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:12:02.910756 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 1347/2707] 10.222.0.30 () {60 vars in 1482 bytes} [Mon May 23 13:12:02 2022] PUT /identity/v3/projects/7be608a3c1f64d75b6c67258bce03b80/users/3aa50e67b22947488f2b8998762b369b/roles/bc3c3c9d654940cbaa5177a6be23d1bc => generated 0 bytes in 107 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) May 23 13:12:02.914816 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0c84c16-390e-4285-9b15-1325de9ce6ed admin admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:12:02.925580 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0c84c16-390e-4285-9b15-1325de9ce6ed admin admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:12:02.945600 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:DELETE=6,keystone:SELECT=90,keystone:INSERT=4 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 13:12:02.981755 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6021d7d3-69c1-4870-9a26-2119f61aed61 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:12:02.982451 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6021d7d3-69c1-4870-9a26-2119f61aed61 admin admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:12:02.982749 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6021d7d3-69c1-4870-9a26-2119f61aed61 admin admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:12:02.983019 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6021d7d3-69c1-4870-9a26-2119f61aed61 admin admin] PATH_INFO: `/v3/roles` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:12:02.983751 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6021d7d3-69c1-4870-9a26-2119f61aed61 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:12:02.985345 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6021d7d3-69c1-4870-9a26-2119f61aed61 admin admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:12:02.996638 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 1361/2708] 10.222.0.30 () {58 vars in 1121 bytes} [Mon May 23 13:12:02 2022] GET /identity/v3/roles => generated 1420 bytes in 82 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) May 23 13:12:03.000703 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-309d85c6-a963-4123-8772-780a6bd52fe2 admin admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:12:03.012059 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-309d85c6-a963-4123-8772-780a6bd52fe2 admin admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:12:03.017736 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80402998-445c-4785-838e-b058dc614dd4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:12:03.018874 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80402998-445c-4785-838e-b058dc614dd4 admin admin] REQUEST_METHOD: `PUT` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:12:03.019162 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80402998-445c-4785-838e-b058dc614dd4 admin admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:12:03.019434 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80402998-445c-4785-838e-b058dc614dd4 admin admin] PATH_INFO: `/v3/projects/7be608a3c1f64d75b6c67258bce03b80/users/3aa50e67b22947488f2b8998762b369b/roles/bc3c3c9d654940cbaa5177a6be23d1bc` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:12:03.021548 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80402998-445c-4785-838e-b058dc614dd4 admin admin] RBAC: Authorizing `identity:create_grant(project_id=7be608a3c1f64d75b6c67258bce03b80, user_id=3aa50e67b22947488f2b8998762b369b, role_id=bc3c3c9d654940cbaa5177a6be23d1bc)` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:12:03.024541 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80402998-445c-4785-838e-b058dc614dd4 admin admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:12:03.043815 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 1348/2709] 10.222.0.30 () {60 vars in 1482 bytes} [Mon May 23 13:12:02 2022] PUT /identity/v3/projects/7be608a3c1f64d75b6c67258bce03b80/users/3aa50e67b22947488f2b8998762b369b/roles/bc3c3c9d654940cbaa5177a6be23d1bc => generated 0 bytes in 35 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) May 23 13:12:03.045559 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb9ca98d-3cf8-475f-b39c-f5ae6363d786 None None] REQUEST_METHOD: `POST` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:12:03.045901 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb9ca98d-3cf8-475f-b39c-f5ae6363d786 None None] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:12:03.046197 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb9ca98d-3cf8-475f-b39c-f5ae6363d786 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:12:03.073395 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.auth.core [None req-bb9ca98d-3cf8-475f-b39c-f5ae6363d786 None None] MFA Rules not processed for user `3aa50e67b22947488f2b8998762b369b`. Rule list: `[]` (Enabled: `True`). {{(pid=92057) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 13:12:03.074668 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.fernet_utils [None req-bb9ca98d-3cf8-475f-b39c-f5ae6363d786 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92057) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:12:03.144705 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 1362/2710] 10.222.0.30 () {58 vars in 957 bytes} [Mon May 23 13:12:03 2022] POST /identity/v3/auth/tokens => generated 3309 bytes in 100 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) May 23 13:12:03.155952 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80402998-445c-4785-838e-b058dc614dd4 admin admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:12:03.159225 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.fernet_utils [None req-80402998-445c-4785-838e-b058dc614dd4 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92056) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:12:03.219099 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80402998-445c-4785-838e-b058dc614dd4 admin admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:12:03.223490 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c0d9b27-0d89-4c69-819c-1a32cf79394d service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1666097f8ed545c6b0e4809c93e4775a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9c389ae65aaa4542bb5a5c4fd0bacf5b', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:12:03.224137 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c0d9b27-0d89-4c69-819c-1a32cf79394d service cinder] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:12:03.224425 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c0d9b27-0d89-4c69-819c-1a32cf79394d service cinder] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:12:03.224705 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c0d9b27-0d89-4c69-819c-1a32cf79394d service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:12:03.281962 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c0d9b27-0d89-4c69-819c-1a32cf79394d service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:12:03.284015 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c0d9b27-0d89-4c69-819c-1a32cf79394d service cinder] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:12:03.292690 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 1349/2711] 10.222.0.30 () {60 vars in 1365 bytes} [Mon May 23 13:12:03 2022] GET /identity/v3/auth/tokens => generated 3244 bytes in 137 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) May 23 13:12:05.773815 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb9ca98d-3cf8-475f-b39c-f5ae6363d786 None None] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:12:05.775216 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.fernet_utils [None req-bb9ca98d-3cf8-475f-b39c-f5ae6363d786 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92057) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:12:05.787107 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb9ca98d-3cf8-475f-b39c-f5ae6363d786 None None] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:12:05.790673 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f6323fc-98c6-45a7-88d0-45eafb952ab8 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1666097f8ed545c6b0e4809c93e4775a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9c389ae65aaa4542bb5a5c4fd0bacf5b', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:12:05.791311 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f6323fc-98c6-45a7-88d0-45eafb952ab8 service cinder] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:12:05.791593 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f6323fc-98c6-45a7-88d0-45eafb952ab8 service cinder] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:12:05.791865 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f6323fc-98c6-45a7-88d0-45eafb952ab8 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:12:05.793582 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.fernet_utils [None req-5f6323fc-98c6-45a7-88d0-45eafb952ab8 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92057) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:12:05.874809 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f6323fc-98c6-45a7-88d0-45eafb952ab8 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:12:05.876251 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f6323fc-98c6-45a7-88d0-45eafb952ab8 service cinder] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:12:05.895620 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 1363/2712] 10.222.0.30 () {60 vars in 1365 bytes} [Mon May 23 13:12:05 2022] GET /identity/v3/auth/tokens => generated 3254 bytes in 122 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) May 23 13:12:10.430627 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c0d9b27-0d89-4c69-819c-1a32cf79394d service cinder] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:12:10.439873 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c0d9b27-0d89-4c69-819c-1a32cf79394d service cinder] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:12:10.443632 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0931aaaf-739d-47d1-a435-951ace78611e service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1666097f8ed545c6b0e4809c93e4775a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9c389ae65aaa4542bb5a5c4fd0bacf5b', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:12:10.449248 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0931aaaf-739d-47d1-a435-951ace78611e service cinder] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:12:10.449248 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0931aaaf-739d-47d1-a435-951ace78611e service cinder] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:12:10.449248 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0931aaaf-739d-47d1-a435-951ace78611e service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:12:10.455691 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0931aaaf-739d-47d1-a435-951ace78611e service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:12:10.457122 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0931aaaf-739d-47d1-a435-951ace78611e service cinder] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:12:10.466686 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 1350/2713] 10.222.0.30 () {60 vars in 1365 bytes} [Mon May 23 13:12:10 2022] GET /identity/v3/auth/tokens => generated 3309 bytes in 37 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) May 23 13:12:13.196797 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f6323fc-98c6-45a7-88d0-45eafb952ab8 service cinder] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:12:13.198364 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.fernet_utils [None req-5f6323fc-98c6-45a7-88d0-45eafb952ab8 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92057) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:12:13.216388 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=100 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 13:12:13.255153 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f6323fc-98c6-45a7-88d0-45eafb952ab8 service cinder] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:12:13.259242 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-919ed539-0405-41f3-8452-28b36b96129e service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2eb83ca38e79449095f3a2f947bbb06a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9c389ae65aaa4542bb5a5c4fd0bacf5b', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:12:13.259880 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-919ed539-0405-41f3-8452-28b36b96129e service neutron] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:12:13.260163 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-919ed539-0405-41f3-8452-28b36b96129e service neutron] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:12:13.260439 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-919ed539-0405-41f3-8452-28b36b96129e service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:12:13.262235 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.fernet_utils [None req-919ed539-0405-41f3-8452-28b36b96129e service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92057) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:12:13.327438 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-919ed539-0405-41f3-8452-28b36b96129e service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:12:13.328894 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-919ed539-0405-41f3-8452-28b36b96129e service neutron] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:12:13.339470 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 1364/2714] 10.222.0.30 () {60 vars in 1365 bytes} [Mon May 23 13:12:13 2022] GET /identity/v3/auth/tokens => generated 3317 bytes in 143 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) May 23 13:12:20.462263 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=74,keystone:INSERT=5 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 13:12:23.335527 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=43 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 13:12:25.797363 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0931aaaf-739d-47d1-a435-951ace78611e service cinder] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:12:25.798769 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.fernet_utils [None req-0931aaaf-739d-47d1-a435-951ace78611e service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92056) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:12:25.810862 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0931aaaf-739d-47d1-a435-951ace78611e service cinder] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:12:25.814406 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-024636a8-4209-483d-af3b-1c8e173885fb service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2eb83ca38e79449095f3a2f947bbb06a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9c389ae65aaa4542bb5a5c4fd0bacf5b', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:12:25.815058 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-024636a8-4209-483d-af3b-1c8e173885fb service neutron] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:12:25.815378 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-024636a8-4209-483d-af3b-1c8e173885fb service neutron] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:12:25.815703 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-024636a8-4209-483d-af3b-1c8e173885fb service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:12:25.817380 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.fernet_utils [None req-024636a8-4209-483d-af3b-1c8e173885fb service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92056) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:12:25.880108 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-024636a8-4209-483d-af3b-1c8e173885fb service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:12:25.881539 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-024636a8-4209-483d-af3b-1c8e173885fb service neutron] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:12:25.891040 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 1351/2715] 10.222.0.30 () {60 vars in 1365 bytes} [Mon May 23 13:12:25 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 94 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) May 23 13:12:29.498214 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-919ed539-0405-41f3-8452-28b36b96129e service neutron] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:12:29.500281 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.fernet_utils [None req-919ed539-0405-41f3-8452-28b36b96129e service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92057) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:12:29.516684 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-919ed539-0405-41f3-8452-28b36b96129e service neutron] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:12:29.521216 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-edafaebe-2e60-4227-a1ce-46ab61766406 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:12:29.524109 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edafaebe-2e60-4227-a1ce-46ab61766406 admin admin] REQUEST_METHOD: `DELETE` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:12:29.524109 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edafaebe-2e60-4227-a1ce-46ab61766406 admin admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:12:29.524109 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edafaebe-2e60-4227-a1ce-46ab61766406 admin admin] PATH_INFO: `/v3/users/f78ab5496c1e4008b83fa3866601c06c` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:12:29.524758 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-edafaebe-2e60-4227-a1ce-46ab61766406 admin admin] RBAC: Authorizing `identity:delete_user(user_id=f78ab5496c1e4008b83fa3866601c06c)` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:12:29.526737 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-edafaebe-2e60-4227-a1ce-46ab61766406 admin admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:12:29.705754 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [None req-edafaebe-2e60-4227-a1ce-46ab61766406 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'f78ab5496c1e4008b83fa3866601c06c'} {{(pid=92057) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:12:29.793556 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [None req-edafaebe-2e60-4227-a1ce-46ab61766406 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'f78ab5496c1e4008b83fa3866601c06c'} {{(pid=92057) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:12:29.800880 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [None req-edafaebe-2e60-4227-a1ce-46ab61766406 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'f78ab5496c1e4008b83fa3866601c06c'} {{(pid=92057) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:12:29.801504 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [None req-edafaebe-2e60-4227-a1ce-46ab61766406 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'f78ab5496c1e4008b83fa3866601c06c'} {{(pid=92057) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:12:29.814368 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [None req-edafaebe-2e60-4227-a1ce-46ab61766406 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'f78ab5496c1e4008b83fa3866601c06c'} {{(pid=92057) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:12:29.820488 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 1365/2716] 10.222.0.30 () {58 vars in 1223 bytes} [Mon May 23 13:12:29 2022] DELETE /identity/v3/users/f78ab5496c1e4008b83fa3866601c06c => generated 0 bytes in 323 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) May 23 13:12:30.068680 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-024636a8-4209-483d-af3b-1c8e173885fb service neutron] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:12:30.079290 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-024636a8-4209-483d-af3b-1c8e173885fb service neutron] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:12:30.137815 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7dd3ec04-a10d-4b9c-94f4-1269eab3309d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:12:30.138699 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7dd3ec04-a10d-4b9c-94f4-1269eab3309d admin admin] REQUEST_METHOD: `DELETE` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:12:30.138988 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7dd3ec04-a10d-4b9c-94f4-1269eab3309d admin admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:12:30.139266 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7dd3ec04-a10d-4b9c-94f4-1269eab3309d admin admin] PATH_INFO: `/v3/projects/5b4d791d4aca45429094aa0fea349041` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:12:30.140449 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7dd3ec04-a10d-4b9c-94f4-1269eab3309d admin admin] RBAC: Authorizing `identity:delete_project(project_id=5b4d791d4aca45429094aa0fea349041)` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:12:30.142420 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7dd3ec04-a10d-4b9c-94f4-1269eab3309d admin admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:12:30.285868 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-7dd3ec04-a10d-4b9c-94f4-1269eab3309d admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '5b4d791d4aca45429094aa0fea349041', 'request_id': 'req-7dd3ec04-a10d-4b9c-94f4-1269eab3309d'} {{(pid=92056) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:12:30.376740 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-7dd3ec04-a10d-4b9c-94f4-1269eab3309d admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '5b4d791d4aca45429094aa0fea349041', 'request_id': 'req-7dd3ec04-a10d-4b9c-94f4-1269eab3309d'} {{(pid=92056) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:12:30.383653 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-7dd3ec04-a10d-4b9c-94f4-1269eab3309d admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '5b4d791d4aca45429094aa0fea349041', 'request_id': 'req-7dd3ec04-a10d-4b9c-94f4-1269eab3309d'} {{(pid=92056) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:12:30.395347 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-7dd3ec04-a10d-4b9c-94f4-1269eab3309d admin admin] The token cache is being invalidate because project 5b4d791d4aca45429094aa0fea349041 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=92056) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} May 23 13:12:30.395674 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-7dd3ec04-a10d-4b9c-94f4-1269eab3309d admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=92056) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:12:30.396308 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-7dd3ec04-a10d-4b9c-94f4-1269eab3309d admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=92056) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:12:30.398734 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 1352/2717] 10.222.0.30 () {58 vars in 1232 bytes} [Mon May 23 13:12:30 2022] DELETE /identity/v3/projects/5b4d791d4aca45429094aa0fea349041 => generated 0 bytes in 331 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) May 23 13:12:30.403611 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b7b275b-62e4-4f8f-9096-515a92fcc793 None None] REQUEST_METHOD: `POST` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:12:30.403942 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b7b275b-62e4-4f8f-9096-515a92fcc793 None None] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:12:30.404217 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b7b275b-62e4-4f8f-9096-515a92fcc793 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:12:30.424675 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.auth.core [None req-2b7b275b-62e4-4f8f-9096-515a92fcc793 None None] MFA Rules not processed for user `0f7119f8fa5c4445977671140ded91a4`. Rule list: `[]` (Enabled: `True`). {{(pid=92057) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 13:12:30.433818 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.fernet_utils [None req-2b7b275b-62e4-4f8f-9096-515a92fcc793 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92057) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:12:30.485743 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 1366/2718] 10.222.0.30 () {58 vars in 957 bytes} [Mon May 23 13:12:30 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 83 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) May 23 13:12:30.493756 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-684dd037-634c-4ae2-a78a-5e5f7aecdd7f None None] REQUEST_METHOD: `POST` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:12:30.494068 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-684dd037-634c-4ae2-a78a-5e5f7aecdd7f None None] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:12:30.494373 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-684dd037-634c-4ae2-a78a-5e5f7aecdd7f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:12:30.514000 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.auth.core [None req-684dd037-634c-4ae2-a78a-5e5f7aecdd7f None None] MFA Rules not processed for user `0f7119f8fa5c4445977671140ded91a4`. Rule list: `[]` (Enabled: `True`). {{(pid=92056) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 13:12:30.515221 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.fernet_utils [None req-684dd037-634c-4ae2-a78a-5e5f7aecdd7f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92056) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:12:30.524029 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 1353/2719] 10.222.0.30 () {58 vars in 957 bytes} [Mon May 23 13:12:30 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) May 23 13:12:30.528219 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b7b275b-62e4-4f8f-9096-515a92fcc793 None None] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:12:30.538669 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b7b275b-62e4-4f8f-9096-515a92fcc793 None None] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:12:30.543787 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2293b324-8c83-4c4a-8e24-04a749700332 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:12:30.544525 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2293b324-8c83-4c4a-8e24-04a749700332 admin admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:12:30.544813 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2293b324-8c83-4c4a-8e24-04a749700332 admin admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:12:30.545088 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2293b324-8c83-4c4a-8e24-04a749700332 admin admin] PATH_INFO: `/v3/domains` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:12:30.545816 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2293b324-8c83-4c4a-8e24-04a749700332 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=92057) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} May 23 13:12:30.546192 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2293b324-8c83-4c4a-8e24-04a749700332 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:12:30.547795 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2293b324-8c83-4c4a-8e24-04a749700332 admin admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:12:30.559977 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 1367/2720] 10.222.0.30 () {58 vars in 1152 bytes} [Mon May 23 13:12:30 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) May 23 13:12:30.564324 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-684dd037-634c-4ae2-a78a-5e5f7aecdd7f None None] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:12:30.574021 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-684dd037-634c-4ae2-a78a-5e5f7aecdd7f None None] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:12:30.579182 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7f6cc07-ce1e-490c-9ddd-892d73765955 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:12:30.579847 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7f6cc07-ce1e-490c-9ddd-892d73765955 admin admin] REQUEST_METHOD: `POST` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:12:30.580126 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7f6cc07-ce1e-490c-9ddd-892d73765955 admin admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:12:30.580400 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7f6cc07-ce1e-490c-9ddd-892d73765955 admin admin] PATH_INFO: `/v3/projects` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:12:30.581098 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7f6cc07-ce1e-490c-9ddd-892d73765955 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:12:30.583234 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7f6cc07-ce1e-490c-9ddd-892d73765955 admin admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:12:30.598144 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=91,keystone:DELETE=7,keystone:INSERT=2 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 13:12:30.699069 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 1354/2721] 10.222.0.30 () {60 vars in 1152 bytes} [Mon May 23 13:12:30 2022] POST /identity/v3/projects => generated 377 bytes in 135 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) May 23 13:12:30.703585 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2293b324-8c83-4c4a-8e24-04a749700332 admin admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:12:30.714173 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2293b324-8c83-4c4a-8e24-04a749700332 admin admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:12:30.772010 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69abd279-de5d-4a8b-b4c5-4e2b6bab5182 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:12:30.773585 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69abd279-de5d-4a8b-b4c5-4e2b6bab5182 admin admin] REQUEST_METHOD: `POST` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:12:30.773862 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69abd279-de5d-4a8b-b4c5-4e2b6bab5182 admin admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:12:30.774170 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69abd279-de5d-4a8b-b4c5-4e2b6bab5182 admin admin] PATH_INFO: `/v3/users` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:12:30.774849 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69abd279-de5d-4a8b-b4c5-4e2b6bab5182 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:12:30.776747 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69abd279-de5d-4a8b-b4c5-4e2b6bab5182 admin admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:12:30.783472 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 13:12:30.855862 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 1368/2722] 10.222.0.30 () {60 vars in 1143 bytes} [Mon May 23 13:12:30 2022] POST /identity/v3/users => generated 289 bytes in 153 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) May 23 13:12:30.859967 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7f6cc07-ce1e-490c-9ddd-892d73765955 admin admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:12:30.870768 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7f6cc07-ce1e-490c-9ddd-892d73765955 admin admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:12:30.875296 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1880664-444a-485b-8f90-ef839a7c704b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:12:30.875940 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1880664-444a-485b-8f90-ef839a7c704b admin admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:12:30.876221 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1880664-444a-485b-8f90-ef839a7c704b admin admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:12:30.876496 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1880664-444a-485b-8f90-ef839a7c704b admin admin] PATH_INFO: `/v3/roles` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:12:30.877212 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1880664-444a-485b-8f90-ef839a7c704b admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:12:30.878854 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1880664-444a-485b-8f90-ef839a7c704b admin admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:12:30.889715 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 1355/2723] 10.222.0.30 () {58 vars in 1121 bytes} [Mon May 23 13:12:30 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) May 23 13:12:30.893583 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69abd279-de5d-4a8b-b4c5-4e2b6bab5182 admin admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:12:30.904487 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69abd279-de5d-4a8b-b4c5-4e2b6bab5182 admin admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:12:30.909188 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-089b0ee0-5715-4d7d-9e86-8b0023de0175 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:12:30.909891 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-089b0ee0-5715-4d7d-9e86-8b0023de0175 admin admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:12:30.910207 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-089b0ee0-5715-4d7d-9e86-8b0023de0175 admin admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:12:30.910487 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-089b0ee0-5715-4d7d-9e86-8b0023de0175 admin admin] PATH_INFO: `/v3/roles` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:12:30.911325 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-089b0ee0-5715-4d7d-9e86-8b0023de0175 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:12:30.913466 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-089b0ee0-5715-4d7d-9e86-8b0023de0175 admin admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:12:30.923635 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 1369/2724] 10.222.0.30 () {58 vars in 1121 bytes} [Mon May 23 13:12:30 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) May 23 13:12:30.927749 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1880664-444a-485b-8f90-ef839a7c704b admin admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:12:30.938484 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1880664-444a-485b-8f90-ef839a7c704b admin admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:12:30.943171 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13dab6e4-7ec9-4505-8c63-5417417c5a3d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:12:30.944672 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13dab6e4-7ec9-4505-8c63-5417417c5a3d admin admin] REQUEST_METHOD: `PUT` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:12:30.945092 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13dab6e4-7ec9-4505-8c63-5417417c5a3d admin admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:12:30.945253 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13dab6e4-7ec9-4505-8c63-5417417c5a3d admin admin] PATH_INFO: `/v3/projects/11bdfcfe4dbb42d3bb07585d85bff52a/users/fe5a3e6e6a0b4509b92cdcd54da0f4fe/roles/80fb22507c9a4ff09d44d8f12f6fff1a` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:12:30.960156 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13dab6e4-7ec9-4505-8c63-5417417c5a3d admin admin] RBAC: Authorizing `identity:create_grant(project_id=11bdfcfe4dbb42d3bb07585d85bff52a, user_id=fe5a3e6e6a0b4509b92cdcd54da0f4fe, role_id=80fb22507c9a4ff09d44d8f12f6fff1a)` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:12:30.962835 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13dab6e4-7ec9-4505-8c63-5417417c5a3d admin admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:12:31.061824 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 1356/2725] 10.222.0.30 () {60 vars in 1482 bytes} [Mon May 23 13:12:30 2022] PUT /identity/v3/projects/11bdfcfe4dbb42d3bb07585d85bff52a/users/fe5a3e6e6a0b4509b92cdcd54da0f4fe/roles/80fb22507c9a4ff09d44d8f12f6fff1a => generated 0 bytes in 135 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) May 23 13:12:31.066942 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd74b57d-4096-4030-8ba8-68c2575cdca5 None None] REQUEST_METHOD: `POST` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:12:31.067259 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd74b57d-4096-4030-8ba8-68c2575cdca5 None None] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:12:31.067544 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd74b57d-4096-4030-8ba8-68c2575cdca5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:12:31.086280 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.auth.core [None req-fd74b57d-4096-4030-8ba8-68c2575cdca5 None None] MFA Rules not processed for user `0f7119f8fa5c4445977671140ded91a4`. Rule list: `[]` (Enabled: `True`). {{(pid=92057) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 13:12:31.087570 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.fernet_utils [None req-fd74b57d-4096-4030-8ba8-68c2575cdca5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92057) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:12:31.145148 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 1370/2726] 10.222.0.30 () {58 vars in 957 bytes} [Mon May 23 13:12:31 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 79 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) May 23 13:12:31.154923 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13dab6e4-7ec9-4505-8c63-5417417c5a3d admin admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:12:31.158980 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.fernet_utils [None req-13dab6e4-7ec9-4505-8c63-5417417c5a3d admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92056) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:12:31.212935 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13dab6e4-7ec9-4505-8c63-5417417c5a3d admin admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:12:31.216571 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-898a92a6-bd56-484c-aca2-90eee5f551af service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2eb83ca38e79449095f3a2f947bbb06a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9c389ae65aaa4542bb5a5c4fd0bacf5b', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:12:31.217204 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-898a92a6-bd56-484c-aca2-90eee5f551af service neutron] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:12:31.217485 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-898a92a6-bd56-484c-aca2-90eee5f551af service neutron] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:12:31.217762 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-898a92a6-bd56-484c-aca2-90eee5f551af service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:12:31.227085 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-898a92a6-bd56-484c-aca2-90eee5f551af service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:12:31.228499 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-898a92a6-bd56-484c-aca2-90eee5f551af service neutron] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:12:31.238185 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 1357/2727] 10.222.0.30 () {60 vars in 1365 bytes} [Mon May 23 13:12:31 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 84 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) May 23 13:12:31.315601 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd74b57d-4096-4030-8ba8-68c2575cdca5 None None] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:12:31.325637 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd74b57d-4096-4030-8ba8-68c2575cdca5 None None] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:12:31.329235 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c521565-c8b6-425e-8ffb-d9899dcfc96d service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2eb83ca38e79449095f3a2f947bbb06a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9c389ae65aaa4542bb5a5c4fd0bacf5b', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:12:31.329861 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c521565-c8b6-425e-8ffb-d9899dcfc96d service neutron] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:12:31.336229 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c521565-c8b6-425e-8ffb-d9899dcfc96d service neutron] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:12:31.336229 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c521565-c8b6-425e-8ffb-d9899dcfc96d service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:12:31.336229 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.fernet_utils [None req-5c521565-c8b6-425e-8ffb-d9899dcfc96d service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92057) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:12:31.419081 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c521565-c8b6-425e-8ffb-d9899dcfc96d service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:12:31.420633 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c521565-c8b6-425e-8ffb-d9899dcfc96d service neutron] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:12:31.440345 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 1371/2728] 10.222.0.30 () {60 vars in 1365 bytes} [Mon May 23 13:12:31 2022] GET /identity/v3/auth/tokens => generated 3246 bytes in 125 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) May 23 13:12:32.569055 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-898a92a6-bd56-484c-aca2-90eee5f551af service neutron] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:12:32.571106 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.fernet_utils [None req-898a92a6-bd56-484c-aca2-90eee5f551af service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92056) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:12:32.699743 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-898a92a6-bd56-484c-aca2-90eee5f551af service neutron] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:12:32.703442 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d36b6a9-6acf-4f1e-8a22-112dc206b132 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cbe9931d94d44cc7ace0e83a7e5015fc', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9c389ae65aaa4542bb5a5c4fd0bacf5b', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'member', 'reader', 'service', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:12:32.704548 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d36b6a9-6acf-4f1e-8a22-112dc206b132 service nova] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:12:32.704548 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d36b6a9-6acf-4f1e-8a22-112dc206b132 service nova] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:12:32.712496 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d36b6a9-6acf-4f1e-8a22-112dc206b132 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:12:32.715001 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.fernet_utils [None req-9d36b6a9-6acf-4f1e-8a22-112dc206b132 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92056) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:12:32.732324 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d36b6a9-6acf-4f1e-8a22-112dc206b132 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:12:32.734173 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d36b6a9-6acf-4f1e-8a22-112dc206b132 service nova] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:12:32.746516 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 1358/2729] 10.222.0.30 () {60 vars in 1365 bytes} [Mon May 23 13:12:32 2022] GET /identity/v3/auth/tokens => generated 3362 bytes in 178 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) May 23 13:12:37.054856 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eae4f8d2-24d2-4236-9e6d-52b20c060a6e None None] REQUEST_METHOD: `POST` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:12:37.055392 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eae4f8d2-24d2-4236-9e6d-52b20c060a6e None None] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:12:37.055719 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eae4f8d2-24d2-4236-9e6d-52b20c060a6e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:12:37.086167 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.auth.core [None req-eae4f8d2-24d2-4236-9e6d-52b20c060a6e None None] MFA Rules not processed for user `fe5a3e6e6a0b4509b92cdcd54da0f4fe`. Rule list: `[]` (Enabled: `True`). {{(pid=92057) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 13:12:37.087476 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.fernet_utils [None req-eae4f8d2-24d2-4236-9e6d-52b20c060a6e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92057) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:12:37.131524 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:INSERT=3,keystone:SELECT=97 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 13:12:37.158738 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 1372/2730] 10.222.0.30 () {58 vars in 957 bytes} [Mon May 23 13:12:37 2022] POST /identity/v3/auth/tokens => generated 3244 bytes in 105 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) May 23 13:12:37.175479 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d36b6a9-6acf-4f1e-8a22-112dc206b132 service nova] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:12:37.176960 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.fernet_utils [None req-9d36b6a9-6acf-4f1e-8a22-112dc206b132 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92056) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:12:37.191749 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d36b6a9-6acf-4f1e-8a22-112dc206b132 service nova] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:12:37.197743 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b0acb0f-ef34-4b3b-9848-033e4e9b307d service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cbe9931d94d44cc7ace0e83a7e5015fc', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9c389ae65aaa4542bb5a5c4fd0bacf5b', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'member', 'reader', 'service', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:12:37.198486 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b0acb0f-ef34-4b3b-9848-033e4e9b307d service nova] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:12:37.198778 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b0acb0f-ef34-4b3b-9848-033e4e9b307d service nova] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:12:37.199056 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b0acb0f-ef34-4b3b-9848-033e4e9b307d service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:12:37.207937 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b0acb0f-ef34-4b3b-9848-033e4e9b307d service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:12:37.210318 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b0acb0f-ef34-4b3b-9848-033e4e9b307d service nova] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:12:37.220558 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 1359/2731] 10.222.0.30 () {60 vars in 1365 bytes} [Mon May 23 13:12:37 2022] GET /identity/v3/auth/tokens => generated 3244 bytes in 46 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) May 23 13:12:39.511889 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2f50813-5815-4834-b9fc-2965d87a412f None None] REQUEST_METHOD: `POST` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:12:39.512451 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2f50813-5815-4834-b9fc-2965d87a412f None None] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:12:39.512565 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2f50813-5815-4834-b9fc-2965d87a412f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:12:39.538227 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.auth.core [None req-b2f50813-5815-4834-b9fc-2965d87a412f None None] MFA Rules not processed for user `fe5a3e6e6a0b4509b92cdcd54da0f4fe`. Rule list: `[]` (Enabled: `True`). {{(pid=92057) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 13:12:39.539751 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.fernet_utils [None req-b2f50813-5815-4834-b9fc-2965d87a412f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92057) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:12:39.547515 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 1373/2732] 10.222.0.30 () {58 vars in 957 bytes} [Mon May 23 13:12:39 2022] POST /identity/v3/auth/tokens => generated 3244 bytes in 37 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) May 23 13:12:39.867449 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b0acb0f-ef34-4b3b-9848-033e4e9b307d service nova] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:12:39.869546 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.fernet_utils [None req-4b0acb0f-ef34-4b3b-9848-033e4e9b307d service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92056) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:12:39.881519 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b0acb0f-ef34-4b3b-9848-033e4e9b307d service nova] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:12:39.885723 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f97ec15d-790d-4a3b-b5c0-64c0ec2b367e service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2eb83ca38e79449095f3a2f947bbb06a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9c389ae65aaa4542bb5a5c4fd0bacf5b', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:12:39.886481 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f97ec15d-790d-4a3b-b5c0-64c0ec2b367e service neutron] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:12:39.886831 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f97ec15d-790d-4a3b-b5c0-64c0ec2b367e service neutron] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:12:39.887175 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f97ec15d-790d-4a3b-b5c0-64c0ec2b367e service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:12:39.888986 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.fernet_utils [None req-f97ec15d-790d-4a3b-b5c0-64c0ec2b367e service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92056) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:12:39.899061 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f97ec15d-790d-4a3b-b5c0-64c0ec2b367e service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:12:39.900710 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f97ec15d-790d-4a3b-b5c0-64c0ec2b367e service neutron] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:12:39.909114 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 1360/2733] 10.222.0.30 () {60 vars in 1365 bytes} [Mon May 23 13:12:39 2022] GET /identity/v3/auth/tokens => generated 3111 bytes in 42 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) May 23 13:12:45.267587 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2f50813-5815-4834-b9fc-2965d87a412f None None] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:12:45.269238 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.fernet_utils [None req-b2f50813-5815-4834-b9fc-2965d87a412f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92057) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:12:45.285187 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2f50813-5815-4834-b9fc-2965d87a412f None None] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:12:45.290600 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0eefe6aa-4542-4fc2-b4e8-737d59f632fe admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:12:45.291842 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0eefe6aa-4542-4fc2-b4e8-737d59f632fe admin admin] REQUEST_METHOD: `DELETE` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:12:45.292130 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0eefe6aa-4542-4fc2-b4e8-737d59f632fe admin admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:12:45.292663 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0eefe6aa-4542-4fc2-b4e8-737d59f632fe admin admin] PATH_INFO: `/v3/users/466293b17aa2478daafee579b9b0a69d` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:12:45.294128 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0eefe6aa-4542-4fc2-b4e8-737d59f632fe admin admin] RBAC: Authorizing `identity:delete_user(user_id=466293b17aa2478daafee579b9b0a69d)` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:12:45.296092 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0eefe6aa-4542-4fc2-b4e8-737d59f632fe admin admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:12:45.498799 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [None req-0eefe6aa-4542-4fc2-b4e8-737d59f632fe admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '466293b17aa2478daafee579b9b0a69d'} {{(pid=92057) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:12:45.575960 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [None req-0eefe6aa-4542-4fc2-b4e8-737d59f632fe admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '466293b17aa2478daafee579b9b0a69d'} {{(pid=92057) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:12:45.584253 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [None req-0eefe6aa-4542-4fc2-b4e8-737d59f632fe admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '466293b17aa2478daafee579b9b0a69d'} {{(pid=92057) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:12:45.586288 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [None req-0eefe6aa-4542-4fc2-b4e8-737d59f632fe admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '466293b17aa2478daafee579b9b0a69d'} {{(pid=92057) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:12:45.600129 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [None req-0eefe6aa-4542-4fc2-b4e8-737d59f632fe admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '466293b17aa2478daafee579b9b0a69d'} {{(pid=92057) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:12:45.607088 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 1374/2734] 10.222.0.30 () {58 vars in 1223 bytes} [Mon May 23 13:12:45 2022] DELETE /identity/v3/users/466293b17aa2478daafee579b9b0a69d => generated 0 bytes in 340 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) May 23 13:12:45.612134 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74c08f7f-3949-452c-9e2f-8208c53c4143 None None] REQUEST_METHOD: `POST` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:12:45.612689 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74c08f7f-3949-452c-9e2f-8208c53c4143 None None] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:12:45.612689 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74c08f7f-3949-452c-9e2f-8208c53c4143 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:12:45.631332 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.auth.core [None req-74c08f7f-3949-452c-9e2f-8208c53c4143 None None] MFA Rules not processed for user `0f7119f8fa5c4445977671140ded91a4`. Rule list: `[]` (Enabled: `True`). {{(pid=92056) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 13:12:45.632544 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.fernet_utils [None req-74c08f7f-3949-452c-9e2f-8208c53c4143 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92056) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:12:45.651389 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 13:12:45.710348 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 1361/2735] 10.222.0.30 () {58 vars in 957 bytes} [Mon May 23 13:12:45 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 99 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) May 23 13:12:45.727997 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0eefe6aa-4542-4fc2-b4e8-737d59f632fe admin admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:12:45.733331 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.fernet_utils [None req-0eefe6aa-4542-4fc2-b4e8-737d59f632fe admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92057) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:12:45.792983 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0eefe6aa-4542-4fc2-b4e8-737d59f632fe admin admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:12:45.796663 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-860ada09-5352-4277-ba04-f058754449af service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2eb83ca38e79449095f3a2f947bbb06a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9c389ae65aaa4542bb5a5c4fd0bacf5b', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:12:45.797311 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-860ada09-5352-4277-ba04-f058754449af service neutron] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:12:45.797600 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-860ada09-5352-4277-ba04-f058754449af service neutron] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:12:45.797921 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-860ada09-5352-4277-ba04-f058754449af service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:12:45.808353 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-860ada09-5352-4277-ba04-f058754449af service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:12:45.810120 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-860ada09-5352-4277-ba04-f058754449af service neutron] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:12:45.821276 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 1375/2736] 10.222.0.30 () {60 vars in 1365 bytes} [Mon May 23 13:12:45 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 94 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) May 23 13:12:46.344425 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74c08f7f-3949-452c-9e2f-8208c53c4143 None None] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:12:46.346118 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.fernet_utils [None req-74c08f7f-3949-452c-9e2f-8208c53c4143 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92056) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:12:46.376383 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74c08f7f-3949-452c-9e2f-8208c53c4143 None None] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:12:46.381696 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0ff3bd6-50d9-478c-8957-3848499cbc98 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:12:46.382837 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0ff3bd6-50d9-478c-8957-3848499cbc98 admin admin] REQUEST_METHOD: `DELETE` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:12:46.383180 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0ff3bd6-50d9-478c-8957-3848499cbc98 admin admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:12:46.383456 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0ff3bd6-50d9-478c-8957-3848499cbc98 admin admin] PATH_INFO: `/v3/projects/6d45f708c06b46b29ec24bf5076a8662` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:12:46.385866 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0ff3bd6-50d9-478c-8957-3848499cbc98 admin admin] RBAC: Authorizing `identity:delete_project(project_id=6d45f708c06b46b29ec24bf5076a8662)` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:12:46.387852 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0ff3bd6-50d9-478c-8957-3848499cbc98 admin admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:12:46.522279 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-d0ff3bd6-50d9-478c-8957-3848499cbc98 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '6d45f708c06b46b29ec24bf5076a8662', 'request_id': 'req-d0ff3bd6-50d9-478c-8957-3848499cbc98'} {{(pid=92056) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:12:46.610742 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-d0ff3bd6-50d9-478c-8957-3848499cbc98 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '6d45f708c06b46b29ec24bf5076a8662', 'request_id': 'req-d0ff3bd6-50d9-478c-8957-3848499cbc98'} {{(pid=92056) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:12:46.619761 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-d0ff3bd6-50d9-478c-8957-3848499cbc98 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '6d45f708c06b46b29ec24bf5076a8662', 'request_id': 'req-d0ff3bd6-50d9-478c-8957-3848499cbc98'} {{(pid=92056) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:12:46.637328 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-d0ff3bd6-50d9-478c-8957-3848499cbc98 admin admin] The token cache is being invalidate because project 6d45f708c06b46b29ec24bf5076a8662 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=92056) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} May 23 13:12:46.637738 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-d0ff3bd6-50d9-478c-8957-3848499cbc98 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=92056) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:12:46.638348 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-d0ff3bd6-50d9-478c-8957-3848499cbc98 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=92056) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:12:46.643648 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 1362/2737] 10.222.0.30 () {58 vars in 1232 bytes} [Mon May 23 13:12:46 2022] DELETE /identity/v3/projects/6d45f708c06b46b29ec24bf5076a8662 => generated 0 bytes in 297 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) May 23 13:12:46.644927 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-860ada09-5352-4277-ba04-f058754449af service neutron] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:12:46.656712 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-860ada09-5352-4277-ba04-f058754449af service neutron] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:12:46.727319 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 13:12:46.740777 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e69fffc7-a861-4022-9f51-10e0e5b881e3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:12:46.741777 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e69fffc7-a861-4022-9f51-10e0e5b881e3 admin admin] REQUEST_METHOD: `DELETE` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:12:46.742327 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e69fffc7-a861-4022-9f51-10e0e5b881e3 admin admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:12:46.742670 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e69fffc7-a861-4022-9f51-10e0e5b881e3 admin admin] PATH_INFO: `/v3/users/3aa50e67b22947488f2b8998762b369b` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:12:46.744134 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e69fffc7-a861-4022-9f51-10e0e5b881e3 admin admin] RBAC: Authorizing `identity:delete_user(user_id=3aa50e67b22947488f2b8998762b369b)` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:12:46.746161 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e69fffc7-a861-4022-9f51-10e0e5b881e3 admin admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:12:46.927337 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [None req-e69fffc7-a861-4022-9f51-10e0e5b881e3 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '3aa50e67b22947488f2b8998762b369b'} {{(pid=92057) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:12:47.004023 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [None req-e69fffc7-a861-4022-9f51-10e0e5b881e3 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '3aa50e67b22947488f2b8998762b369b'} {{(pid=92057) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:12:47.013094 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [None req-e69fffc7-a861-4022-9f51-10e0e5b881e3 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '3aa50e67b22947488f2b8998762b369b'} {{(pid=92057) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:12:47.013807 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [None req-e69fffc7-a861-4022-9f51-10e0e5b881e3 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '3aa50e67b22947488f2b8998762b369b'} {{(pid=92057) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:12:47.029703 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [None req-e69fffc7-a861-4022-9f51-10e0e5b881e3 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '3aa50e67b22947488f2b8998762b369b'} {{(pid=92057) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:12:47.036157 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 1376/2738] 10.222.0.30 () {58 vars in 1223 bytes} [Mon May 23 13:12:46 2022] DELETE /identity/v3/users/3aa50e67b22947488f2b8998762b369b => generated 0 bytes in 392 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) May 23 13:12:47.458298 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0ff3bd6-50d9-478c-8957-3848499cbc98 admin admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:12:47.459773 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.fernet_utils [None req-d0ff3bd6-50d9-478c-8957-3848499cbc98 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92056) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:12:47.473546 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0ff3bd6-50d9-478c-8957-3848499cbc98 admin admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:12:47.530872 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41339151-0b13-4828-a4e5-9b8e5d20c8b9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:12:47.531644 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41339151-0b13-4828-a4e5-9b8e5d20c8b9 admin admin] REQUEST_METHOD: `DELETE` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:12:47.531925 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41339151-0b13-4828-a4e5-9b8e5d20c8b9 admin admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:12:47.532198 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41339151-0b13-4828-a4e5-9b8e5d20c8b9 admin admin] PATH_INFO: `/v3/projects/7be608a3c1f64d75b6c67258bce03b80` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:12:47.533360 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-41339151-0b13-4828-a4e5-9b8e5d20c8b9 admin admin] RBAC: Authorizing `identity:delete_project(project_id=7be608a3c1f64d75b6c67258bce03b80)` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:12:47.535291 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-41339151-0b13-4828-a4e5-9b8e5d20c8b9 admin admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:12:47.690640 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=88,keystone:DELETE=11,keystone:INSERT=1 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 13:12:47.692042 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-41339151-0b13-4828-a4e5-9b8e5d20c8b9 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '7be608a3c1f64d75b6c67258bce03b80', 'request_id': 'req-41339151-0b13-4828-a4e5-9b8e5d20c8b9'} {{(pid=92056) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:12:47.765013 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-41339151-0b13-4828-a4e5-9b8e5d20c8b9 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '7be608a3c1f64d75b6c67258bce03b80', 'request_id': 'req-41339151-0b13-4828-a4e5-9b8e5d20c8b9'} {{(pid=92056) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:12:47.772306 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-41339151-0b13-4828-a4e5-9b8e5d20c8b9 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '7be608a3c1f64d75b6c67258bce03b80', 'request_id': 'req-41339151-0b13-4828-a4e5-9b8e5d20c8b9'} {{(pid=92056) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:12:47.785075 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-41339151-0b13-4828-a4e5-9b8e5d20c8b9 admin admin] The token cache is being invalidate because project 7be608a3c1f64d75b6c67258bce03b80 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=92056) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} May 23 13:12:47.785298 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-41339151-0b13-4828-a4e5-9b8e5d20c8b9 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=92056) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:12:47.785987 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-41339151-0b13-4828-a4e5-9b8e5d20c8b9 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=92056) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:12:47.787587 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 1363/2739] 10.222.0.30 () {58 vars in 1232 bytes} [Mon May 23 13:12:47 2022] DELETE /identity/v3/projects/7be608a3c1f64d75b6c67258bce03b80 => generated 0 bytes in 330 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) May 23 13:12:47.793013 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37037f46-12c6-49f0-8a8f-0a3ef2b34150 None None] REQUEST_METHOD: `POST` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:12:47.793558 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37037f46-12c6-49f0-8a8f-0a3ef2b34150 None None] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:12:47.793690 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37037f46-12c6-49f0-8a8f-0a3ef2b34150 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:12:47.812238 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.auth.core [None req-37037f46-12c6-49f0-8a8f-0a3ef2b34150 None None] MFA Rules not processed for user `0f7119f8fa5c4445977671140ded91a4`. Rule list: `[]` (Enabled: `True`). {{(pid=92057) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 13:12:47.813426 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.fernet_utils [None req-37037f46-12c6-49f0-8a8f-0a3ef2b34150 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92057) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:12:47.880518 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 1377/2740] 10.222.0.30 () {58 vars in 957 bytes} [Mon May 23 13:12:47 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 89 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) May 23 13:12:47.888703 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bce1743-77a0-4d88-96e6-f06e83df738b None None] REQUEST_METHOD: `POST` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:12:47.888989 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bce1743-77a0-4d88-96e6-f06e83df738b None None] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:12:47.889277 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bce1743-77a0-4d88-96e6-f06e83df738b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:12:47.914512 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.auth.core [None req-3bce1743-77a0-4d88-96e6-f06e83df738b None None] MFA Rules not processed for user `0f7119f8fa5c4445977671140ded91a4`. Rule list: `[]` (Enabled: `True`). {{(pid=92056) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 13:12:47.915974 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.fernet_utils [None req-3bce1743-77a0-4d88-96e6-f06e83df738b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92056) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:12:47.923906 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 1364/2741] 10.222.0.30 () {58 vars in 957 bytes} [Mon May 23 13:12:47 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 36 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) May 23 13:12:47.928120 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37037f46-12c6-49f0-8a8f-0a3ef2b34150 None None] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:12:47.937583 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37037f46-12c6-49f0-8a8f-0a3ef2b34150 None None] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:12:47.942512 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-810258a9-afb4-4fa1-9c19-e4923997e169 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:12:47.943253 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-810258a9-afb4-4fa1-9c19-e4923997e169 admin admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:12:47.943626 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-810258a9-afb4-4fa1-9c19-e4923997e169 admin admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:12:47.943964 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-810258a9-afb4-4fa1-9c19-e4923997e169 admin admin] PATH_INFO: `/v3/domains` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:12:47.944926 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-810258a9-afb4-4fa1-9c19-e4923997e169 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=92057) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} May 23 13:12:47.945292 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-810258a9-afb4-4fa1-9c19-e4923997e169 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:12:47.947675 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-810258a9-afb4-4fa1-9c19-e4923997e169 admin admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:12:47.959802 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 1378/2742] 10.222.0.30 () {58 vars in 1152 bytes} [Mon May 23 13:12:47 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) May 23 13:12:47.964518 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3bce1743-77a0-4d88-96e6-f06e83df738b None None] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:12:47.974112 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3bce1743-77a0-4d88-96e6-f06e83df738b None None] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:12:47.979453 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cef54668-e15b-4c85-ad3f-51fef684497c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:12:47.980141 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cef54668-e15b-4c85-ad3f-51fef684497c admin admin] REQUEST_METHOD: `POST` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:12:47.980427 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cef54668-e15b-4c85-ad3f-51fef684497c admin admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:12:47.980707 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cef54668-e15b-4c85-ad3f-51fef684497c admin admin] PATH_INFO: `/v3/projects` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:12:47.981435 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cef54668-e15b-4c85-ad3f-51fef684497c admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:12:47.983387 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cef54668-e15b-4c85-ad3f-51fef684497c admin admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:12:48.098329 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 1365/2743] 10.222.0.30 () {60 vars in 1152 bytes} [Mon May 23 13:12:47 2022] POST /identity/v3/projects => generated 369 bytes in 134 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) May 23 13:12:48.103278 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-810258a9-afb4-4fa1-9c19-e4923997e169 admin admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:12:48.113820 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-810258a9-afb4-4fa1-9c19-e4923997e169 admin admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:12:48.162924 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 13:12:48.172410 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ae8b329-b697-42bb-9987-70336fcc7e60 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:12:48.173087 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ae8b329-b697-42bb-9987-70336fcc7e60 admin admin] REQUEST_METHOD: `POST` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:12:48.173371 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ae8b329-b697-42bb-9987-70336fcc7e60 admin admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:12:48.173651 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ae8b329-b697-42bb-9987-70336fcc7e60 admin admin] PATH_INFO: `/v3/users` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:12:48.174370 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ae8b329-b697-42bb-9987-70336fcc7e60 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:12:48.176371 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ae8b329-b697-42bb-9987-70336fcc7e60 admin admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:12:48.301573 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 1379/2744] 10.222.0.30 () {60 vars in 1143 bytes} [Mon May 23 13:12:48 2022] POST /identity/v3/users => generated 285 bytes in 199 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) May 23 13:12:48.305607 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cef54668-e15b-4c85-ad3f-51fef684497c admin admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:12:48.316796 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cef54668-e15b-4c85-ad3f-51fef684497c admin admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:12:48.325520 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4758c751-3b94-4e5b-80b8-b86debd17667 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:12:48.325520 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4758c751-3b94-4e5b-80b8-b86debd17667 admin admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:12:48.325520 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4758c751-3b94-4e5b-80b8-b86debd17667 admin admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:12:48.325520 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4758c751-3b94-4e5b-80b8-b86debd17667 admin admin] PATH_INFO: `/v3/roles` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:12:48.325950 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4758c751-3b94-4e5b-80b8-b86debd17667 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:12:48.325950 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4758c751-3b94-4e5b-80b8-b86debd17667 admin admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:12:48.336233 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 1366/2745] 10.222.0.30 () {58 vars in 1121 bytes} [Mon May 23 13:12:48 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) May 23 13:12:48.340411 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ae8b329-b697-42bb-9987-70336fcc7e60 admin admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:12:48.350301 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ae8b329-b697-42bb-9987-70336fcc7e60 admin admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:12:48.355429 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5017f4e-0693-4c54-be79-73690b7043b0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:12:48.356080 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5017f4e-0693-4c54-be79-73690b7043b0 admin admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:12:48.356365 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5017f4e-0693-4c54-be79-73690b7043b0 admin admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:12:48.356638 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5017f4e-0693-4c54-be79-73690b7043b0 admin admin] PATH_INFO: `/v3/roles` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:12:48.357357 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5017f4e-0693-4c54-be79-73690b7043b0 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:12:48.359004 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5017f4e-0693-4c54-be79-73690b7043b0 admin admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:12:48.370456 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 1380/2746] 10.222.0.30 () {58 vars in 1121 bytes} [Mon May 23 13:12:48 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) May 23 13:12:48.374443 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4758c751-3b94-4e5b-80b8-b86debd17667 admin admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:12:48.384845 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4758c751-3b94-4e5b-80b8-b86debd17667 admin admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:12:48.389593 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc97b214-cd7d-46f2-90fb-fe41db7c34da admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:12:48.390497 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc97b214-cd7d-46f2-90fb-fe41db7c34da admin admin] REQUEST_METHOD: `PUT` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:12:48.390777 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc97b214-cd7d-46f2-90fb-fe41db7c34da admin admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:12:48.391049 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc97b214-cd7d-46f2-90fb-fe41db7c34da admin admin] PATH_INFO: `/v3/projects/567884b30ed64e7392b76d2d80c573a5/users/f665c01235df405b8cbd54cf63884b1e/roles/80fb22507c9a4ff09d44d8f12f6fff1a` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:12:48.407330 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc97b214-cd7d-46f2-90fb-fe41db7c34da admin admin] RBAC: Authorizing `identity:create_grant(project_id=567884b30ed64e7392b76d2d80c573a5, user_id=f665c01235df405b8cbd54cf63884b1e, role_id=80fb22507c9a4ff09d44d8f12f6fff1a)` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:12:48.410161 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc97b214-cd7d-46f2-90fb-fe41db7c34da admin admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:12:48.486342 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 1367/2747] 10.222.0.30 () {60 vars in 1482 bytes} [Mon May 23 13:12:48 2022] PUT /identity/v3/projects/567884b30ed64e7392b76d2d80c573a5/users/f665c01235df405b8cbd54cf63884b1e/roles/80fb22507c9a4ff09d44d8f12f6fff1a => generated 0 bytes in 112 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) May 23 13:12:48.498925 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e77d96e9-cd36-446a-a4b2-e1b3df89d73d None None] REQUEST_METHOD: `POST` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:12:48.499272 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e77d96e9-cd36-446a-a4b2-e1b3df89d73d None None] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:12:48.499549 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e77d96e9-cd36-446a-a4b2-e1b3df89d73d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:12:48.526651 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.auth.core [None req-e77d96e9-cd36-446a-a4b2-e1b3df89d73d None None] MFA Rules not processed for user `f665c01235df405b8cbd54cf63884b1e`. Rule list: `[]` (Enabled: `True`). {{(pid=92057) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 13:12:48.527855 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.fernet_utils [None req-e77d96e9-cd36-446a-a4b2-e1b3df89d73d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92057) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:12:48.591879 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 1381/2748] 10.222.0.30 () {58 vars in 957 bytes} [Mon May 23 13:12:48 2022] POST /identity/v3/auth/tokens => generated 3236 bytes in 94 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) May 23 13:12:48.596339 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc97b214-cd7d-46f2-90fb-fe41db7c34da admin admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:12:48.668806 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc97b214-cd7d-46f2-90fb-fe41db7c34da admin admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:12:48.673551 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62d1b452-a4c5-4e30-9ecf-dc07b2e96c06 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:12:48.674586 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62d1b452-a4c5-4e30-9ecf-dc07b2e96c06 admin admin] REQUEST_METHOD: `POST` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:12:48.674586 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62d1b452-a4c5-4e30-9ecf-dc07b2e96c06 admin admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:12:48.674803 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62d1b452-a4c5-4e30-9ecf-dc07b2e96c06 admin admin] PATH_INFO: `/v3/projects` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:12:48.675492 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-62d1b452-a4c5-4e30-9ecf-dc07b2e96c06 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:12:48.677379 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-62d1b452-a4c5-4e30-9ecf-dc07b2e96c06 admin admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:12:48.737834 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 1368/2749] 10.222.0.30 () {60 vars in 1152 bytes} [Mon May 23 13:12:48 2022] POST /identity/v3/projects => generated 369 bytes in 142 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) May 23 13:12:48.741919 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e77d96e9-cd36-446a-a4b2-e1b3df89d73d None None] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:12:48.753449 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e77d96e9-cd36-446a-a4b2-e1b3df89d73d None None] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:12:48.809765 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dbb51b02-8a56-4570-828d-7b2384eabf1e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:12:48.810474 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbb51b02-8a56-4570-828d-7b2384eabf1e admin admin] REQUEST_METHOD: `POST` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:12:48.810761 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbb51b02-8a56-4570-828d-7b2384eabf1e admin admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:12:48.811044 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbb51b02-8a56-4570-828d-7b2384eabf1e admin admin] PATH_INFO: `/v3/users` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:12:48.811743 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dbb51b02-8a56-4570-828d-7b2384eabf1e admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:12:48.813668 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dbb51b02-8a56-4570-828d-7b2384eabf1e admin admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:12:48.908455 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 1382/2750] 10.222.0.30 () {60 vars in 1143 bytes} [Mon May 23 13:12:48 2022] POST /identity/v3/users => generated 291 bytes in 167 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) May 23 13:12:48.912366 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62d1b452-a4c5-4e30-9ecf-dc07b2e96c06 admin admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:12:48.923446 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62d1b452-a4c5-4e30-9ecf-dc07b2e96c06 admin admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:12:48.928226 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53335189-254b-4dcd-a863-ccddcc40c765 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:12:48.930360 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53335189-254b-4dcd-a863-ccddcc40c765 admin admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:12:48.930360 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53335189-254b-4dcd-a863-ccddcc40c765 admin admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:12:48.930360 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53335189-254b-4dcd-a863-ccddcc40c765 admin admin] PATH_INFO: `/v3/roles` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:12:48.930360 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53335189-254b-4dcd-a863-ccddcc40c765 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:12:48.931980 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53335189-254b-4dcd-a863-ccddcc40c765 admin admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:12:48.942804 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 1369/2751] 10.222.0.30 () {58 vars in 1121 bytes} [Mon May 23 13:12:48 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) May 23 13:12:48.947173 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dbb51b02-8a56-4570-828d-7b2384eabf1e admin admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:12:48.956928 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dbb51b02-8a56-4570-828d-7b2384eabf1e admin admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:12:48.961567 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e036e1b4-8369-45c9-ba05-f533dd9cdb06 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:12:48.963055 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e036e1b4-8369-45c9-ba05-f533dd9cdb06 admin admin] REQUEST_METHOD: `PUT` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:12:48.963349 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e036e1b4-8369-45c9-ba05-f533dd9cdb06 admin admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:12:48.963652 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e036e1b4-8369-45c9-ba05-f533dd9cdb06 admin admin] PATH_INFO: `/v3/projects/5d889f687d5748ada80946ea7f2ee9c2/users/d8005bf77369443eb9f5f444c5223caa/roles/bc3c3c9d654940cbaa5177a6be23d1bc` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:12:48.979598 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e036e1b4-8369-45c9-ba05-f533dd9cdb06 admin admin] RBAC: Authorizing `identity:create_grant(project_id=5d889f687d5748ada80946ea7f2ee9c2, user_id=d8005bf77369443eb9f5f444c5223caa, role_id=bc3c3c9d654940cbaa5177a6be23d1bc)` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:12:48.982553 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e036e1b4-8369-45c9-ba05-f533dd9cdb06 admin admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:12:49.034354 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 1383/2752] 10.222.0.30 () {60 vars in 1482 bytes} [Mon May 23 13:12:48 2022] PUT /identity/v3/projects/5d889f687d5748ada80946ea7f2ee9c2/users/d8005bf77369443eb9f5f444c5223caa/roles/bc3c3c9d654940cbaa5177a6be23d1bc => generated 0 bytes in 88 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) May 23 13:12:49.038451 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53335189-254b-4dcd-a863-ccddcc40c765 admin admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:12:49.049238 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53335189-254b-4dcd-a863-ccddcc40c765 admin admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:12:49.097294 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=93,keystone:INSERT=4,keystone:DELETE=3 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 13:12:49.110668 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f77cef19-b8cb-45bd-9273-e390c7f207fe admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:12:49.111424 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f77cef19-b8cb-45bd-9273-e390c7f207fe admin admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:12:49.111703 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f77cef19-b8cb-45bd-9273-e390c7f207fe admin admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:12:49.111977 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f77cef19-b8cb-45bd-9273-e390c7f207fe admin admin] PATH_INFO: `/v3/roles` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:12:49.112701 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f77cef19-b8cb-45bd-9273-e390c7f207fe admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:12:49.114336 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f77cef19-b8cb-45bd-9273-e390c7f207fe admin admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:12:49.125011 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 1370/2753] 10.222.0.30 () {58 vars in 1121 bytes} [Mon May 23 13:12:49 2022] GET /identity/v3/roles => generated 1420 bytes in 87 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) May 23 13:12:49.129389 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e036e1b4-8369-45c9-ba05-f533dd9cdb06 admin admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:12:49.140320 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e036e1b4-8369-45c9-ba05-f533dd9cdb06 admin admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:12:49.145743 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c054ba1-9ec7-4078-8e07-a65c87189941 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:12:49.146687 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c054ba1-9ec7-4078-8e07-a65c87189941 admin admin] REQUEST_METHOD: `PUT` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:12:49.146977 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c054ba1-9ec7-4078-8e07-a65c87189941 admin admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:12:49.147528 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c054ba1-9ec7-4078-8e07-a65c87189941 admin admin] PATH_INFO: `/v3/projects/5d889f687d5748ada80946ea7f2ee9c2/users/d8005bf77369443eb9f5f444c5223caa/roles/bc3c3c9d654940cbaa5177a6be23d1bc` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:12:49.150084 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c054ba1-9ec7-4078-8e07-a65c87189941 admin admin] RBAC: Authorizing `identity:create_grant(project_id=5d889f687d5748ada80946ea7f2ee9c2, user_id=d8005bf77369443eb9f5f444c5223caa, role_id=bc3c3c9d654940cbaa5177a6be23d1bc)` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:12:49.152864 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c054ba1-9ec7-4078-8e07-a65c87189941 admin admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:12:49.167313 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 1384/2754] 10.222.0.30 () {60 vars in 1482 bytes} [Mon May 23 13:12:49 2022] PUT /identity/v3/projects/5d889f687d5748ada80946ea7f2ee9c2/users/d8005bf77369443eb9f5f444c5223caa/roles/bc3c3c9d654940cbaa5177a6be23d1bc => generated 0 bytes in 38 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) May 23 13:12:49.177860 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b5ceb61-0e7e-4311-bb78-2f0158cc7acd None None] REQUEST_METHOD: `POST` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:12:49.178259 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b5ceb61-0e7e-4311-bb78-2f0158cc7acd None None] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:12:49.178627 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b5ceb61-0e7e-4311-bb78-2f0158cc7acd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:12:49.206363 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.auth.core [None req-7b5ceb61-0e7e-4311-bb78-2f0158cc7acd None None] MFA Rules not processed for user `d8005bf77369443eb9f5f444c5223caa`. Rule list: `[]` (Enabled: `True`). {{(pid=92056) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 13:12:49.207659 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.fernet_utils [None req-7b5ceb61-0e7e-4311-bb78-2f0158cc7acd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92056) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:12:49.300332 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 1371/2755] 10.222.0.30 () {58 vars in 957 bytes} [Mon May 23 13:12:49 2022] POST /identity/v3/auth/tokens => generated 3303 bytes in 123 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) May 23 13:12:49.309455 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c054ba1-9ec7-4078-8e07-a65c87189941 admin admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:12:49.311394 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.fernet_utils [None req-6c054ba1-9ec7-4078-8e07-a65c87189941 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92057) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:12:49.333633 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=92,keystone:INSERT=8 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 13:12:49.396404 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c054ba1-9ec7-4078-8e07-a65c87189941 admin admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:12:49.400629 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01d4a465-9dde-47a9-947f-61de66d1d8d1 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1666097f8ed545c6b0e4809c93e4775a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9c389ae65aaa4542bb5a5c4fd0bacf5b', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:12:49.401273 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01d4a465-9dde-47a9-947f-61de66d1d8d1 service cinder] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:12:49.401556 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01d4a465-9dde-47a9-947f-61de66d1d8d1 service cinder] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:12:49.401838 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01d4a465-9dde-47a9-947f-61de66d1d8d1 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:12:49.411040 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01d4a465-9dde-47a9-947f-61de66d1d8d1 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:12:49.412482 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01d4a465-9dde-47a9-947f-61de66d1d8d1 service cinder] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:12:49.433203 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 1385/2756] 10.222.0.30 () {60 vars in 1365 bytes} [Mon May 23 13:12:49 2022] GET /identity/v3/auth/tokens => generated 3303 bytes in 115 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) May 23 13:12:49.564748 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b5ceb61-0e7e-4311-bb78-2f0158cc7acd None None] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:12:49.566211 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.fernet_utils [None req-7b5ceb61-0e7e-4311-bb78-2f0158cc7acd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92056) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:12:49.579383 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b5ceb61-0e7e-4311-bb78-2f0158cc7acd None None] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:12:49.583325 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8d93037-e083-4bc7-9e76-574dc76d7525 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1666097f8ed545c6b0e4809c93e4775a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9c389ae65aaa4542bb5a5c4fd0bacf5b', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:12:49.583979 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8d93037-e083-4bc7-9e76-574dc76d7525 service cinder] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:12:49.584292 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8d93037-e083-4bc7-9e76-574dc76d7525 service cinder] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:12:49.584572 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8d93037-e083-4bc7-9e76-574dc76d7525 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:12:49.652753 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d8d93037-e083-4bc7-9e76-574dc76d7525 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:12:49.654321 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d8d93037-e083-4bc7-9e76-574dc76d7525 service cinder] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:12:49.663656 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 1372/2757] 10.222.0.30 () {60 vars in 1365 bytes} [Mon May 23 13:12:49 2022] GET /identity/v3/auth/tokens => generated 3236 bytes in 99 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) May 23 13:12:53.792114 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01d4a465-9dde-47a9-947f-61de66d1d8d1 service cinder] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:12:53.793571 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.fernet_utils [None req-01d4a465-9dde-47a9-947f-61de66d1d8d1 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92057) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:12:53.862270 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01d4a465-9dde-47a9-947f-61de66d1d8d1 service cinder] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:12:53.865833 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6b2c45c-8dfd-417b-99ed-725be0008956 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2eb83ca38e79449095f3a2f947bbb06a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9c389ae65aaa4542bb5a5c4fd0bacf5b', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:12:53.866505 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6b2c45c-8dfd-417b-99ed-725be0008956 service neutron] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:12:53.866787 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6b2c45c-8dfd-417b-99ed-725be0008956 service neutron] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:12:53.867059 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6b2c45c-8dfd-417b-99ed-725be0008956 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:12:53.868790 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.fernet_utils [None req-f6b2c45c-8dfd-417b-99ed-725be0008956 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92057) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:12:53.941660 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6b2c45c-8dfd-417b-99ed-725be0008956 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:12:53.943593 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6b2c45c-8dfd-417b-99ed-725be0008956 service neutron] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:12:53.953807 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 1386/2758] 10.222.0.30 () {60 vars in 1365 bytes} [Mon May 23 13:12:53 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 163 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) May 23 13:12:54.703923 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8d93037-e083-4bc7-9e76-574dc76d7525 service cinder] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:12:54.716984 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8d93037-e083-4bc7-9e76-574dc76d7525 service cinder] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:12:54.721736 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1d4fbf2-9987-4b37-95c5-0352eb026513 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:12:54.722665 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1d4fbf2-9987-4b37-95c5-0352eb026513 admin admin] REQUEST_METHOD: `DELETE` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:12:54.722953 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1d4fbf2-9987-4b37-95c5-0352eb026513 admin admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:12:54.723229 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1d4fbf2-9987-4b37-95c5-0352eb026513 admin admin] PATH_INFO: `/v3/users/f665c01235df405b8cbd54cf63884b1e` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:12:54.724633 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1d4fbf2-9987-4b37-95c5-0352eb026513 admin admin] RBAC: Authorizing `identity:delete_user(user_id=f665c01235df405b8cbd54cf63884b1e)` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:12:54.726609 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1d4fbf2-9987-4b37-95c5-0352eb026513 admin admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:12:54.894239 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-c1d4fbf2-9987-4b37-95c5-0352eb026513 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'f665c01235df405b8cbd54cf63884b1e'} {{(pid=92056) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:12:54.894941 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-c1d4fbf2-9987-4b37-95c5-0352eb026513 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'f665c01235df405b8cbd54cf63884b1e'} {{(pid=92056) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:12:54.942210 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-c1d4fbf2-9987-4b37-95c5-0352eb026513 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'f665c01235df405b8cbd54cf63884b1e'} {{(pid=92056) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:12:54.948751 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=90,keystone:DELETE=9,keystone:INSERT=1 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 13:12:54.956096 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-c1d4fbf2-9987-4b37-95c5-0352eb026513 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'f665c01235df405b8cbd54cf63884b1e'} {{(pid=92056) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:12:54.962884 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-c1d4fbf2-9987-4b37-95c5-0352eb026513 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'f665c01235df405b8cbd54cf63884b1e'} {{(pid=92056) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:12:54.969341 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 1373/2759] 10.222.0.30 () {58 vars in 1223 bytes} [Mon May 23 13:12:54 2022] DELETE /identity/v3/users/f665c01235df405b8cbd54cf63884b1e => generated 0 bytes in 266 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) May 23 13:12:54.974321 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fb36042-1199-486e-aba4-fb2bcc40ce25 None None] REQUEST_METHOD: `POST` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:12:54.974837 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fb36042-1199-486e-aba4-fb2bcc40ce25 None None] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:12:54.974951 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fb36042-1199-486e-aba4-fb2bcc40ce25 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:12:54.992675 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.auth.core [None req-9fb36042-1199-486e-aba4-fb2bcc40ce25 None None] MFA Rules not processed for user `0f7119f8fa5c4445977671140ded91a4`. Rule list: `[]` (Enabled: `True`). {{(pid=92057) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 13:12:54.994470 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.fernet_utils [None req-9fb36042-1199-486e-aba4-fb2bcc40ce25 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92057) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:12:55.043677 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=100 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 13:12:55.061750 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 1387/2760] 10.222.0.30 () {58 vars in 957 bytes} [Mon May 23 13:12:54 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 88 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) May 23 13:12:55.071705 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1d4fbf2-9987-4b37-95c5-0352eb026513 admin admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:12:55.073907 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.fernet_utils [None req-c1d4fbf2-9987-4b37-95c5-0352eb026513 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92056) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:12:55.135259 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1d4fbf2-9987-4b37-95c5-0352eb026513 admin admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:12:55.139786 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6870293d-6de8-4306-95db-94c7f7e2e2f2 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2eb83ca38e79449095f3a2f947bbb06a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9c389ae65aaa4542bb5a5c4fd0bacf5b', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:12:55.140739 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6870293d-6de8-4306-95db-94c7f7e2e2f2 service neutron] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:12:55.141162 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6870293d-6de8-4306-95db-94c7f7e2e2f2 service neutron] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:12:55.141702 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6870293d-6de8-4306-95db-94c7f7e2e2f2 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:12:55.153104 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6870293d-6de8-4306-95db-94c7f7e2e2f2 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:12:55.154528 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6870293d-6de8-4306-95db-94c7f7e2e2f2 service neutron] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:12:55.171876 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 1374/2761] 10.222.0.30 () {60 vars in 1365 bytes} [Mon May 23 13:12:55 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 101 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) May 23 13:12:55.982119 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9fb36042-1199-486e-aba4-fb2bcc40ce25 None None] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:12:55.983581 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.fernet_utils [None req-9fb36042-1199-486e-aba4-fb2bcc40ce25 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92057) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:12:55.997024 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9fb36042-1199-486e-aba4-fb2bcc40ce25 None None] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:12:56.003109 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-934534b3-5e74-49db-9ef2-eb844d8bc009 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:12:56.005869 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-934534b3-5e74-49db-9ef2-eb844d8bc009 admin admin] REQUEST_METHOD: `DELETE` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:12:56.005869 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-934534b3-5e74-49db-9ef2-eb844d8bc009 admin admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:12:56.005869 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-934534b3-5e74-49db-9ef2-eb844d8bc009 admin admin] PATH_INFO: `/v3/projects/567884b30ed64e7392b76d2d80c573a5` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:12:56.016655 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-934534b3-5e74-49db-9ef2-eb844d8bc009 admin admin] RBAC: Authorizing `identity:delete_project(project_id=567884b30ed64e7392b76d2d80c573a5)` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:12:56.019741 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-934534b3-5e74-49db-9ef2-eb844d8bc009 admin admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:12:56.131147 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [None req-934534b3-5e74-49db-9ef2-eb844d8bc009 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '567884b30ed64e7392b76d2d80c573a5', 'request_id': 'req-934534b3-5e74-49db-9ef2-eb844d8bc009'} {{(pid=92057) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:12:56.141579 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [None req-934534b3-5e74-49db-9ef2-eb844d8bc009 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '567884b30ed64e7392b76d2d80c573a5', 'request_id': 'req-934534b3-5e74-49db-9ef2-eb844d8bc009'} {{(pid=92057) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:12:56.232787 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [None req-934534b3-5e74-49db-9ef2-eb844d8bc009 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '567884b30ed64e7392b76d2d80c573a5', 'request_id': 'req-934534b3-5e74-49db-9ef2-eb844d8bc009'} {{(pid=92057) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:12:56.258989 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [None req-934534b3-5e74-49db-9ef2-eb844d8bc009 admin admin] The token cache is being invalidate because project 567884b30ed64e7392b76d2d80c573a5 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=92057) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} May 23 13:12:56.260117 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [None req-934534b3-5e74-49db-9ef2-eb844d8bc009 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=92057) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:12:56.260862 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [None req-934534b3-5e74-49db-9ef2-eb844d8bc009 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=92057) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:12:56.265509 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 1388/2762] 10.222.0.30 () {58 vars in 1232 bytes} [Mon May 23 13:12:55 2022] DELETE /identity/v3/projects/567884b30ed64e7392b76d2d80c573a5 => generated 0 bytes in 281 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) May 23 13:12:56.266766 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6870293d-6de8-4306-95db-94c7f7e2e2f2 service neutron] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:12:56.277615 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6870293d-6de8-4306-95db-94c7f7e2e2f2 service neutron] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:12:56.340919 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da844d49-a60a-4f61-9f53-88513321659c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:12:56.341849 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da844d49-a60a-4f61-9f53-88513321659c admin admin] REQUEST_METHOD: `DELETE` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:12:56.342170 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da844d49-a60a-4f61-9f53-88513321659c admin admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:12:56.342453 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da844d49-a60a-4f61-9f53-88513321659c admin admin] PATH_INFO: `/v3/users/d8005bf77369443eb9f5f444c5223caa` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:12:56.343679 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da844d49-a60a-4f61-9f53-88513321659c admin admin] RBAC: Authorizing `identity:delete_user(user_id=d8005bf77369443eb9f5f444c5223caa)` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:12:56.345995 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da844d49-a60a-4f61-9f53-88513321659c admin admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:12:56.526531 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-da844d49-a60a-4f61-9f53-88513321659c admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'd8005bf77369443eb9f5f444c5223caa'} {{(pid=92056) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:12:56.527560 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-da844d49-a60a-4f61-9f53-88513321659c admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'd8005bf77369443eb9f5f444c5223caa'} {{(pid=92056) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:12:56.633597 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-da844d49-a60a-4f61-9f53-88513321659c admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'd8005bf77369443eb9f5f444c5223caa'} {{(pid=92056) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:12:56.656588 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-da844d49-a60a-4f61-9f53-88513321659c admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'd8005bf77369443eb9f5f444c5223caa'} {{(pid=92056) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:12:56.663803 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-da844d49-a60a-4f61-9f53-88513321659c admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'd8005bf77369443eb9f5f444c5223caa'} {{(pid=92056) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:12:56.667274 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=88,keystone:DELETE=11,keystone:INSERT=1 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 13:12:56.672194 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 1375/2763] 10.222.0.30 () {58 vars in 1223 bytes} [Mon May 23 13:12:56 2022] DELETE /identity/v3/users/d8005bf77369443eb9f5f444c5223caa => generated 0 bytes in 406 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) May 23 13:12:57.146566 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-934534b3-5e74-49db-9ef2-eb844d8bc009 admin admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:12:57.148435 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.fernet_utils [None req-934534b3-5e74-49db-9ef2-eb844d8bc009 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92057) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:12:57.175318 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-934534b3-5e74-49db-9ef2-eb844d8bc009 admin admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:12:57.261577 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ad3185e-78ec-4649-9ed2-94d1024834ab admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:12:57.262496 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ad3185e-78ec-4649-9ed2-94d1024834ab admin admin] REQUEST_METHOD: `DELETE` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:12:57.262926 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ad3185e-78ec-4649-9ed2-94d1024834ab admin admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:12:57.263258 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ad3185e-78ec-4649-9ed2-94d1024834ab admin admin] PATH_INFO: `/v3/projects/5d889f687d5748ada80946ea7f2ee9c2` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:12:57.264693 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ad3185e-78ec-4649-9ed2-94d1024834ab admin admin] RBAC: Authorizing `identity:delete_project(project_id=5d889f687d5748ada80946ea7f2ee9c2)` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:12:57.266669 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ad3185e-78ec-4649-9ed2-94d1024834ab admin admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:12:57.423101 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [None req-4ad3185e-78ec-4649-9ed2-94d1024834ab admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '5d889f687d5748ada80946ea7f2ee9c2', 'request_id': 'req-4ad3185e-78ec-4649-9ed2-94d1024834ab'} {{(pid=92057) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:12:57.429218 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [None req-4ad3185e-78ec-4649-9ed2-94d1024834ab admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '5d889f687d5748ada80946ea7f2ee9c2', 'request_id': 'req-4ad3185e-78ec-4649-9ed2-94d1024834ab'} {{(pid=92057) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:12:57.491404 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [None req-4ad3185e-78ec-4649-9ed2-94d1024834ab admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '5d889f687d5748ada80946ea7f2ee9c2', 'request_id': 'req-4ad3185e-78ec-4649-9ed2-94d1024834ab'} {{(pid=92057) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:12:57.505213 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=84,keystone:DELETE=14,keystone:INSERT=2 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 13:12:57.511493 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [None req-4ad3185e-78ec-4649-9ed2-94d1024834ab admin admin] The token cache is being invalidate because project 5d889f687d5748ada80946ea7f2ee9c2 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=92057) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} May 23 13:12:57.512157 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [None req-4ad3185e-78ec-4649-9ed2-94d1024834ab admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=92057) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:12:57.513576 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [None req-4ad3185e-78ec-4649-9ed2-94d1024834ab admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=92057) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:12:57.516763 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 1389/2764] 10.222.0.30 () {58 vars in 1232 bytes} [Mon May 23 13:12:57 2022] DELETE /identity/v3/projects/5d889f687d5748ada80946ea7f2ee9c2 => generated 0 bytes in 371 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) May 23 13:12:57.521862 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96da24f9-c2cb-4c4e-84b4-aee9203fa292 None None] REQUEST_METHOD: `POST` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:12:57.522662 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96da24f9-c2cb-4c4e-84b4-aee9203fa292 None None] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:12:57.522775 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96da24f9-c2cb-4c4e-84b4-aee9203fa292 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:12:57.543404 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.auth.core [None req-96da24f9-c2cb-4c4e-84b4-aee9203fa292 None None] MFA Rules not processed for user `0f7119f8fa5c4445977671140ded91a4`. Rule list: `[]` (Enabled: `True`). {{(pid=92056) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 13:12:57.544905 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.fernet_utils [None req-96da24f9-c2cb-4c4e-84b4-aee9203fa292 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92056) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:12:57.631113 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 1376/2765] 10.222.0.30 () {58 vars in 957 bytes} [Mon May 23 13:12:57 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 106 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) May 23 13:12:57.636087 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de9dddc7-c09b-4c96-814a-f334cadfa77e None None] REQUEST_METHOD: `POST` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:12:57.636378 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de9dddc7-c09b-4c96-814a-f334cadfa77e None None] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:12:57.636660 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de9dddc7-c09b-4c96-814a-f334cadfa77e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:12:57.662568 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.auth.core [None req-de9dddc7-c09b-4c96-814a-f334cadfa77e None None] MFA Rules not processed for user `0f7119f8fa5c4445977671140ded91a4`. Rule list: `[]` (Enabled: `True`). {{(pid=92057) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 13:12:57.663813 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.fernet_utils [None req-de9dddc7-c09b-4c96-814a-f334cadfa77e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92057) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:12:57.680391 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 1390/2766] 10.222.0.30 () {58 vars in 957 bytes} [Mon May 23 13:12:57 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 46 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) May 23 13:12:57.685219 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96da24f9-c2cb-4c4e-84b4-aee9203fa292 None None] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:12:57.698082 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96da24f9-c2cb-4c4e-84b4-aee9203fa292 None None] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:12:57.707066 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ccc07a68-2dec-4b2d-97a9-d70e3df6bcfa admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:12:57.708611 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccc07a68-2dec-4b2d-97a9-d70e3df6bcfa admin admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:12:57.708826 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccc07a68-2dec-4b2d-97a9-d70e3df6bcfa admin admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:12:57.709170 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccc07a68-2dec-4b2d-97a9-d70e3df6bcfa admin admin] PATH_INFO: `/v3/domains` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:12:57.710172 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ccc07a68-2dec-4b2d-97a9-d70e3df6bcfa admin admin] RBAC: Adding query filter params (name=Default) {{(pid=92056) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} May 23 13:12:57.710579 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ccc07a68-2dec-4b2d-97a9-d70e3df6bcfa admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:12:57.712196 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ccc07a68-2dec-4b2d-97a9-d70e3df6bcfa admin admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:12:57.728921 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 1377/2767] 10.222.0.30 () {58 vars in 1152 bytes} [Mon May 23 13:12:57 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 44 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) May 23 13:12:57.733428 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de9dddc7-c09b-4c96-814a-f334cadfa77e None None] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:12:57.745122 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de9dddc7-c09b-4c96-814a-f334cadfa77e None None] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:12:57.751577 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69a5ef29-be73-4152-9dac-42d87ab8b542 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:12:57.752537 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69a5ef29-be73-4152-9dac-42d87ab8b542 admin admin] REQUEST_METHOD: `POST` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:12:57.753251 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69a5ef29-be73-4152-9dac-42d87ab8b542 admin admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:12:57.753726 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69a5ef29-be73-4152-9dac-42d87ab8b542 admin admin] PATH_INFO: `/v3/projects` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:12:57.754939 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69a5ef29-be73-4152-9dac-42d87ab8b542 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:12:57.757382 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69a5ef29-be73-4152-9dac-42d87ab8b542 admin admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:12:57.861932 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 1391/2768] 10.222.0.30 () {60 vars in 1152 bytes} [Mon May 23 13:12:57 2022] POST /identity/v3/projects => generated 385 bytes in 126 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) May 23 13:12:57.863674 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ccc07a68-2dec-4b2d-97a9-d70e3df6bcfa admin admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:12:57.875528 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ccc07a68-2dec-4b2d-97a9-d70e3df6bcfa admin admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:12:57.976830 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef4ad16f-ae14-4739-bf91-3ca2eccc25f2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:12:57.979296 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef4ad16f-ae14-4739-bf91-3ca2eccc25f2 admin admin] REQUEST_METHOD: `POST` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:12:57.979593 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef4ad16f-ae14-4739-bf91-3ca2eccc25f2 admin admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:12:57.979873 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef4ad16f-ae14-4739-bf91-3ca2eccc25f2 admin admin] PATH_INFO: `/v3/users` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:12:57.982221 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef4ad16f-ae14-4739-bf91-3ca2eccc25f2 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:12:57.984641 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef4ad16f-ae14-4739-bf91-3ca2eccc25f2 admin admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:12:58.094334 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 1378/2769] 10.222.0.30 () {60 vars in 1143 bytes} [Mon May 23 13:12:57 2022] POST /identity/v3/users => generated 293 bytes in 231 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) May 23 13:12:58.098299 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69a5ef29-be73-4152-9dac-42d87ab8b542 admin admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:12:58.112975 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69a5ef29-be73-4152-9dac-42d87ab8b542 admin admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:12:58.117717 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06fdb7c9-9576-4bf0-ac63-bfec727afc81 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:12:58.118441 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06fdb7c9-9576-4bf0-ac63-bfec727afc81 admin admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:12:58.118921 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06fdb7c9-9576-4bf0-ac63-bfec727afc81 admin admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:12:58.119337 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06fdb7c9-9576-4bf0-ac63-bfec727afc81 admin admin] PATH_INFO: `/v3/roles` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:12:58.120200 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06fdb7c9-9576-4bf0-ac63-bfec727afc81 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:12:58.122576 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06fdb7c9-9576-4bf0-ac63-bfec727afc81 admin admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:12:58.128745 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef4ad16f-ae14-4739-bf91-3ca2eccc25f2 admin admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:12:58.131038 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.fernet_utils [None req-ef4ad16f-ae14-4739-bf91-3ca2eccc25f2 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92056) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:12:58.134914 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 1392/2770] 10.222.0.30 () {58 vars in 1121 bytes} [Mon May 23 13:12:58 2022] GET /identity/v3/roles => generated 1420 bytes in 37 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) May 23 13:12:58.140715 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06fdb7c9-9576-4bf0-ac63-bfec727afc81 admin admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:12:58.153926 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06fdb7c9-9576-4bf0-ac63-bfec727afc81 admin admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:12:58.159761 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f660500b-4d32-496b-9592-f7b67e0b7169 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:12:58.160494 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f660500b-4d32-496b-9592-f7b67e0b7169 admin admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:12:58.160827 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f660500b-4d32-496b-9592-f7b67e0b7169 admin admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:12:58.161289 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f660500b-4d32-496b-9592-f7b67e0b7169 admin admin] PATH_INFO: `/v3/roles` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:12:58.170958 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f660500b-4d32-496b-9592-f7b67e0b7169 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:12:58.172891 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f660500b-4d32-496b-9592-f7b67e0b7169 admin admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:12:58.185944 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 1393/2771] 10.222.0.30 () {58 vars in 1121 bytes} [Mon May 23 13:12:58 2022] GET /identity/v3/roles => generated 1420 bytes in 48 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) May 23 13:12:58.190331 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f660500b-4d32-496b-9592-f7b67e0b7169 admin admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:12:58.202399 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f660500b-4d32-496b-9592-f7b67e0b7169 admin admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:12:58.207253 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a12db27a-3d1a-49fd-8d20-3db969cf3808 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:12:58.208131 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a12db27a-3d1a-49fd-8d20-3db969cf3808 admin admin] REQUEST_METHOD: `PUT` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:12:58.208417 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a12db27a-3d1a-49fd-8d20-3db969cf3808 admin admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:12:58.208949 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a12db27a-3d1a-49fd-8d20-3db969cf3808 admin admin] PATH_INFO: `/v3/projects/79ed2e0df6944a398fa4d41dd9d5579b/users/94bebfa8c29640b9b406d889485f5018/roles/80fb22507c9a4ff09d44d8f12f6fff1a` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:12:58.224664 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a12db27a-3d1a-49fd-8d20-3db969cf3808 admin admin] RBAC: Authorizing `identity:create_grant(project_id=79ed2e0df6944a398fa4d41dd9d5579b, user_id=94bebfa8c29640b9b406d889485f5018, role_id=80fb22507c9a4ff09d44d8f12f6fff1a)` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:12:58.227355 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a12db27a-3d1a-49fd-8d20-3db969cf3808 admin admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:12:58.234621 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef4ad16f-ae14-4739-bf91-3ca2eccc25f2 admin admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:12:58.239705 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d57297f2-af99-4747-8c09-83982fc981b0 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd5fb51f8d0fa46439cb29666970b885b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9c389ae65aaa4542bb5a5c4fd0bacf5b', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'service', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:12:58.240375 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d57297f2-af99-4747-8c09-83982fc981b0 service placement] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:12:58.240683 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d57297f2-af99-4747-8c09-83982fc981b0 service placement] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:12:58.240959 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d57297f2-af99-4747-8c09-83982fc981b0 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:12:58.242739 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.fernet_utils [None req-d57297f2-af99-4747-8c09-83982fc981b0 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92056) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:12:58.254126 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d57297f2-af99-4747-8c09-83982fc981b0 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:12:58.255737 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d57297f2-af99-4747-8c09-83982fc981b0 service placement] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:12:58.265963 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 1379/2772] 10.222.0.30 () {60 vars in 1365 bytes} [Mon May 23 13:12:58 2022] GET /identity/v3/auth/tokens => generated 3298 bytes in 138 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) May 23 13:12:58.331411 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 1394/2773] 10.222.0.30 () {60 vars in 1482 bytes} [Mon May 23 13:12:58 2022] PUT /identity/v3/projects/79ed2e0df6944a398fa4d41dd9d5579b/users/94bebfa8c29640b9b406d889485f5018/roles/80fb22507c9a4ff09d44d8f12f6fff1a => generated 0 bytes in 142 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) May 23 13:12:58.342561 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0720228-c70f-40b7-b880-1691607a2bc9 None None] REQUEST_METHOD: `POST` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:12:58.342854 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0720228-c70f-40b7-b880-1691607a2bc9 None None] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:12:58.343133 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0720228-c70f-40b7-b880-1691607a2bc9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:12:58.354965 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 13:12:58.369983 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.auth.core [None req-b0720228-c70f-40b7-b880-1691607a2bc9 None None] MFA Rules not processed for user `94bebfa8c29640b9b406d889485f5018`. Rule list: `[]` (Enabled: `True`). {{(pid=92056) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 13:12:58.371209 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.fernet_utils [None req-b0720228-c70f-40b7-b880-1691607a2bc9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92056) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:12:58.441498 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 1380/2774] 10.222.0.30 () {58 vars in 957 bytes} [Mon May 23 13:12:58 2022] POST /identity/v3/auth/tokens => generated 3252 bytes in 100 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) May 23 13:12:58.446042 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a12db27a-3d1a-49fd-8d20-3db969cf3808 admin admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:12:58.508995 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a12db27a-3d1a-49fd-8d20-3db969cf3808 admin admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:12:58.514976 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20df9b88-3ec4-42c5-bac9-17cc512214ee admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:12:58.515633 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20df9b88-3ec4-42c5-bac9-17cc512214ee admin admin] REQUEST_METHOD: `POST` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:12:58.515919 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20df9b88-3ec4-42c5-bac9-17cc512214ee admin admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:12:58.516258 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20df9b88-3ec4-42c5-bac9-17cc512214ee admin admin] PATH_INFO: `/v3/projects` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:12:58.516933 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20df9b88-3ec4-42c5-bac9-17cc512214ee admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:12:58.518884 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20df9b88-3ec4-42c5-bac9-17cc512214ee admin admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:12:58.599680 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 1395/2775] 10.222.0.30 () {60 vars in 1152 bytes} [Mon May 23 13:12:58 2022] POST /identity/v3/projects => generated 387 bytes in 154 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) May 23 13:12:58.603536 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0720228-c70f-40b7-b880-1691607a2bc9 None None] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:12:58.614912 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0720228-c70f-40b7-b880-1691607a2bc9 None None] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:12:58.674680 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-247f908a-eb73-4dde-868a-bcbe38689e7c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:12:58.675353 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-247f908a-eb73-4dde-868a-bcbe38689e7c admin admin] REQUEST_METHOD: `POST` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:12:58.675636 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-247f908a-eb73-4dde-868a-bcbe38689e7c admin admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:12:58.675914 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-247f908a-eb73-4dde-868a-bcbe38689e7c admin admin] PATH_INFO: `/v3/users` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:12:58.676727 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-247f908a-eb73-4dde-868a-bcbe38689e7c admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:12:58.678776 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-247f908a-eb73-4dde-868a-bcbe38689e7c admin admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:12:58.767841 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 1381/2776] 10.222.0.30 () {60 vars in 1143 bytes} [Mon May 23 13:12:58 2022] POST /identity/v3/users => generated 300 bytes in 163 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) May 23 13:12:58.770035 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20df9b88-3ec4-42c5-bac9-17cc512214ee admin admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:12:58.781701 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20df9b88-3ec4-42c5-bac9-17cc512214ee admin admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:12:58.787452 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-030b2a81-d679-48cf-a7c4-75530ee9b273 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:12:58.788222 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-030b2a81-d679-48cf-a7c4-75530ee9b273 admin admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:12:58.788513 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-030b2a81-d679-48cf-a7c4-75530ee9b273 admin admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:12:58.788836 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-030b2a81-d679-48cf-a7c4-75530ee9b273 admin admin] PATH_INFO: `/v3/roles` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:12:58.789917 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-030b2a81-d679-48cf-a7c4-75530ee9b273 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:12:58.791948 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-030b2a81-d679-48cf-a7c4-75530ee9b273 admin admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:12:58.803678 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 1396/2777] 10.222.0.30 () {58 vars in 1121 bytes} [Mon May 23 13:12:58 2022] GET /identity/v3/roles => generated 1420 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) May 23 13:12:58.807863 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-247f908a-eb73-4dde-868a-bcbe38689e7c admin admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:12:58.822526 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-247f908a-eb73-4dde-868a-bcbe38689e7c admin admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:12:58.823949 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5353a25-1dba-4cce-89d3-25fa099eb482 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:12:58.824836 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5353a25-1dba-4cce-89d3-25fa099eb482 admin admin] REQUEST_METHOD: `PUT` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:12:58.825120 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5353a25-1dba-4cce-89d3-25fa099eb482 admin admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:12:58.825395 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5353a25-1dba-4cce-89d3-25fa099eb482 admin admin] PATH_INFO: `/v3/projects/a6aa513726a4422c8d954779502511a0/users/bc667a8924f44881b1b87c2ed7ac2283/roles/bc3c3c9d654940cbaa5177a6be23d1bc` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:12:58.842592 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a5353a25-1dba-4cce-89d3-25fa099eb482 admin admin] RBAC: Authorizing `identity:create_grant(project_id=a6aa513726a4422c8d954779502511a0, user_id=bc667a8924f44881b1b87c2ed7ac2283, role_id=bc3c3c9d654940cbaa5177a6be23d1bc)` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:12:58.845454 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a5353a25-1dba-4cce-89d3-25fa099eb482 admin admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:12:58.920422 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 1382/2778] 10.222.0.30 () {60 vars in 1482 bytes} [Mon May 23 13:12:58 2022] PUT /identity/v3/projects/a6aa513726a4422c8d954779502511a0/users/bc667a8924f44881b1b87c2ed7ac2283/roles/bc3c3c9d654940cbaa5177a6be23d1bc => generated 0 bytes in 113 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) May 23 13:12:58.924902 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-030b2a81-d679-48cf-a7c4-75530ee9b273 admin admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:12:58.935227 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-030b2a81-d679-48cf-a7c4-75530ee9b273 admin admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:12:58.994679 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0adefb39-9a9e-4331-8583-5e81ee83a52f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:12:58.995346 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0adefb39-9a9e-4331-8583-5e81ee83a52f admin admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:12:58.995632 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0adefb39-9a9e-4331-8583-5e81ee83a52f admin admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:12:58.995910 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0adefb39-9a9e-4331-8583-5e81ee83a52f admin admin] PATH_INFO: `/v3/roles` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:12:58.996669 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0adefb39-9a9e-4331-8583-5e81ee83a52f admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:12:58.998345 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0adefb39-9a9e-4331-8583-5e81ee83a52f admin admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:12:58.999641 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 13:12:59.008426 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 1397/2779] 10.222.0.30 () {58 vars in 1121 bytes} [Mon May 23 13:12:58 2022] GET /identity/v3/roles => generated 1420 bytes in 84 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) May 23 13:12:59.013828 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5353a25-1dba-4cce-89d3-25fa099eb482 admin admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:12:59.025971 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5353a25-1dba-4cce-89d3-25fa099eb482 admin admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:12:59.030846 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-298d0ab7-3282-484b-98b7-d2c37a4e5bc9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:12:59.031806 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-298d0ab7-3282-484b-98b7-d2c37a4e5bc9 admin admin] REQUEST_METHOD: `PUT` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:12:59.032163 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-298d0ab7-3282-484b-98b7-d2c37a4e5bc9 admin admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:12:59.032513 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-298d0ab7-3282-484b-98b7-d2c37a4e5bc9 admin admin] PATH_INFO: `/v3/projects/a6aa513726a4422c8d954779502511a0/users/bc667a8924f44881b1b87c2ed7ac2283/roles/bc3c3c9d654940cbaa5177a6be23d1bc` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:12:59.034582 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-298d0ab7-3282-484b-98b7-d2c37a4e5bc9 admin admin] RBAC: Authorizing `identity:create_grant(project_id=a6aa513726a4422c8d954779502511a0, user_id=bc667a8924f44881b1b87c2ed7ac2283, role_id=bc3c3c9d654940cbaa5177a6be23d1bc)` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:12:59.037310 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-298d0ab7-3282-484b-98b7-d2c37a4e5bc9 admin admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:12:59.048246 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 1383/2780] 10.222.0.30 () {60 vars in 1482 bytes} [Mon May 23 13:12:59 2022] PUT /identity/v3/projects/a6aa513726a4422c8d954779502511a0/users/bc667a8924f44881b1b87c2ed7ac2283/roles/bc3c3c9d654940cbaa5177a6be23d1bc => generated 0 bytes in 35 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) May 23 13:12:59.059065 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-173763fe-3169-4454-bea2-bbf2afadf157 None None] REQUEST_METHOD: `POST` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:12:59.059360 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-173763fe-3169-4454-bea2-bbf2afadf157 None None] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:12:59.059639 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-173763fe-3169-4454-bea2-bbf2afadf157 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:12:59.088047 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.auth.core [None req-173763fe-3169-4454-bea2-bbf2afadf157 None None] MFA Rules not processed for user `bc667a8924f44881b1b87c2ed7ac2283`. Rule list: `[]` (Enabled: `True`). {{(pid=92057) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 13:12:59.089641 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.fernet_utils [None req-173763fe-3169-4454-bea2-bbf2afadf157 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92057) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:12:59.164390 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 1398/2781] 10.222.0.30 () {58 vars in 957 bytes} [Mon May 23 13:12:59 2022] POST /identity/v3/auth/tokens => generated 3321 bytes in 106 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) May 23 13:12:59.175379 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-298d0ab7-3282-484b-98b7-d2c37a4e5bc9 admin admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:12:59.177285 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.fernet_utils [None req-298d0ab7-3282-484b-98b7-d2c37a4e5bc9 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92056) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:12:59.236357 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 13:12:59.240456 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-298d0ab7-3282-484b-98b7-d2c37a4e5bc9 admin admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:12:59.244273 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17c816ea-cb8b-4bbd-ad8b-742e1783b6b8 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1666097f8ed545c6b0e4809c93e4775a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9c389ae65aaa4542bb5a5c4fd0bacf5b', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:12:59.244919 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17c816ea-cb8b-4bbd-ad8b-742e1783b6b8 service cinder] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:12:59.245201 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17c816ea-cb8b-4bbd-ad8b-742e1783b6b8 service cinder] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:12:59.245485 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17c816ea-cb8b-4bbd-ad8b-742e1783b6b8 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:12:59.256181 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17c816ea-cb8b-4bbd-ad8b-742e1783b6b8 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:12:59.257812 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17c816ea-cb8b-4bbd-ad8b-742e1783b6b8 service cinder] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:12:59.268991 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 1384/2782] 10.222.0.30 () {60 vars in 1365 bytes} [Mon May 23 13:12:59 2022] GET /identity/v3/auth/tokens => generated 3321 bytes in 94 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) May 23 13:13:00.403872 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-173763fe-3169-4454-bea2-bbf2afadf157 None None] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:13:00.474777 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-173763fe-3169-4454-bea2-bbf2afadf157 None None] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:13:00.480207 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-328a183b-22aa-42f2-b8dc-938eea5bdb80 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:13:00.481373 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-328a183b-22aa-42f2-b8dc-938eea5bdb80 admin admin] REQUEST_METHOD: `DELETE` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:13:00.481968 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-328a183b-22aa-42f2-b8dc-938eea5bdb80 admin admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:13:00.482473 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-328a183b-22aa-42f2-b8dc-938eea5bdb80 admin admin] PATH_INFO: `/v3/users/94bebfa8c29640b9b406d889485f5018` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:13:00.484054 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-328a183b-22aa-42f2-b8dc-938eea5bdb80 admin admin] RBAC: Authorizing `identity:delete_user(user_id=94bebfa8c29640b9b406d889485f5018)` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:13:00.486342 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-328a183b-22aa-42f2-b8dc-938eea5bdb80 admin admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:13:00.671483 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [None req-328a183b-22aa-42f2-b8dc-938eea5bdb80 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '94bebfa8c29640b9b406d889485f5018'} {{(pid=92057) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:13:00.736597 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [None req-328a183b-22aa-42f2-b8dc-938eea5bdb80 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '94bebfa8c29640b9b406d889485f5018'} {{(pid=92057) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:13:00.745118 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [None req-328a183b-22aa-42f2-b8dc-938eea5bdb80 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '94bebfa8c29640b9b406d889485f5018'} {{(pid=92057) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:13:00.745937 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [None req-328a183b-22aa-42f2-b8dc-938eea5bdb80 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '94bebfa8c29640b9b406d889485f5018'} {{(pid=92057) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:13:00.758179 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [None req-328a183b-22aa-42f2-b8dc-938eea5bdb80 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '94bebfa8c29640b9b406d889485f5018'} {{(pid=92057) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:13:00.761428 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 13:13:00.765642 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 1399/2783] 10.222.0.30 () {58 vars in 1223 bytes} [Mon May 23 13:13:00 2022] DELETE /identity/v3/users/94bebfa8c29640b9b406d889485f5018 => generated 0 bytes in 362 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) May 23 13:13:00.770267 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bd4a3f7-d0b1-491e-b5ef-4246a4ae4646 None None] REQUEST_METHOD: `POST` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:13:00.770751 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bd4a3f7-d0b1-491e-b5ef-4246a4ae4646 None None] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:13:00.770877 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bd4a3f7-d0b1-491e-b5ef-4246a4ae4646 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:13:00.789855 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.auth.core [None req-6bd4a3f7-d0b1-491e-b5ef-4246a4ae4646 None None] MFA Rules not processed for user `0f7119f8fa5c4445977671140ded91a4`. Rule list: `[]` (Enabled: `True`). {{(pid=92056) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 13:13:00.791179 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.fernet_utils [None req-6bd4a3f7-d0b1-491e-b5ef-4246a4ae4646 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92056) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:13:00.851461 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 1385/2784] 10.222.0.30 () {58 vars in 957 bytes} [Mon May 23 13:13:00 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 82 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) May 23 13:13:00.860991 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-328a183b-22aa-42f2-b8dc-938eea5bdb80 admin admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:13:00.864090 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.fernet_utils [None req-328a183b-22aa-42f2-b8dc-938eea5bdb80 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92057) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:13:00.927731 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-328a183b-22aa-42f2-b8dc-938eea5bdb80 admin admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:13:00.927731 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f66f891e-ef3a-4314-a7bc-693f29098fe7 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2eb83ca38e79449095f3a2f947bbb06a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9c389ae65aaa4542bb5a5c4fd0bacf5b', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:13:00.928512 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f66f891e-ef3a-4314-a7bc-693f29098fe7 service neutron] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:13:00.928876 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f66f891e-ef3a-4314-a7bc-693f29098fe7 service neutron] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:13:00.929358 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f66f891e-ef3a-4314-a7bc-693f29098fe7 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:13:00.944301 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f66f891e-ef3a-4314-a7bc-693f29098fe7 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:13:00.947419 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f66f891e-ef3a-4314-a7bc-693f29098fe7 service neutron] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:13:00.958523 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 1400/2785] 10.222.0.30 () {60 vars in 1365 bytes} [Mon May 23 13:13:00 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 98 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) May 23 13:13:01.383862 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6bd4a3f7-d0b1-491e-b5ef-4246a4ae4646 None None] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:13:01.386736 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.fernet_utils [None req-6bd4a3f7-d0b1-491e-b5ef-4246a4ae4646 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92056) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:13:01.408150 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6bd4a3f7-d0b1-491e-b5ef-4246a4ae4646 None None] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:13:01.416570 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f4b17e7-ecac-4993-adcc-a9d436712f7e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:13:01.417454 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f4b17e7-ecac-4993-adcc-a9d436712f7e admin admin] REQUEST_METHOD: `DELETE` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:13:01.417834 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f4b17e7-ecac-4993-adcc-a9d436712f7e admin admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:13:01.418267 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f4b17e7-ecac-4993-adcc-a9d436712f7e admin admin] PATH_INFO: `/v3/projects/79ed2e0df6944a398fa4d41dd9d5579b` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:13:01.419812 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f4b17e7-ecac-4993-adcc-a9d436712f7e admin admin] RBAC: Authorizing `identity:delete_project(project_id=79ed2e0df6944a398fa4d41dd9d5579b)` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:13:01.422029 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f4b17e7-ecac-4993-adcc-a9d436712f7e admin admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:13:01.531103 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-0f4b17e7-ecac-4993-adcc-a9d436712f7e admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '79ed2e0df6944a398fa4d41dd9d5579b', 'request_id': 'req-0f4b17e7-ecac-4993-adcc-a9d436712f7e'} {{(pid=92056) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:13:01.621724 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-0f4b17e7-ecac-4993-adcc-a9d436712f7e admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '79ed2e0df6944a398fa4d41dd9d5579b', 'request_id': 'req-0f4b17e7-ecac-4993-adcc-a9d436712f7e'} {{(pid=92056) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:13:01.631420 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-0f4b17e7-ecac-4993-adcc-a9d436712f7e admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '79ed2e0df6944a398fa4d41dd9d5579b', 'request_id': 'req-0f4b17e7-ecac-4993-adcc-a9d436712f7e'} {{(pid=92056) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:13:01.643021 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-0f4b17e7-ecac-4993-adcc-a9d436712f7e admin admin] The token cache is being invalidate because project 79ed2e0df6944a398fa4d41dd9d5579b was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=92056) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} May 23 13:13:01.643311 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-0f4b17e7-ecac-4993-adcc-a9d436712f7e admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=92056) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:13:01.643955 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-0f4b17e7-ecac-4993-adcc-a9d436712f7e admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=92056) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:13:01.645567 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 1386/2786] 10.222.0.30 () {58 vars in 1232 bytes} [Mon May 23 13:13:01 2022] DELETE /identity/v3/projects/79ed2e0df6944a398fa4d41dd9d5579b => generated 0 bytes in 263 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) May 23 13:13:01.649773 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f66f891e-ef3a-4314-a7bc-693f29098fe7 service neutron] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:13:01.667641 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f66f891e-ef3a-4314-a7bc-693f29098fe7 service neutron] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:13:01.762205 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a78a2893-c9a6-421e-b55f-f7f964d65cd0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:13:01.763098 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a78a2893-c9a6-421e-b55f-f7f964d65cd0 admin admin] REQUEST_METHOD: `DELETE` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:13:01.763386 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a78a2893-c9a6-421e-b55f-f7f964d65cd0 admin admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:13:01.763668 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a78a2893-c9a6-421e-b55f-f7f964d65cd0 admin admin] PATH_INFO: `/v3/users/bc667a8924f44881b1b87c2ed7ac2283` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:13:01.764973 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a78a2893-c9a6-421e-b55f-f7f964d65cd0 admin admin] RBAC: Authorizing `identity:delete_user(user_id=bc667a8924f44881b1b87c2ed7ac2283)` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:13:01.768417 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a78a2893-c9a6-421e-b55f-f7f964d65cd0 admin admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:13:01.979313 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [None req-a78a2893-c9a6-421e-b55f-f7f964d65cd0 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'bc667a8924f44881b1b87c2ed7ac2283'} {{(pid=92057) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:13:02.016476 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [None req-a78a2893-c9a6-421e-b55f-f7f964d65cd0 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'bc667a8924f44881b1b87c2ed7ac2283'} {{(pid=92057) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:13:02.026775 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [None req-a78a2893-c9a6-421e-b55f-f7f964d65cd0 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'bc667a8924f44881b1b87c2ed7ac2283'} {{(pid=92057) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:13:02.027535 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [None req-a78a2893-c9a6-421e-b55f-f7f964d65cd0 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'bc667a8924f44881b1b87c2ed7ac2283'} {{(pid=92057) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:13:02.042899 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [None req-a78a2893-c9a6-421e-b55f-f7f964d65cd0 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'bc667a8924f44881b1b87c2ed7ac2283'} {{(pid=92057) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:13:02.051217 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 1401/2787] 10.222.0.30 () {58 vars in 1223 bytes} [Mon May 23 13:13:01 2022] DELETE /identity/v3/users/bc667a8924f44881b1b87c2ed7ac2283 => generated 0 bytes in 402 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) May 23 13:13:02.356766 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f4b17e7-ecac-4993-adcc-a9d436712f7e admin admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:13:02.358352 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.fernet_utils [None req-0f4b17e7-ecac-4993-adcc-a9d436712f7e admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92056) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:13:02.375591 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f4b17e7-ecac-4993-adcc-a9d436712f7e admin admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:13:02.436444 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c01b7f4-0132-4dc2-a868-ba557c50d5c7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:13:02.437321 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c01b7f4-0132-4dc2-a868-ba557c50d5c7 admin admin] REQUEST_METHOD: `DELETE` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:13:02.437606 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c01b7f4-0132-4dc2-a868-ba557c50d5c7 admin admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:13:02.437895 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c01b7f4-0132-4dc2-a868-ba557c50d5c7 admin admin] PATH_INFO: `/v3/users/c6a6096760734ac49ce2f26c6065b701` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:13:02.439106 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c01b7f4-0132-4dc2-a868-ba557c50d5c7 admin admin] RBAC: Authorizing `identity:delete_user(user_id=c6a6096760734ac49ce2f26c6065b701)` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:13:02.441085 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c01b7f4-0132-4dc2-a868-ba557c50d5c7 admin admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:13:02.458457 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=91,keystone:DELETE=8,keystone:INSERT=1 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 13:13:02.532939 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a78a2893-c9a6-421e-b55f-f7f964d65cd0 admin admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:13:02.534544 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.fernet_utils [None req-a78a2893-c9a6-421e-b55f-f7f964d65cd0 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92057) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:13:02.548820 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a78a2893-c9a6-421e-b55f-f7f964d65cd0 admin admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:13:02.553367 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26aff776-a49c-4518-a223-c5cce1dc7354 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:13:02.554250 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26aff776-a49c-4518-a223-c5cce1dc7354 admin admin] REQUEST_METHOD: `DELETE` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:13:02.554532 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26aff776-a49c-4518-a223-c5cce1dc7354 admin admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:13:02.554808 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26aff776-a49c-4518-a223-c5cce1dc7354 admin admin] PATH_INFO: `/v3/projects/a6aa513726a4422c8d954779502511a0` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:13:02.555983 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26aff776-a49c-4518-a223-c5cce1dc7354 admin admin] RBAC: Authorizing `identity:delete_project(project_id=a6aa513726a4422c8d954779502511a0)` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:13:02.558142 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26aff776-a49c-4518-a223-c5cce1dc7354 admin admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:13:02.576818 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 13:13:02.647473 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-2c01b7f4-0132-4dc2-a868-ba557c50d5c7 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'c6a6096760734ac49ce2f26c6065b701'} {{(pid=92056) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:13:02.648232 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-2c01b7f4-0132-4dc2-a868-ba557c50d5c7 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'c6a6096760734ac49ce2f26c6065b701'} {{(pid=92056) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:13:02.673959 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [None req-26aff776-a49c-4518-a223-c5cce1dc7354 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'a6aa513726a4422c8d954779502511a0', 'request_id': 'req-26aff776-a49c-4518-a223-c5cce1dc7354'} {{(pid=92057) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:13:02.684947 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [None req-26aff776-a49c-4518-a223-c5cce1dc7354 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'a6aa513726a4422c8d954779502511a0', 'request_id': 'req-26aff776-a49c-4518-a223-c5cce1dc7354'} {{(pid=92057) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:13:02.707819 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-2c01b7f4-0132-4dc2-a868-ba557c50d5c7 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'c6a6096760734ac49ce2f26c6065b701'} {{(pid=92056) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:13:02.725102 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-2c01b7f4-0132-4dc2-a868-ba557c50d5c7 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'c6a6096760734ac49ce2f26c6065b701'} {{(pid=92056) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:13:02.732366 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-2c01b7f4-0132-4dc2-a868-ba557c50d5c7 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'c6a6096760734ac49ce2f26c6065b701'} {{(pid=92056) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:13:02.741936 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 1387/2788] 10.222.0.30 () {58 vars in 1223 bytes} [Mon May 23 13:13:02 2022] DELETE /identity/v3/users/c6a6096760734ac49ce2f26c6065b701 => generated 0 bytes in 383 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) May 23 13:13:02.789660 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [None req-26aff776-a49c-4518-a223-c5cce1dc7354 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'a6aa513726a4422c8d954779502511a0', 'request_id': 'req-26aff776-a49c-4518-a223-c5cce1dc7354'} {{(pid=92057) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:13:02.802493 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [None req-26aff776-a49c-4518-a223-c5cce1dc7354 admin admin] The token cache is being invalidate because project a6aa513726a4422c8d954779502511a0 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=92057) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} May 23 13:13:02.803170 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [None req-26aff776-a49c-4518-a223-c5cce1dc7354 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=92057) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:13:02.803758 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [None req-26aff776-a49c-4518-a223-c5cce1dc7354 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=92057) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:13:02.805919 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 1402/2789] 10.222.0.30 () {58 vars in 1232 bytes} [Mon May 23 13:13:02 2022] DELETE /identity/v3/projects/a6aa513726a4422c8d954779502511a0 => generated 0 bytes in 273 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) May 23 13:13:02.811058 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d359a91f-3a8f-480e-8690-a1f663078c12 None None] REQUEST_METHOD: `POST` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:13:02.812029 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d359a91f-3a8f-480e-8690-a1f663078c12 None None] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:13:02.812382 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d359a91f-3a8f-480e-8690-a1f663078c12 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:13:02.835706 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.auth.core [None req-d359a91f-3a8f-480e-8690-a1f663078c12 None None] MFA Rules not processed for user `0f7119f8fa5c4445977671140ded91a4`. Rule list: `[]` (Enabled: `True`). {{(pid=92056) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 13:13:02.836940 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.fernet_utils [None req-d359a91f-3a8f-480e-8690-a1f663078c12 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92056) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:13:02.897459 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 1388/2790] 10.222.0.30 () {58 vars in 957 bytes} [Mon May 23 13:13:02 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 88 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) May 23 13:13:02.907810 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acab1a81-1f20-4509-b3ef-dbbd5a42dce7 None None] REQUEST_METHOD: `POST` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:13:02.908097 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acab1a81-1f20-4509-b3ef-dbbd5a42dce7 None None] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:13:02.908384 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acab1a81-1f20-4509-b3ef-dbbd5a42dce7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:13:02.927905 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.auth.core [None req-acab1a81-1f20-4509-b3ef-dbbd5a42dce7 None None] MFA Rules not processed for user `0f7119f8fa5c4445977671140ded91a4`. Rule list: `[]` (Enabled: `True`). {{(pid=92057) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 13:13:02.929112 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.fernet_utils [None req-acab1a81-1f20-4509-b3ef-dbbd5a42dce7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92057) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:13:02.941599 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 1403/2791] 10.222.0.30 () {58 vars in 957 bytes} [Mon May 23 13:13:02 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) May 23 13:13:02.942885 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d359a91f-3a8f-480e-8690-a1f663078c12 None None] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:13:02.953490 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d359a91f-3a8f-480e-8690-a1f663078c12 None None] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:13:02.958997 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b150c25f-bc4c-4dd6-966e-4df72710031e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:13:02.959954 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b150c25f-bc4c-4dd6-966e-4df72710031e admin admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:13:02.960243 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b150c25f-bc4c-4dd6-966e-4df72710031e admin admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:13:02.960525 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b150c25f-bc4c-4dd6-966e-4df72710031e admin admin] PATH_INFO: `/v3/domains` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:13:02.961258 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b150c25f-bc4c-4dd6-966e-4df72710031e admin admin] RBAC: Adding query filter params (name=Default) {{(pid=92056) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} May 23 13:13:02.961566 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b150c25f-bc4c-4dd6-966e-4df72710031e admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:13:02.963880 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b150c25f-bc4c-4dd6-966e-4df72710031e admin admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:13:02.976755 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 1389/2792] 10.222.0.30 () {58 vars in 1152 bytes} [Mon May 23 13:13:02 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) May 23 13:13:02.980948 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-acab1a81-1f20-4509-b3ef-dbbd5a42dce7 None None] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:13:02.990195 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-acab1a81-1f20-4509-b3ef-dbbd5a42dce7 None None] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:13:02.995445 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c11f56ff-04b5-4b20-97f5-4fcd216129ed admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:13:02.996112 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c11f56ff-04b5-4b20-97f5-4fcd216129ed admin admin] REQUEST_METHOD: `POST` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:13:02.996393 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c11f56ff-04b5-4b20-97f5-4fcd216129ed admin admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:13:02.996666 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c11f56ff-04b5-4b20-97f5-4fcd216129ed admin admin] PATH_INFO: `/v3/projects` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:13:02.997377 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c11f56ff-04b5-4b20-97f5-4fcd216129ed admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:13:02.999372 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c11f56ff-04b5-4b20-97f5-4fcd216129ed admin admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:13:03.092868 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 1404/2793] 10.222.0.30 () {60 vars in 1152 bytes} [Mon May 23 13:13:02 2022] POST /identity/v3/projects => generated 379 bytes in 112 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) May 23 13:13:03.097150 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b150c25f-bc4c-4dd6-966e-4df72710031e admin admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:13:03.107705 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b150c25f-bc4c-4dd6-966e-4df72710031e admin admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:13:03.170653 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29376bc3-be23-44ef-8685-7ca28c56c9d2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:13:03.171199 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29376bc3-be23-44ef-8685-7ca28c56c9d2 admin admin] REQUEST_METHOD: `POST` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:13:03.171493 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29376bc3-be23-44ef-8685-7ca28c56c9d2 admin admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:13:03.171773 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29376bc3-be23-44ef-8685-7ca28c56c9d2 admin admin] PATH_INFO: `/v3/users` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:13:03.172618 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c11f56ff-04b5-4b20-97f5-4fcd216129ed admin admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:13:03.172771 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-29376bc3-be23-44ef-8685-7ca28c56c9d2 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:13:03.174374 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.fernet_utils [None req-c11f56ff-04b5-4b20-97f5-4fcd216129ed admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92057) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:13:03.174960 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-29376bc3-be23-44ef-8685-7ca28c56c9d2 admin admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:13:03.189558 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c11f56ff-04b5-4b20-97f5-4fcd216129ed admin admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:13:03.194335 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df37b979-dd8c-4413-9b3b-46c1bb4ea1a2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:13:03.195185 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df37b979-dd8c-4413-9b3b-46c1bb4ea1a2 admin admin] REQUEST_METHOD: `DELETE` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:13:03.195470 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df37b979-dd8c-4413-9b3b-46c1bb4ea1a2 admin admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:13:03.195743 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df37b979-dd8c-4413-9b3b-46c1bb4ea1a2 admin admin] PATH_INFO: `/v3/projects/c7ef8e9bc68d4e60b8993eac9f160a65` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:13:03.197856 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df37b979-dd8c-4413-9b3b-46c1bb4ea1a2 admin admin] RBAC: Authorizing `identity:delete_project(project_id=c7ef8e9bc68d4e60b8993eac9f160a65)` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:13:03.199781 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df37b979-dd8c-4413-9b3b-46c1bb4ea1a2 admin admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:13:03.274440 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 1390/2794] 10.222.0.30 () {60 vars in 1143 bytes} [Mon May 23 13:13:03 2022] POST /identity/v3/users => generated 290 bytes in 178 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) May 23 13:13:03.278833 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29376bc3-be23-44ef-8685-7ca28c56c9d2 admin admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:13:03.288499 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29376bc3-be23-44ef-8685-7ca28c56c9d2 admin admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:13:03.293457 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d502e457-08b6-4552-b9b9-7b54b99d9f31 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:13:03.294128 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d502e457-08b6-4552-b9b9-7b54b99d9f31 admin admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:13:03.294416 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d502e457-08b6-4552-b9b9-7b54b99d9f31 admin admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:13:03.294714 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d502e457-08b6-4552-b9b9-7b54b99d9f31 admin admin] PATH_INFO: `/v3/roles` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:13:03.295429 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d502e457-08b6-4552-b9b9-7b54b99d9f31 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:13:03.297006 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d502e457-08b6-4552-b9b9-7b54b99d9f31 admin admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:13:03.302250 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:DELETE=9,keystone:SELECT=87,keystone:INSERT=4 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 13:13:03.309476 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 1391/2795] 10.222.0.30 () {58 vars in 1121 bytes} [Mon May 23 13:13:03 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) May 23 13:13:03.313416 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d502e457-08b6-4552-b9b9-7b54b99d9f31 admin admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:13:03.323683 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d502e457-08b6-4552-b9b9-7b54b99d9f31 admin admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:13:03.328405 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a061d11-9a26-4fab-91d7-fb8e756897ef admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:13:03.329033 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a061d11-9a26-4fab-91d7-fb8e756897ef admin admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:13:03.329321 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a061d11-9a26-4fab-91d7-fb8e756897ef admin admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:13:03.329594 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a061d11-9a26-4fab-91d7-fb8e756897ef admin admin] PATH_INFO: `/v3/roles` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:13:03.330509 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a061d11-9a26-4fab-91d7-fb8e756897ef admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:13:03.331985 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a061d11-9a26-4fab-91d7-fb8e756897ef admin admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:13:03.342286 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 1392/2796] 10.222.0.30 () {58 vars in 1121 bytes} [Mon May 23 13:13:03 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) May 23 13:13:03.353613 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a061d11-9a26-4fab-91d7-fb8e756897ef admin admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:13:03.356030 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a061d11-9a26-4fab-91d7-fb8e756897ef admin admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:13:03.367902 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [None req-df37b979-dd8c-4413-9b3b-46c1bb4ea1a2 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'c7ef8e9bc68d4e60b8993eac9f160a65', 'request_id': 'req-df37b979-dd8c-4413-9b3b-46c1bb4ea1a2'} {{(pid=92057) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:13:03.376623 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [None req-df37b979-dd8c-4413-9b3b-46c1bb4ea1a2 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'c7ef8e9bc68d4e60b8993eac9f160a65', 'request_id': 'req-df37b979-dd8c-4413-9b3b-46c1bb4ea1a2'} {{(pid=92057) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:13:03.414461 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4cebe2d4-fbcd-4661-bcb8-ed696f7d1b88 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:13:03.415885 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4cebe2d4-fbcd-4661-bcb8-ed696f7d1b88 admin admin] REQUEST_METHOD: `PUT` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:13:03.416198 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4cebe2d4-fbcd-4661-bcb8-ed696f7d1b88 admin admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:13:03.416457 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4cebe2d4-fbcd-4661-bcb8-ed696f7d1b88 admin admin] PATH_INFO: `/v3/projects/bcf25abc268b4d53b8b7f4d384b68e25/users/fc61a55166ef4f909d9abacabbe556fe/roles/80fb22507c9a4ff09d44d8f12f6fff1a` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:13:03.430752 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4cebe2d4-fbcd-4661-bcb8-ed696f7d1b88 admin admin] RBAC: Authorizing `identity:create_grant(project_id=bcf25abc268b4d53b8b7f4d384b68e25, user_id=fc61a55166ef4f909d9abacabbe556fe, role_id=80fb22507c9a4ff09d44d8f12f6fff1a)` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:13:03.434170 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4cebe2d4-fbcd-4661-bcb8-ed696f7d1b88 admin admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:13:03.444347 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [None req-df37b979-dd8c-4413-9b3b-46c1bb4ea1a2 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'c7ef8e9bc68d4e60b8993eac9f160a65', 'request_id': 'req-df37b979-dd8c-4413-9b3b-46c1bb4ea1a2'} {{(pid=92057) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:13:03.457043 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [None req-df37b979-dd8c-4413-9b3b-46c1bb4ea1a2 admin admin] The token cache is being invalidate because project c7ef8e9bc68d4e60b8993eac9f160a65 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=92057) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} May 23 13:13:03.457364 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [None req-df37b979-dd8c-4413-9b3b-46c1bb4ea1a2 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=92057) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:13:03.458042 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [None req-df37b979-dd8c-4413-9b3b-46c1bb4ea1a2 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=92057) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:13:03.459615 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 1405/2797] 10.222.0.30 () {58 vars in 1232 bytes} [Mon May 23 13:13:03 2022] DELETE /identity/v3/projects/c7ef8e9bc68d4e60b8993eac9f160a65 => generated 0 bytes in 288 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) May 23 13:13:03.463544 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df37b979-dd8c-4413-9b3b-46c1bb4ea1a2 admin admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:13:03.464842 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.fernet_utils [None req-df37b979-dd8c-4413-9b3b-46c1bb4ea1a2 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92057) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:13:03.477213 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df37b979-dd8c-4413-9b3b-46c1bb4ea1a2 admin admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:13:03.483319 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-738b42c7-19c3-4315-b79a-e7a08fa9dca5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:13:03.484176 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-738b42c7-19c3-4315-b79a-e7a08fa9dca5 admin admin] REQUEST_METHOD: `DELETE` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:13:03.485484 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-738b42c7-19c3-4315-b79a-e7a08fa9dca5 admin admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:13:03.485484 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-738b42c7-19c3-4315-b79a-e7a08fa9dca5 admin admin] PATH_INFO: `/v3/users/a6a7f78a024b43a6b89f4e2fe5c6c28b` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:13:03.499951 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-738b42c7-19c3-4315-b79a-e7a08fa9dca5 admin admin] RBAC: Authorizing `identity:delete_user(user_id=a6a7f78a024b43a6b89f4e2fe5c6c28b)` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:13:03.501927 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-738b42c7-19c3-4315-b79a-e7a08fa9dca5 admin admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:13:03.536343 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 1393/2798] 10.222.0.30 () {60 vars in 1482 bytes} [Mon May 23 13:13:03 2022] PUT /identity/v3/projects/bcf25abc268b4d53b8b7f4d384b68e25/users/fc61a55166ef4f909d9abacabbe556fe/roles/80fb22507c9a4ff09d44d8f12f6fff1a => generated 0 bytes in 190 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) May 23 13:13:03.547180 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77ad8e54-2acc-4fed-8297-d83e38793523 None None] REQUEST_METHOD: `POST` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:13:03.547470 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77ad8e54-2acc-4fed-8297-d83e38793523 None None] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:13:03.547745 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77ad8e54-2acc-4fed-8297-d83e38793523 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:13:03.574187 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.auth.core [None req-77ad8e54-2acc-4fed-8297-d83e38793523 None None] MFA Rules not processed for user `fc61a55166ef4f909d9abacabbe556fe`. Rule list: `[]` (Enabled: `True`). {{(pid=92056) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 13:13:03.575379 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.fernet_utils [None req-77ad8e54-2acc-4fed-8297-d83e38793523 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92056) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:13:03.617003 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=77,keystone:DELETE=20,keystone:INSERT=3 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 13:13:03.620267 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [None req-738b42c7-19c3-4315-b79a-e7a08fa9dca5 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'a6a7f78a024b43a6b89f4e2fe5c6c28b'} {{(pid=92057) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:13:03.639720 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 1394/2799] 10.222.0.30 () {58 vars in 957 bytes} [Mon May 23 13:13:03 2022] POST /identity/v3/auth/tokens => generated 3246 bytes in 94 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) May 23 13:13:03.643902 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77ad8e54-2acc-4fed-8297-d83e38793523 None None] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:13:03.645227 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.fernet_utils [None req-77ad8e54-2acc-4fed-8297-d83e38793523 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92056) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:13:03.655619 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [None req-738b42c7-19c3-4315-b79a-e7a08fa9dca5 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'a6a7f78a024b43a6b89f4e2fe5c6c28b'} {{(pid=92057) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:13:03.663204 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [None req-738b42c7-19c3-4315-b79a-e7a08fa9dca5 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'a6a7f78a024b43a6b89f4e2fe5c6c28b'} {{(pid=92057) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:13:03.668656 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [None req-738b42c7-19c3-4315-b79a-e7a08fa9dca5 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'a6a7f78a024b43a6b89f4e2fe5c6c28b'} {{(pid=92057) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:13:03.676436 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [None req-738b42c7-19c3-4315-b79a-e7a08fa9dca5 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'a6a7f78a024b43a6b89f4e2fe5c6c28b'} {{(pid=92057) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:13:03.688331 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 1406/2800] 10.222.0.30 () {58 vars in 1223 bytes} [Mon May 23 13:13:03 2022] DELETE /identity/v3/users/a6a7f78a024b43a6b89f4e2fe5c6c28b => generated 0 bytes in 225 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) May 23 13:13:03.711163 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77ad8e54-2acc-4fed-8297-d83e38793523 None None] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:13:03.756025 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 13:13:03.778083 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-467ca012-127b-44ff-8c9a-c50f615d08cd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:13:03.778758 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-467ca012-127b-44ff-8c9a-c50f615d08cd admin admin] REQUEST_METHOD: `POST` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:13:03.779092 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-467ca012-127b-44ff-8c9a-c50f615d08cd admin admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:13:03.779360 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-467ca012-127b-44ff-8c9a-c50f615d08cd admin admin] PATH_INFO: `/v3/projects` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:13:03.780089 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-467ca012-127b-44ff-8c9a-c50f615d08cd admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:13:03.782268 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-467ca012-127b-44ff-8c9a-c50f615d08cd admin admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:13:03.859041 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 1395/2801] 10.222.0.30 () {60 vars in 1152 bytes} [Mon May 23 13:13:03 2022] POST /identity/v3/projects => generated 379 bytes in 216 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) May 23 13:13:03.864008 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-738b42c7-19c3-4315-b79a-e7a08fa9dca5 admin admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:13:03.866038 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.fernet_utils [None req-738b42c7-19c3-4315-b79a-e7a08fa9dca5 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92057) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:13:03.886024 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-738b42c7-19c3-4315-b79a-e7a08fa9dca5 admin admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:13:03.940486 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58db7ed1-1f80-49c7-997c-5b1428743009 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:13:03.941162 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58db7ed1-1f80-49c7-997c-5b1428743009 admin admin] REQUEST_METHOD: `POST` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:13:03.941449 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58db7ed1-1f80-49c7-997c-5b1428743009 admin admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:13:03.941735 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58db7ed1-1f80-49c7-997c-5b1428743009 admin admin] PATH_INFO: `/v3/users` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:13:03.942467 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58db7ed1-1f80-49c7-997c-5b1428743009 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:13:03.944465 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58db7ed1-1f80-49c7-997c-5b1428743009 admin admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:13:03.960360 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-467ca012-127b-44ff-8c9a-c50f615d08cd admin admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:13:03.962095 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.fernet_utils [None req-467ca012-127b-44ff-8c9a-c50f615d08cd admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92056) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:13:03.975355 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-467ca012-127b-44ff-8c9a-c50f615d08cd admin admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:13:03.980683 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8bb6d06-d63b-4094-a010-957c70583d39 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:13:03.981551 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8bb6d06-d63b-4094-a010-957c70583d39 admin admin] REQUEST_METHOD: `DELETE` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:13:03.981838 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8bb6d06-d63b-4094-a010-957c70583d39 admin admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:13:03.985334 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8bb6d06-d63b-4094-a010-957c70583d39 admin admin] PATH_INFO: `/v3/projects/6ea0b13fcc8d4dc78571a933bbf475b7` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:13:03.994076 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8bb6d06-d63b-4094-a010-957c70583d39 admin admin] RBAC: Authorizing `identity:delete_project(project_id=6ea0b13fcc8d4dc78571a933bbf475b7)` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:13:03.996002 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8bb6d06-d63b-4094-a010-957c70583d39 admin admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:13:04.046119 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 1407/2802] 10.222.0.30 () {60 vars in 1143 bytes} [Mon May 23 13:13:03 2022] POST /identity/v3/users => generated 296 bytes in 183 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) May 23 13:13:04.049749 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58db7ed1-1f80-49c7-997c-5b1428743009 admin admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:13:04.059567 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58db7ed1-1f80-49c7-997c-5b1428743009 admin admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:13:04.064202 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c7ca0a1-08ad-47e1-9092-3bff7590dc51 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:13:04.064840 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c7ca0a1-08ad-47e1-9092-3bff7590dc51 admin admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:13:04.065122 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c7ca0a1-08ad-47e1-9092-3bff7590dc51 admin admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:13:04.065466 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c7ca0a1-08ad-47e1-9092-3bff7590dc51 admin admin] PATH_INFO: `/v3/roles` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:13:04.066212 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c7ca0a1-08ad-47e1-9092-3bff7590dc51 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:13:04.067796 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c7ca0a1-08ad-47e1-9092-3bff7590dc51 admin admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:13:04.077681 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 1408/2803] 10.222.0.30 () {58 vars in 1121 bytes} [Mon May 23 13:13:04 2022] GET /identity/v3/roles => generated 1420 bytes in 28 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) May 23 13:13:04.081517 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c7ca0a1-08ad-47e1-9092-3bff7590dc51 admin admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:13:04.091433 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c7ca0a1-08ad-47e1-9092-3bff7590dc51 admin admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:13:04.096017 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af0a01bf-828e-410c-a3f7-c233a492cd97 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:13:04.096835 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af0a01bf-828e-410c-a3f7-c233a492cd97 admin admin] REQUEST_METHOD: `PUT` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:13:04.097115 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af0a01bf-828e-410c-a3f7-c233a492cd97 admin admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:13:04.097389 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af0a01bf-828e-410c-a3f7-c233a492cd97 admin admin] PATH_INFO: `/v3/projects/345e3cc06e9545c188889b34f53dcaab/users/c5ed59bb16ac4ad1a5be82653950077c/roles/bc3c3c9d654940cbaa5177a6be23d1bc` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:13:04.112364 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af0a01bf-828e-410c-a3f7-c233a492cd97 admin admin] RBAC: Authorizing `identity:create_grant(project_id=345e3cc06e9545c188889b34f53dcaab, user_id=c5ed59bb16ac4ad1a5be82653950077c, role_id=bc3c3c9d654940cbaa5177a6be23d1bc)` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:13:04.115008 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af0a01bf-828e-410c-a3f7-c233a492cd97 admin admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:13:04.134251 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-e8bb6d06-d63b-4094-a010-957c70583d39 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '6ea0b13fcc8d4dc78571a933bbf475b7', 'request_id': 'req-e8bb6d06-d63b-4094-a010-957c70583d39'} {{(pid=92056) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:13:04.210543 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 1409/2804] 10.222.0.30 () {60 vars in 1482 bytes} [Mon May 23 13:13:04 2022] PUT /identity/v3/projects/345e3cc06e9545c188889b34f53dcaab/users/c5ed59bb16ac4ad1a5be82653950077c/roles/bc3c3c9d654940cbaa5177a6be23d1bc => generated 0 bytes in 129 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) May 23 13:13:04.215124 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af0a01bf-828e-410c-a3f7-c233a492cd97 admin admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:13:04.216285 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-e8bb6d06-d63b-4094-a010-957c70583d39 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '6ea0b13fcc8d4dc78571a933bbf475b7', 'request_id': 'req-e8bb6d06-d63b-4094-a010-957c70583d39'} {{(pid=92056) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:13:04.223126 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-e8bb6d06-d63b-4094-a010-957c70583d39 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '6ea0b13fcc8d4dc78571a933bbf475b7', 'request_id': 'req-e8bb6d06-d63b-4094-a010-957c70583d39'} {{(pid=92056) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:13:04.235010 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-e8bb6d06-d63b-4094-a010-957c70583d39 admin admin] The token cache is being invalidate because project 6ea0b13fcc8d4dc78571a933bbf475b7 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=92056) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} May 23 13:13:04.235408 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-e8bb6d06-d63b-4094-a010-957c70583d39 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=92056) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:13:04.237577 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-e8bb6d06-d63b-4094-a010-957c70583d39 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=92056) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:13:04.238873 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 1396/2805] 10.222.0.30 () {58 vars in 1232 bytes} [Mon May 23 13:13:03 2022] DELETE /identity/v3/projects/6ea0b13fcc8d4dc78571a933bbf475b7 => generated 0 bytes in 278 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) May 23 13:13:04.282865 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af0a01bf-828e-410c-a3f7-c233a492cd97 admin admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:13:04.287902 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0ed587e-5ec2-41af-b463-400e06962068 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:13:04.288588 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0ed587e-5ec2-41af-b463-400e06962068 admin admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:13:04.288875 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0ed587e-5ec2-41af-b463-400e06962068 admin admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:13:04.289156 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0ed587e-5ec2-41af-b463-400e06962068 admin admin] PATH_INFO: `/v3/roles` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:13:04.289901 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0ed587e-5ec2-41af-b463-400e06962068 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:13:04.291600 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0ed587e-5ec2-41af-b463-400e06962068 admin admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:13:04.302589 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 1410/2806] 10.222.0.30 () {58 vars in 1121 bytes} [Mon May 23 13:13:04 2022] GET /identity/v3/roles => generated 1420 bytes in 88 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) May 23 13:13:04.307148 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8bb6d06-d63b-4094-a010-957c70583d39 admin admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:13:04.308615 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.fernet_utils [None req-e8bb6d06-d63b-4094-a010-957c70583d39 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92056) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:13:04.382036 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8bb6d06-d63b-4094-a010-957c70583d39 admin admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:13:04.387569 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da157780-0790-4790-b42a-f3f301dab9a6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:13:04.395012 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da157780-0790-4790-b42a-f3f301dab9a6 admin admin] REQUEST_METHOD: `PUT` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:13:04.395012 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da157780-0790-4790-b42a-f3f301dab9a6 admin admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:13:04.395012 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da157780-0790-4790-b42a-f3f301dab9a6 admin admin] PATH_INFO: `/v3/projects/345e3cc06e9545c188889b34f53dcaab/users/c5ed59bb16ac4ad1a5be82653950077c/roles/bc3c3c9d654940cbaa5177a6be23d1bc` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:13:04.395012 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da157780-0790-4790-b42a-f3f301dab9a6 admin admin] RBAC: Authorizing `identity:create_grant(project_id=345e3cc06e9545c188889b34f53dcaab, user_id=c5ed59bb16ac4ad1a5be82653950077c, role_id=bc3c3c9d654940cbaa5177a6be23d1bc)` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:13:04.395012 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da157780-0790-4790-b42a-f3f301dab9a6 admin admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:13:04.406844 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 1397/2807] 10.222.0.30 () {60 vars in 1482 bytes} [Mon May 23 13:13:04 2022] PUT /identity/v3/projects/345e3cc06e9545c188889b34f53dcaab/users/c5ed59bb16ac4ad1a5be82653950077c/roles/bc3c3c9d654940cbaa5177a6be23d1bc => generated 0 bytes in 100 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) May 23 13:13:04.418600 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ea56e8a-18fe-4d3b-909f-0ff434f0cfd2 None None] REQUEST_METHOD: `POST` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:13:04.418900 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ea56e8a-18fe-4d3b-909f-0ff434f0cfd2 None None] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:13:04.419183 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ea56e8a-18fe-4d3b-909f-0ff434f0cfd2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:13:04.441352 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:DELETE=4,keystone:SELECT=91,keystone:INSERT=5 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 13:13:04.450642 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.auth.core [None req-2ea56e8a-18fe-4d3b-909f-0ff434f0cfd2 None None] MFA Rules not processed for user `c5ed59bb16ac4ad1a5be82653950077c`. Rule list: `[]` (Enabled: `True`). {{(pid=92057) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 13:13:04.451921 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.fernet_utils [None req-2ea56e8a-18fe-4d3b-909f-0ff434f0cfd2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92057) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:13:04.523610 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 1411/2808] 10.222.0.30 () {58 vars in 957 bytes} [Mon May 23 13:13:04 2022] POST /identity/v3/auth/tokens => generated 3313 bytes in 106 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) May 23 13:13:04.536047 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da157780-0790-4790-b42a-f3f301dab9a6 admin admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:13:04.537770 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.fernet_utils [None req-da157780-0790-4790-b42a-f3f301dab9a6 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92056) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:13:04.543088 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=90,keystone:INSERT=3,keystone:DELETE=7 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 13:13:04.595140 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da157780-0790-4790-b42a-f3f301dab9a6 admin admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:13:04.598823 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5aecdbd-0d5e-4898-a167-dfc4b7d228a2 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1666097f8ed545c6b0e4809c93e4775a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9c389ae65aaa4542bb5a5c4fd0bacf5b', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:13:04.599461 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5aecdbd-0d5e-4898-a167-dfc4b7d228a2 service cinder] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:13:04.599743 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5aecdbd-0d5e-4898-a167-dfc4b7d228a2 service cinder] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:13:04.600030 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5aecdbd-0d5e-4898-a167-dfc4b7d228a2 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:13:04.608953 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a5aecdbd-0d5e-4898-a167-dfc4b7d228a2 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:13:04.612752 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a5aecdbd-0d5e-4898-a167-dfc4b7d228a2 service cinder] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:13:04.620507 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 1398/2809] 10.222.0.30 () {60 vars in 1365 bytes} [Mon May 23 13:13:04 2022] GET /identity/v3/auth/tokens => generated 3313 bytes in 85 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) May 23 13:13:05.058507 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ea56e8a-18fe-4d3b-909f-0ff434f0cfd2 None None] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:13:05.119485 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ea56e8a-18fe-4d3b-909f-0ff434f0cfd2 None None] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:13:05.123954 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6afca54f-8885-451d-b4d3-e816a3892c42 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:13:05.124813 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6afca54f-8885-451d-b4d3-e816a3892c42 admin admin] REQUEST_METHOD: `DELETE` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:13:05.125096 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6afca54f-8885-451d-b4d3-e816a3892c42 admin admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:13:05.125371 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6afca54f-8885-451d-b4d3-e816a3892c42 admin admin] PATH_INFO: `/v3/users/fc61a55166ef4f909d9abacabbe556fe` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:13:05.126686 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6afca54f-8885-451d-b4d3-e816a3892c42 admin admin] RBAC: Authorizing `identity:delete_user(user_id=fc61a55166ef4f909d9abacabbe556fe)` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:13:05.128610 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6afca54f-8885-451d-b4d3-e816a3892c42 admin admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:13:05.343878 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [None req-6afca54f-8885-451d-b4d3-e816a3892c42 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'fc61a55166ef4f909d9abacabbe556fe'} {{(pid=92057) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:13:05.409890 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [None req-6afca54f-8885-451d-b4d3-e816a3892c42 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'fc61a55166ef4f909d9abacabbe556fe'} {{(pid=92057) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:13:05.417155 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [None req-6afca54f-8885-451d-b4d3-e816a3892c42 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'fc61a55166ef4f909d9abacabbe556fe'} {{(pid=92057) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:13:05.417785 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [None req-6afca54f-8885-451d-b4d3-e816a3892c42 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'fc61a55166ef4f909d9abacabbe556fe'} {{(pid=92057) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:13:05.429688 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [None req-6afca54f-8885-451d-b4d3-e816a3892c42 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'fc61a55166ef4f909d9abacabbe556fe'} {{(pid=92057) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:13:05.436652 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 1412/2810] 10.222.0.30 () {58 vars in 1223 bytes} [Mon May 23 13:13:05 2022] DELETE /identity/v3/users/fc61a55166ef4f909d9abacabbe556fe => generated 0 bytes in 379 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) May 23 13:13:05.441181 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-037fff77-7d22-4876-8761-7271dcd29bbc None None] REQUEST_METHOD: `POST` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:13:05.441477 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-037fff77-7d22-4876-8761-7271dcd29bbc None None] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:13:05.441758 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-037fff77-7d22-4876-8761-7271dcd29bbc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:13:05.459711 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.auth.core [None req-037fff77-7d22-4876-8761-7271dcd29bbc None None] MFA Rules not processed for user `0f7119f8fa5c4445977671140ded91a4`. Rule list: `[]` (Enabled: `True`). {{(pid=92056) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 13:13:05.460944 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.fernet_utils [None req-037fff77-7d22-4876-8761-7271dcd29bbc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92056) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:13:05.526210 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 1399/2811] 10.222.0.30 () {58 vars in 957 bytes} [Mon May 23 13:13:05 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) May 23 13:13:05.536348 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6afca54f-8885-451d-b4d3-e816a3892c42 admin admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:13:05.537820 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.fernet_utils [None req-6afca54f-8885-451d-b4d3-e816a3892c42 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92057) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:13:05.557507 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 13:13:05.599721 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6afca54f-8885-451d-b4d3-e816a3892c42 admin admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:13:05.603831 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44777f32-9ca4-4d45-b3f0-453b664ec25c service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2eb83ca38e79449095f3a2f947bbb06a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9c389ae65aaa4542bb5a5c4fd0bacf5b', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:13:05.604753 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44777f32-9ca4-4d45-b3f0-453b664ec25c service neutron] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:13:05.605232 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44777f32-9ca4-4d45-b3f0-453b664ec25c service neutron] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:13:05.606048 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44777f32-9ca4-4d45-b3f0-453b664ec25c service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:13:05.615680 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44777f32-9ca4-4d45-b3f0-453b664ec25c service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:13:05.617131 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44777f32-9ca4-4d45-b3f0-453b664ec25c service neutron] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:13:05.627589 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 1413/2812] 10.222.0.30 () {60 vars in 1365 bytes} [Mon May 23 13:13:05 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 92 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) May 23 13:13:06.011809 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-037fff77-7d22-4876-8761-7271dcd29bbc None None] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:13:06.013369 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.fernet_utils [None req-037fff77-7d22-4876-8761-7271dcd29bbc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92056) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:13:06.026330 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-037fff77-7d22-4876-8761-7271dcd29bbc None None] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:13:06.031248 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d67535b-d6d0-4133-b2f0-fbf76e5685d2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:13:06.033829 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d67535b-d6d0-4133-b2f0-fbf76e5685d2 admin admin] REQUEST_METHOD: `DELETE` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:13:06.033829 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d67535b-d6d0-4133-b2f0-fbf76e5685d2 admin admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:13:06.033829 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d67535b-d6d0-4133-b2f0-fbf76e5685d2 admin admin] PATH_INFO: `/v3/projects/bcf25abc268b4d53b8b7f4d384b68e25` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:13:06.033829 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d67535b-d6d0-4133-b2f0-fbf76e5685d2 admin admin] RBAC: Authorizing `identity:delete_project(project_id=bcf25abc268b4d53b8b7f4d384b68e25)` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:13:06.035745 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d67535b-d6d0-4133-b2f0-fbf76e5685d2 admin admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:13:06.130407 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-1d67535b-d6d0-4133-b2f0-fbf76e5685d2 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'bcf25abc268b4d53b8b7f4d384b68e25', 'request_id': 'req-1d67535b-d6d0-4133-b2f0-fbf76e5685d2'} {{(pid=92056) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:13:06.181293 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-1d67535b-d6d0-4133-b2f0-fbf76e5685d2 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'bcf25abc268b4d53b8b7f4d384b68e25', 'request_id': 'req-1d67535b-d6d0-4133-b2f0-fbf76e5685d2'} {{(pid=92056) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:13:06.188286 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-1d67535b-d6d0-4133-b2f0-fbf76e5685d2 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'bcf25abc268b4d53b8b7f4d384b68e25', 'request_id': 'req-1d67535b-d6d0-4133-b2f0-fbf76e5685d2'} {{(pid=92056) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:13:06.205967 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-1d67535b-d6d0-4133-b2f0-fbf76e5685d2 admin admin] The token cache is being invalidate because project bcf25abc268b4d53b8b7f4d384b68e25 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=92056) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} May 23 13:13:06.206259 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-1d67535b-d6d0-4133-b2f0-fbf76e5685d2 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=92056) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:13:06.206956 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-1d67535b-d6d0-4133-b2f0-fbf76e5685d2 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=92056) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:13:06.208479 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 1400/2813] 10.222.0.30 () {58 vars in 1232 bytes} [Mon May 23 13:13:06 2022] DELETE /identity/v3/projects/bcf25abc268b4d53b8b7f4d384b68e25 => generated 0 bytes in 197 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) May 23 13:13:06.213118 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44777f32-9ca4-4d45-b3f0-453b664ec25c service neutron] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:13:06.222891 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44777f32-9ca4-4d45-b3f0-453b664ec25c service neutron] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:13:06.280134 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83f75598-a9d8-4817-8d8a-d31aaaf921c8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:13:06.280996 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83f75598-a9d8-4817-8d8a-d31aaaf921c8 admin admin] REQUEST_METHOD: `DELETE` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:13:06.281283 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83f75598-a9d8-4817-8d8a-d31aaaf921c8 admin admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:13:06.281561 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83f75598-a9d8-4817-8d8a-d31aaaf921c8 admin admin] PATH_INFO: `/v3/users/c5ed59bb16ac4ad1a5be82653950077c` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:13:06.282762 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83f75598-a9d8-4817-8d8a-d31aaaf921c8 admin admin] RBAC: Authorizing `identity:delete_user(user_id=c5ed59bb16ac4ad1a5be82653950077c)` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:13:06.284752 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83f75598-a9d8-4817-8d8a-d31aaaf921c8 admin admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:13:06.464457 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [None req-83f75598-a9d8-4817-8d8a-d31aaaf921c8 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'c5ed59bb16ac4ad1a5be82653950077c'} {{(pid=92057) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:13:06.536062 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [None req-83f75598-a9d8-4817-8d8a-d31aaaf921c8 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'c5ed59bb16ac4ad1a5be82653950077c'} {{(pid=92057) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:13:06.544229 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [None req-83f75598-a9d8-4817-8d8a-d31aaaf921c8 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'c5ed59bb16ac4ad1a5be82653950077c'} {{(pid=92057) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:13:06.544908 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [None req-83f75598-a9d8-4817-8d8a-d31aaaf921c8 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'c5ed59bb16ac4ad1a5be82653950077c'} {{(pid=92057) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:13:06.557822 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [None req-83f75598-a9d8-4817-8d8a-d31aaaf921c8 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'c5ed59bb16ac4ad1a5be82653950077c'} {{(pid=92057) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:13:06.564050 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 1414/2814] 10.222.0.30 () {58 vars in 1223 bytes} [Mon May 23 13:13:06 2022] DELETE /identity/v3/users/c5ed59bb16ac4ad1a5be82653950077c => generated 0 bytes in 352 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) May 23 13:13:06.995275 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d67535b-d6d0-4133-b2f0-fbf76e5685d2 admin admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:13:06.996730 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.fernet_utils [None req-1d67535b-d6d0-4133-b2f0-fbf76e5685d2 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92056) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:13:07.015455 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d67535b-d6d0-4133-b2f0-fbf76e5685d2 admin admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:13:07.043461 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 13:13:07.075814 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1924e229-190f-453d-b033-f5ac3f8a18e0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:13:07.076678 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1924e229-190f-453d-b033-f5ac3f8a18e0 admin admin] REQUEST_METHOD: `DELETE` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:13:07.076965 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1924e229-190f-453d-b033-f5ac3f8a18e0 admin admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:13:07.077250 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1924e229-190f-453d-b033-f5ac3f8a18e0 admin admin] PATH_INFO: `/v3/projects/345e3cc06e9545c188889b34f53dcaab` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:13:07.078517 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1924e229-190f-453d-b033-f5ac3f8a18e0 admin admin] RBAC: Authorizing `identity:delete_project(project_id=345e3cc06e9545c188889b34f53dcaab)` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:13:07.080527 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1924e229-190f-453d-b033-f5ac3f8a18e0 admin admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:13:07.164097 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-1924e229-190f-453d-b033-f5ac3f8a18e0 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '345e3cc06e9545c188889b34f53dcaab', 'request_id': 'req-1924e229-190f-453d-b033-f5ac3f8a18e0'} {{(pid=92056) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:13:07.221083 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-1924e229-190f-453d-b033-f5ac3f8a18e0 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '345e3cc06e9545c188889b34f53dcaab', 'request_id': 'req-1924e229-190f-453d-b033-f5ac3f8a18e0'} {{(pid=92056) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:13:07.230409 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-1924e229-190f-453d-b033-f5ac3f8a18e0 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '345e3cc06e9545c188889b34f53dcaab', 'request_id': 'req-1924e229-190f-453d-b033-f5ac3f8a18e0'} {{(pid=92056) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:13:07.245738 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-1924e229-190f-453d-b033-f5ac3f8a18e0 admin admin] The token cache is being invalidate because project 345e3cc06e9545c188889b34f53dcaab was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=92056) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} May 23 13:13:07.245738 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-1924e229-190f-453d-b033-f5ac3f8a18e0 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=92056) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:13:07.245738 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-1924e229-190f-453d-b033-f5ac3f8a18e0 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=92056) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:13:07.245738 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 1401/2815] 10.222.0.30 () {58 vars in 1232 bytes} [Mon May 23 13:13:06 2022] DELETE /identity/v3/projects/345e3cc06e9545c188889b34f53dcaab => generated 0 bytes in 249 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) May 23 13:13:07.249584 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c60e926-66b9-44cb-8b85-2865b1279280 None None] REQUEST_METHOD: `POST` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:13:07.250070 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c60e926-66b9-44cb-8b85-2865b1279280 None None] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:13:07.250206 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c60e926-66b9-44cb-8b85-2865b1279280 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:13:07.268038 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.auth.core [None req-8c60e926-66b9-44cb-8b85-2865b1279280 None None] MFA Rules not processed for user `0f7119f8fa5c4445977671140ded91a4`. Rule list: `[]` (Enabled: `True`). {{(pid=92057) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 13:13:07.269216 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.fernet_utils [None req-8c60e926-66b9-44cb-8b85-2865b1279280 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92057) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:13:07.302239 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 13:13:07.329025 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 1415/2816] 10.222.0.30 () {58 vars in 957 bytes} [Mon May 23 13:13:07 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 80 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) May 23 13:13:07.336766 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a5e190b-65e9-4ad1-a030-75d7f964a6c8 None None] REQUEST_METHOD: `POST` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:13:07.337079 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a5e190b-65e9-4ad1-a030-75d7f964a6c8 None None] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:13:07.337402 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a5e190b-65e9-4ad1-a030-75d7f964a6c8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:13:07.356570 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.auth.core [None req-6a5e190b-65e9-4ad1-a030-75d7f964a6c8 None None] MFA Rules not processed for user `0f7119f8fa5c4445977671140ded91a4`. Rule list: `[]` (Enabled: `True`). {{(pid=92056) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 13:13:07.357746 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.fernet_utils [None req-6a5e190b-65e9-4ad1-a030-75d7f964a6c8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92056) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:13:07.366322 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 1402/2817] 10.222.0.30 () {58 vars in 957 bytes} [Mon May 23 13:13:07 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) May 23 13:13:07.371551 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c60e926-66b9-44cb-8b85-2865b1279280 None None] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:13:07.381395 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c60e926-66b9-44cb-8b85-2865b1279280 None None] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:13:07.386617 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec05ba54-9ef8-478f-a157-8e71b00922ae admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:13:07.387271 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec05ba54-9ef8-478f-a157-8e71b00922ae admin admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:13:07.387559 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec05ba54-9ef8-478f-a157-8e71b00922ae admin admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:13:07.387845 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec05ba54-9ef8-478f-a157-8e71b00922ae admin admin] PATH_INFO: `/v3/domains` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:13:07.388574 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec05ba54-9ef8-478f-a157-8e71b00922ae admin admin] RBAC: Adding query filter params (name=Default) {{(pid=92057) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} May 23 13:13:07.388883 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec05ba54-9ef8-478f-a157-8e71b00922ae admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:13:07.390533 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec05ba54-9ef8-478f-a157-8e71b00922ae admin admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:13:07.402207 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 1416/2818] 10.222.0.30 () {58 vars in 1152 bytes} [Mon May 23 13:13:07 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) May 23 13:13:07.406365 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a5e190b-65e9-4ad1-a030-75d7f964a6c8 None None] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:13:07.416172 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a5e190b-65e9-4ad1-a030-75d7f964a6c8 None None] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:13:07.420933 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2927f459-e36b-4f91-823b-63920aaa526b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:13:07.421583 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2927f459-e36b-4f91-823b-63920aaa526b admin admin] REQUEST_METHOD: `POST` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:13:07.421868 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2927f459-e36b-4f91-823b-63920aaa526b admin admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:13:07.422179 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2927f459-e36b-4f91-823b-63920aaa526b admin admin] PATH_INFO: `/v3/projects` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:13:07.422882 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2927f459-e36b-4f91-823b-63920aaa526b admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:13:07.424823 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2927f459-e36b-4f91-823b-63920aaa526b admin admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:13:07.502713 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 1403/2819] 10.222.0.30 () {60 vars in 1152 bytes} [Mon May 23 13:13:07 2022] POST /identity/v3/projects => generated 361 bytes in 97 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) May 23 13:13:07.506757 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec05ba54-9ef8-478f-a157-8e71b00922ae admin admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:13:07.516925 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec05ba54-9ef8-478f-a157-8e71b00922ae admin admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:13:07.574442 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4846520-39d7-429a-a68e-4477d9d7c8ee admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:13:07.575103 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4846520-39d7-429a-a68e-4477d9d7c8ee admin admin] REQUEST_METHOD: `POST` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:13:07.575389 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4846520-39d7-429a-a68e-4477d9d7c8ee admin admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:13:07.575668 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4846520-39d7-429a-a68e-4477d9d7c8ee admin admin] PATH_INFO: `/v3/users` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:13:07.576355 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c4846520-39d7-429a-a68e-4477d9d7c8ee admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:13:07.578307 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c4846520-39d7-429a-a68e-4477d9d7c8ee admin admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:13:07.646480 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 1417/2820] 10.222.0.30 () {60 vars in 1143 bytes} [Mon May 23 13:13:07 2022] POST /identity/v3/users => generated 281 bytes in 137 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) May 23 13:13:07.648019 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2927f459-e36b-4f91-823b-63920aaa526b admin admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:13:07.658937 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2927f459-e36b-4f91-823b-63920aaa526b admin admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:13:07.663596 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba55343c-0abc-4051-a8bb-3ffd991ff485 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:13:07.664251 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba55343c-0abc-4051-a8bb-3ffd991ff485 admin admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:13:07.664536 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba55343c-0abc-4051-a8bb-3ffd991ff485 admin admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:13:07.664820 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba55343c-0abc-4051-a8bb-3ffd991ff485 admin admin] PATH_INFO: `/v3/roles` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:13:07.666139 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba55343c-0abc-4051-a8bb-3ffd991ff485 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:13:07.667744 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba55343c-0abc-4051-a8bb-3ffd991ff485 admin admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:13:07.677936 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 1404/2821] 10.222.0.30 () {58 vars in 1121 bytes} [Mon May 23 13:13:07 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) May 23 13:13:07.681905 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4846520-39d7-429a-a68e-4477d9d7c8ee admin admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:13:07.692014 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4846520-39d7-429a-a68e-4477d9d7c8ee admin admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:13:07.696839 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b51260ef-a34e-4652-9ecf-02ed49365afe admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:13:07.697497 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b51260ef-a34e-4652-9ecf-02ed49365afe admin admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:13:07.697780 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b51260ef-a34e-4652-9ecf-02ed49365afe admin admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:13:07.698089 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b51260ef-a34e-4652-9ecf-02ed49365afe admin admin] PATH_INFO: `/v3/roles` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:13:07.698807 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b51260ef-a34e-4652-9ecf-02ed49365afe admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:13:07.700403 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b51260ef-a34e-4652-9ecf-02ed49365afe admin admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:13:07.710521 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 1418/2822] 10.222.0.30 () {58 vars in 1121 bytes} [Mon May 23 13:13:07 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) May 23 13:13:07.715269 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba55343c-0abc-4051-a8bb-3ffd991ff485 admin admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:13:07.725451 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba55343c-0abc-4051-a8bb-3ffd991ff485 admin admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:13:07.731136 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6473831-c843-4eea-bd57-0869e458f6a1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:13:07.732104 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6473831-c843-4eea-bd57-0869e458f6a1 admin admin] REQUEST_METHOD: `PUT` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:13:07.732493 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6473831-c843-4eea-bd57-0869e458f6a1 admin admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:13:07.732890 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6473831-c843-4eea-bd57-0869e458f6a1 admin admin] PATH_INFO: `/v3/projects/e85e2312cfa54696a35ae5d6f688857e/users/a66d08b73a9a4b8ea62ebd7853cef13d/roles/80fb22507c9a4ff09d44d8f12f6fff1a` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:13:07.753462 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6473831-c843-4eea-bd57-0869e458f6a1 admin admin] RBAC: Authorizing `identity:create_grant(project_id=e85e2312cfa54696a35ae5d6f688857e, user_id=a66d08b73a9a4b8ea62ebd7853cef13d, role_id=80fb22507c9a4ff09d44d8f12f6fff1a)` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:13:07.756097 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6473831-c843-4eea-bd57-0869e458f6a1 admin admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:13:07.845026 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 1405/2823] 10.222.0.30 () {60 vars in 1482 bytes} [Mon May 23 13:13:07 2022] PUT /identity/v3/projects/e85e2312cfa54696a35ae5d6f688857e/users/a66d08b73a9a4b8ea62ebd7853cef13d/roles/80fb22507c9a4ff09d44d8f12f6fff1a => generated 0 bytes in 130 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) May 23 13:13:07.856195 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68ecb314-a3e5-4066-8e7a-e8289fa8b886 None None] REQUEST_METHOD: `POST` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:13:07.856487 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68ecb314-a3e5-4066-8e7a-e8289fa8b886 None None] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:13:07.856764 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68ecb314-a3e5-4066-8e7a-e8289fa8b886 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:13:07.883833 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.auth.core [None req-68ecb314-a3e5-4066-8e7a-e8289fa8b886 None None] MFA Rules not processed for user `a66d08b73a9a4b8ea62ebd7853cef13d`. Rule list: `[]` (Enabled: `True`). {{(pid=92057) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 13:13:07.885519 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.fernet_utils [None req-68ecb314-a3e5-4066-8e7a-e8289fa8b886 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92057) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:13:07.950997 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 1419/2824] 10.222.0.30 () {58 vars in 957 bytes} [Mon May 23 13:13:07 2022] POST /identity/v3/auth/tokens => generated 3228 bytes in 96 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) May 23 13:13:07.981940 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6473831-c843-4eea-bd57-0869e458f6a1 admin admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:13:07.983615 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.fernet_utils [None req-c6473831-c843-4eea-bd57-0869e458f6a1 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92056) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:13:08.040280 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6473831-c843-4eea-bd57-0869e458f6a1 admin admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:13:08.043902 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a8a299c-5a00-4b20-b649-802347d79c56 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1666097f8ed545c6b0e4809c93e4775a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9c389ae65aaa4542bb5a5c4fd0bacf5b', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:13:08.044535 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a8a299c-5a00-4b20-b649-802347d79c56 service cinder] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:13:08.044818 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a8a299c-5a00-4b20-b649-802347d79c56 service cinder] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:13:08.045094 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a8a299c-5a00-4b20-b649-802347d79c56 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:13:08.053369 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a8a299c-5a00-4b20-b649-802347d79c56 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:13:08.054826 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a8a299c-5a00-4b20-b649-802347d79c56 service cinder] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:13:08.058454 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 13:13:08.064221 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 1406/2825] 10.222.0.30 () {60 vars in 1365 bytes} [Mon May 23 13:13:07 2022] GET /identity/v3/auth/tokens => generated 3228 bytes in 83 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) May 23 13:13:08.078593 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68ecb314-a3e5-4066-8e7a-e8289fa8b886 None None] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:13:08.112165 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 13:13:08.140635 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68ecb314-a3e5-4066-8e7a-e8289fa8b886 None None] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:13:08.145205 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01d631f4-4bde-4f10-aeaf-128e42e45063 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:13:08.146117 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01d631f4-4bde-4f10-aeaf-128e42e45063 admin admin] REQUEST_METHOD: `DELETE` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:13:08.146398 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01d631f4-4bde-4f10-aeaf-128e42e45063 admin admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:13:08.146697 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01d631f4-4bde-4f10-aeaf-128e42e45063 admin admin] PATH_INFO: `/v3/users/a66d08b73a9a4b8ea62ebd7853cef13d` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:13:08.148112 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01d631f4-4bde-4f10-aeaf-128e42e45063 admin admin] RBAC: Authorizing `identity:delete_user(user_id=a66d08b73a9a4b8ea62ebd7853cef13d)` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:13:08.150685 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01d631f4-4bde-4f10-aeaf-128e42e45063 admin admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:13:08.373047 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [None req-01d631f4-4bde-4f10-aeaf-128e42e45063 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'a66d08b73a9a4b8ea62ebd7853cef13d'} {{(pid=92057) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:13:08.443863 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [None req-01d631f4-4bde-4f10-aeaf-128e42e45063 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'a66d08b73a9a4b8ea62ebd7853cef13d'} {{(pid=92057) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:13:08.451664 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [None req-01d631f4-4bde-4f10-aeaf-128e42e45063 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'a66d08b73a9a4b8ea62ebd7853cef13d'} {{(pid=92057) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:13:08.452960 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [None req-01d631f4-4bde-4f10-aeaf-128e42e45063 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'a66d08b73a9a4b8ea62ebd7853cef13d'} {{(pid=92057) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:13:08.464179 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [None req-01d631f4-4bde-4f10-aeaf-128e42e45063 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'a66d08b73a9a4b8ea62ebd7853cef13d'} {{(pid=92057) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:13:08.470532 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 1420/2826] 10.222.0.30 () {58 vars in 1223 bytes} [Mon May 23 13:13:08 2022] DELETE /identity/v3/users/a66d08b73a9a4b8ea62ebd7853cef13d => generated 0 bytes in 392 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) May 23 13:13:08.475045 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48d82e1e-83ab-407b-9784-dcd9b393d503 None None] REQUEST_METHOD: `POST` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:13:08.475338 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48d82e1e-83ab-407b-9784-dcd9b393d503 None None] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:13:08.475620 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48d82e1e-83ab-407b-9784-dcd9b393d503 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:13:08.492617 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.auth.core [None req-48d82e1e-83ab-407b-9784-dcd9b393d503 None None] MFA Rules not processed for user `0f7119f8fa5c4445977671140ded91a4`. Rule list: `[]` (Enabled: `True`). {{(pid=92056) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 13:13:08.494439 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.fernet_utils [None req-48d82e1e-83ab-407b-9784-dcd9b393d503 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92056) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:13:08.552033 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 1407/2827] 10.222.0.30 () {58 vars in 957 bytes} [Mon May 23 13:13:08 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 78 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) May 23 13:13:08.562294 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01d631f4-4bde-4f10-aeaf-128e42e45063 admin admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:13:08.563702 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.fernet_utils [None req-01d631f4-4bde-4f10-aeaf-128e42e45063 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92057) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:13:08.619885 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01d631f4-4bde-4f10-aeaf-128e42e45063 admin admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:13:08.623760 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1768a14-9eec-4c0e-acd3-88895d3771a4 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2eb83ca38e79449095f3a2f947bbb06a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9c389ae65aaa4542bb5a5c4fd0bacf5b', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:13:08.624348 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1768a14-9eec-4c0e-acd3-88895d3771a4 service neutron] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:13:08.624635 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1768a14-9eec-4c0e-acd3-88895d3771a4 service neutron] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:13:08.624907 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1768a14-9eec-4c0e-acd3-88895d3771a4 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:13:08.634367 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1768a14-9eec-4c0e-acd3-88895d3771a4 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:13:08.636289 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1768a14-9eec-4c0e-acd3-88895d3771a4 service neutron] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:13:08.645352 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 1421/2828] 10.222.0.30 () {60 vars in 1365 bytes} [Mon May 23 13:13:08 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 84 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) May 23 13:13:09.045710 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48d82e1e-83ab-407b-9784-dcd9b393d503 None None] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:13:09.047214 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.fernet_utils [None req-48d82e1e-83ab-407b-9784-dcd9b393d503 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92056) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:13:09.061100 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48d82e1e-83ab-407b-9784-dcd9b393d503 None None] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:13:09.064425 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1ba8d3e-8580-47a5-a1f2-17d2c591e03f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:13:09.065291 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1ba8d3e-8580-47a5-a1f2-17d2c591e03f admin admin] REQUEST_METHOD: `DELETE` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:13:09.065578 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1ba8d3e-8580-47a5-a1f2-17d2c591e03f admin admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:13:09.065864 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1ba8d3e-8580-47a5-a1f2-17d2c591e03f admin admin] PATH_INFO: `/v3/projects/e85e2312cfa54696a35ae5d6f688857e` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:13:09.067082 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1ba8d3e-8580-47a5-a1f2-17d2c591e03f admin admin] RBAC: Authorizing `identity:delete_project(project_id=e85e2312cfa54696a35ae5d6f688857e)` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:13:09.069028 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1ba8d3e-8580-47a5-a1f2-17d2c591e03f admin admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:13:09.171071 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-c1ba8d3e-8580-47a5-a1f2-17d2c591e03f admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'e85e2312cfa54696a35ae5d6f688857e', 'request_id': 'req-c1ba8d3e-8580-47a5-a1f2-17d2c591e03f'} {{(pid=92056) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:13:09.244640 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-c1ba8d3e-8580-47a5-a1f2-17d2c591e03f admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'e85e2312cfa54696a35ae5d6f688857e', 'request_id': 'req-c1ba8d3e-8580-47a5-a1f2-17d2c591e03f'} {{(pid=92056) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:13:09.251455 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-c1ba8d3e-8580-47a5-a1f2-17d2c591e03f admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'e85e2312cfa54696a35ae5d6f688857e', 'request_id': 'req-c1ba8d3e-8580-47a5-a1f2-17d2c591e03f'} {{(pid=92056) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:13:09.262159 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-c1ba8d3e-8580-47a5-a1f2-17d2c591e03f admin admin] The token cache is being invalidate because project e85e2312cfa54696a35ae5d6f688857e was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=92056) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} May 23 13:13:09.262445 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-c1ba8d3e-8580-47a5-a1f2-17d2c591e03f admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=92056) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:13:09.263077 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-c1ba8d3e-8580-47a5-a1f2-17d2c591e03f admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=92056) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:13:09.264645 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 1408/2829] 10.222.0.30 () {58 vars in 1232 bytes} [Mon May 23 13:13:09 2022] DELETE /identity/v3/projects/e85e2312cfa54696a35ae5d6f688857e => generated 0 bytes in 219 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) May 23 13:13:09.269494 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5795d136-95b6-4106-9de5-1f7f4a2d3bb3 None None] REQUEST_METHOD: `POST` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:13:09.269780 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5795d136-95b6-4106-9de5-1f7f4a2d3bb3 None None] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:13:09.270090 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5795d136-95b6-4106-9de5-1f7f4a2d3bb3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:13:09.288368 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.auth.core [None req-5795d136-95b6-4106-9de5-1f7f4a2d3bb3 None None] MFA Rules not processed for user `0f7119f8fa5c4445977671140ded91a4`. Rule list: `[]` (Enabled: `True`). {{(pid=92057) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 13:13:09.289552 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.fernet_utils [None req-5795d136-95b6-4106-9de5-1f7f4a2d3bb3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92057) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:13:09.338639 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 13:13:09.349631 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 1422/2830] 10.222.0.30 () {58 vars in 957 bytes} [Mon May 23 13:13:09 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 81 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) May 23 13:13:09.413680 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c564c37-eeea-4883-86d5-7cb9625a6e14 None None] REQUEST_METHOD: `POST` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:13:09.414001 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c564c37-eeea-4883-86d5-7cb9625a6e14 None None] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:13:09.414279 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c564c37-eeea-4883-86d5-7cb9625a6e14 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:13:09.434216 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.auth.core [None req-8c564c37-eeea-4883-86d5-7cb9625a6e14 None None] MFA Rules not processed for user `0f7119f8fa5c4445977671140ded91a4`. Rule list: `[]` (Enabled: `True`). {{(pid=92056) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 13:13:09.435348 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.fernet_utils [None req-8c564c37-eeea-4883-86d5-7cb9625a6e14 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92056) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:13:09.443668 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 1409/2831] 10.222.0.30 () {58 vars in 957 bytes} [Mon May 23 13:13:09 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) May 23 13:13:09.448749 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5795d136-95b6-4106-9de5-1f7f4a2d3bb3 None None] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:13:09.458590 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5795d136-95b6-4106-9de5-1f7f4a2d3bb3 None None] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:13:09.464277 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68f40ed2-67a2-4c7d-9f66-822061f8202f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:13:09.464931 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68f40ed2-67a2-4c7d-9f66-822061f8202f admin admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:13:09.465274 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68f40ed2-67a2-4c7d-9f66-822061f8202f admin admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:13:09.465554 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68f40ed2-67a2-4c7d-9f66-822061f8202f admin admin] PATH_INFO: `/v3/domains` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:13:09.466309 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68f40ed2-67a2-4c7d-9f66-822061f8202f admin admin] RBAC: Adding query filter params (name=Default) {{(pid=92057) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} May 23 13:13:09.466621 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68f40ed2-67a2-4c7d-9f66-822061f8202f admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:13:09.468257 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68f40ed2-67a2-4c7d-9f66-822061f8202f admin admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:13:09.480253 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 1423/2832] 10.222.0.30 () {58 vars in 1152 bytes} [Mon May 23 13:13:09 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) May 23 13:13:09.484416 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c564c37-eeea-4883-86d5-7cb9625a6e14 None None] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:13:09.493957 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c564c37-eeea-4883-86d5-7cb9625a6e14 None None] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:13:09.498733 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab54dc77-aa03-409b-9387-6ea7d86b03a9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:13:09.499378 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab54dc77-aa03-409b-9387-6ea7d86b03a9 admin admin] REQUEST_METHOD: `POST` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:13:09.499662 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab54dc77-aa03-409b-9387-6ea7d86b03a9 admin admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:13:09.499938 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab54dc77-aa03-409b-9387-6ea7d86b03a9 admin admin] PATH_INFO: `/v3/projects` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:13:09.500626 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab54dc77-aa03-409b-9387-6ea7d86b03a9 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:13:09.502565 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab54dc77-aa03-409b-9387-6ea7d86b03a9 admin admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:13:09.588672 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 1410/2833] 10.222.0.30 () {60 vars in 1152 bytes} [Mon May 23 13:13:09 2022] POST /identity/v3/projects => generated 373 bytes in 105 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) May 23 13:13:09.592841 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68f40ed2-67a2-4c7d-9f66-822061f8202f admin admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:13:09.602817 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68f40ed2-67a2-4c7d-9f66-822061f8202f admin admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:13:09.659752 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f95c316-0ab4-46d4-99ce-65930cb2c5bc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:13:09.660409 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f95c316-0ab4-46d4-99ce-65930cb2c5bc admin admin] REQUEST_METHOD: `POST` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:13:09.660696 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f95c316-0ab4-46d4-99ce-65930cb2c5bc admin admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:13:09.660975 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f95c316-0ab4-46d4-99ce-65930cb2c5bc admin admin] PATH_INFO: `/v3/users` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:13:09.661672 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f95c316-0ab4-46d4-99ce-65930cb2c5bc admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:13:09.663610 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f95c316-0ab4-46d4-99ce-65930cb2c5bc admin admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:13:09.730556 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 1424/2834] 10.222.0.30 () {60 vars in 1143 bytes} [Mon May 23 13:13:09 2022] POST /identity/v3/users => generated 287 bytes in 138 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) May 23 13:13:09.741531 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab54dc77-aa03-409b-9387-6ea7d86b03a9 admin admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:13:09.751045 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab54dc77-aa03-409b-9387-6ea7d86b03a9 admin admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:13:09.755827 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3db1d56-36c7-42b8-b134-6b21f82314d6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:13:09.756486 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3db1d56-36c7-42b8-b134-6b21f82314d6 admin admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:13:09.756770 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3db1d56-36c7-42b8-b134-6b21f82314d6 admin admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:13:09.757049 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3db1d56-36c7-42b8-b134-6b21f82314d6 admin admin] PATH_INFO: `/v3/roles` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:13:09.757780 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3db1d56-36c7-42b8-b134-6b21f82314d6 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:13:09.759962 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3db1d56-36c7-42b8-b134-6b21f82314d6 admin admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:13:09.770705 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 1411/2835] 10.222.0.30 () {58 vars in 1121 bytes} [Mon May 23 13:13:09 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) May 23 13:13:09.775769 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f95c316-0ab4-46d4-99ce-65930cb2c5bc admin admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:13:09.785633 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f95c316-0ab4-46d4-99ce-65930cb2c5bc admin admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:13:09.790277 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2f098a9-badf-4026-bb0d-04b3478a8f01 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:13:09.791022 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2f098a9-badf-4026-bb0d-04b3478a8f01 admin admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:13:09.791309 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2f098a9-badf-4026-bb0d-04b3478a8f01 admin admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:13:09.791593 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2f098a9-badf-4026-bb0d-04b3478a8f01 admin admin] PATH_INFO: `/v3/roles` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:13:09.792307 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2f098a9-badf-4026-bb0d-04b3478a8f01 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:13:09.793930 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2f098a9-badf-4026-bb0d-04b3478a8f01 admin admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:13:09.803632 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 1425/2836] 10.222.0.30 () {58 vars in 1121 bytes} [Mon May 23 13:13:09 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) May 23 13:13:09.808167 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3db1d56-36c7-42b8-b134-6b21f82314d6 admin admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:13:09.817959 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3db1d56-36c7-42b8-b134-6b21f82314d6 admin admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:13:09.823110 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2799d122-4499-422a-b845-8f01193a2414 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:13:09.823971 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2799d122-4499-422a-b845-8f01193a2414 admin admin] REQUEST_METHOD: `PUT` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:13:09.824249 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2799d122-4499-422a-b845-8f01193a2414 admin admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:13:09.824520 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2799d122-4499-422a-b845-8f01193a2414 admin admin] PATH_INFO: `/v3/projects/f7c3ade76fb9432d813a5288f22d823f/users/597df5aeba214e4da8e18416671c7681/roles/80fb22507c9a4ff09d44d8f12f6fff1a` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:13:09.838746 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2799d122-4499-422a-b845-8f01193a2414 admin admin] RBAC: Authorizing `identity:create_grant(project_id=f7c3ade76fb9432d813a5288f22d823f, user_id=597df5aeba214e4da8e18416671c7681, role_id=80fb22507c9a4ff09d44d8f12f6fff1a)` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:13:09.841370 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2799d122-4499-422a-b845-8f01193a2414 admin admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:13:09.895542 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 1412/2837] 10.222.0.30 () {60 vars in 1482 bytes} [Mon May 23 13:13:09 2022] PUT /identity/v3/projects/f7c3ade76fb9432d813a5288f22d823f/users/597df5aeba214e4da8e18416671c7681/roles/80fb22507c9a4ff09d44d8f12f6fff1a => generated 0 bytes in 88 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) May 23 13:13:09.906384 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22c13d31-eea4-42d3-8836-9199bf078ec2 None None] REQUEST_METHOD: `POST` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:13:09.906702 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22c13d31-eea4-42d3-8836-9199bf078ec2 None None] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:13:09.906981 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22c13d31-eea4-42d3-8836-9199bf078ec2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:13:09.933272 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.auth.core [None req-22c13d31-eea4-42d3-8836-9199bf078ec2 None None] MFA Rules not processed for user `597df5aeba214e4da8e18416671c7681`. Rule list: `[]` (Enabled: `True`). {{(pid=92057) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 13:13:09.934565 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.fernet_utils [None req-22c13d31-eea4-42d3-8836-9199bf078ec2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92057) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:13:09.997175 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 1426/2838] 10.222.0.30 () {58 vars in 957 bytes} [Mon May 23 13:13:09 2022] POST /identity/v3/auth/tokens => generated 3240 bytes in 92 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) May 23 13:13:10.008045 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2799d122-4499-422a-b845-8f01193a2414 admin admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:13:10.010419 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.fernet_utils [None req-2799d122-4499-422a-b845-8f01193a2414 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92056) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:13:10.030369 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 13:13:10.066012 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2799d122-4499-422a-b845-8f01193a2414 admin admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:13:10.069592 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-716bf9c7-5b2b-478e-b34f-d22747ca2059 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1666097f8ed545c6b0e4809c93e4775a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9c389ae65aaa4542bb5a5c4fd0bacf5b', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:13:10.070251 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-716bf9c7-5b2b-478e-b34f-d22747ca2059 service cinder] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:13:10.070531 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-716bf9c7-5b2b-478e-b34f-d22747ca2059 service cinder] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:13:10.073832 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-716bf9c7-5b2b-478e-b34f-d22747ca2059 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:13:10.079835 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-716bf9c7-5b2b-478e-b34f-d22747ca2059 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:13:10.081251 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-716bf9c7-5b2b-478e-b34f-d22747ca2059 service cinder] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:13:10.089766 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 1413/2839] 10.222.0.30 () {60 vars in 1365 bytes} [Mon May 23 13:13:10 2022] GET /identity/v3/auth/tokens => generated 3240 bytes in 82 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) May 23 13:13:10.114664 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22c13d31-eea4-42d3-8836-9199bf078ec2 None None] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:13:10.169851 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 13:13:10.182716 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22c13d31-eea4-42d3-8836-9199bf078ec2 None None] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:13:10.187425 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3db94403-6f93-4b9a-b3d9-7046078e04cc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:13:10.188297 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3db94403-6f93-4b9a-b3d9-7046078e04cc admin admin] REQUEST_METHOD: `DELETE` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:13:10.188579 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3db94403-6f93-4b9a-b3d9-7046078e04cc admin admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:13:10.188855 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3db94403-6f93-4b9a-b3d9-7046078e04cc admin admin] PATH_INFO: `/v3/users/597df5aeba214e4da8e18416671c7681` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:13:10.192567 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3db94403-6f93-4b9a-b3d9-7046078e04cc admin admin] RBAC: Authorizing `identity:delete_user(user_id=597df5aeba214e4da8e18416671c7681)` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:13:10.192567 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3db94403-6f93-4b9a-b3d9-7046078e04cc admin admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:13:10.325473 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [None req-3db94403-6f93-4b9a-b3d9-7046078e04cc admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '597df5aeba214e4da8e18416671c7681'} {{(pid=92057) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:13:10.397477 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [None req-3db94403-6f93-4b9a-b3d9-7046078e04cc admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '597df5aeba214e4da8e18416671c7681'} {{(pid=92057) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:13:10.404774 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [None req-3db94403-6f93-4b9a-b3d9-7046078e04cc admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '597df5aeba214e4da8e18416671c7681'} {{(pid=92057) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:13:10.405425 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [None req-3db94403-6f93-4b9a-b3d9-7046078e04cc admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '597df5aeba214e4da8e18416671c7681'} {{(pid=92057) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:13:10.418183 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [None req-3db94403-6f93-4b9a-b3d9-7046078e04cc admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '597df5aeba214e4da8e18416671c7681'} {{(pid=92057) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:13:10.424079 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 1427/2840] 10.222.0.30 () {58 vars in 1223 bytes} [Mon May 23 13:13:10 2022] DELETE /identity/v3/users/597df5aeba214e4da8e18416671c7681 => generated 0 bytes in 310 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) May 23 13:13:10.428686 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e277be08-aae4-4e14-ab1f-3aba65b010af None None] REQUEST_METHOD: `POST` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:13:10.428977 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e277be08-aae4-4e14-ab1f-3aba65b010af None None] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:13:10.429253 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e277be08-aae4-4e14-ab1f-3aba65b010af None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:13:10.446535 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.auth.core [None req-e277be08-aae4-4e14-ab1f-3aba65b010af None None] MFA Rules not processed for user `0f7119f8fa5c4445977671140ded91a4`. Rule list: `[]` (Enabled: `True`). {{(pid=92056) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 13:13:10.447649 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.fernet_utils [None req-e277be08-aae4-4e14-ab1f-3aba65b010af None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92056) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:13:10.513414 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 1414/2841] 10.222.0.30 () {58 vars in 957 bytes} [Mon May 23 13:13:10 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) May 23 13:13:10.523166 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3db94403-6f93-4b9a-b3d9-7046078e04cc admin admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:13:10.525066 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.fernet_utils [None req-3db94403-6f93-4b9a-b3d9-7046078e04cc admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92057) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:13:10.583492 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3db94403-6f93-4b9a-b3d9-7046078e04cc admin admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:13:10.587546 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b18b7d21-7a05-4ee1-bf17-9ae223330e8a service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2eb83ca38e79449095f3a2f947bbb06a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9c389ae65aaa4542bb5a5c4fd0bacf5b', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:13:10.588687 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b18b7d21-7a05-4ee1-bf17-9ae223330e8a service neutron] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:13:10.588974 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b18b7d21-7a05-4ee1-bf17-9ae223330e8a service neutron] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:13:10.589251 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b18b7d21-7a05-4ee1-bf17-9ae223330e8a service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:13:10.598160 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b18b7d21-7a05-4ee1-bf17-9ae223330e8a service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:13:10.599605 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b18b7d21-7a05-4ee1-bf17-9ae223330e8a service neutron] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:13:10.610266 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 1428/2842] 10.222.0.30 () {60 vars in 1365 bytes} [Mon May 23 13:13:10 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 88 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) May 23 13:13:11.029072 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e277be08-aae4-4e14-ab1f-3aba65b010af None None] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:13:11.030516 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.fernet_utils [None req-e277be08-aae4-4e14-ab1f-3aba65b010af None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92056) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:13:11.042736 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e277be08-aae4-4e14-ab1f-3aba65b010af None None] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:13:11.047298 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f491e884-7a34-412f-b55f-b21285f363c6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:13:11.048152 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f491e884-7a34-412f-b55f-b21285f363c6 admin admin] REQUEST_METHOD: `DELETE` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:13:11.048429 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f491e884-7a34-412f-b55f-b21285f363c6 admin admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:13:11.048701 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f491e884-7a34-412f-b55f-b21285f363c6 admin admin] PATH_INFO: `/v3/projects/f7c3ade76fb9432d813a5288f22d823f` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:13:11.049900 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f491e884-7a34-412f-b55f-b21285f363c6 admin admin] RBAC: Authorizing `identity:delete_project(project_id=f7c3ade76fb9432d813a5288f22d823f)` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:13:11.051827 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f491e884-7a34-412f-b55f-b21285f363c6 admin admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:13:11.158678 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-f491e884-7a34-412f-b55f-b21285f363c6 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'f7c3ade76fb9432d813a5288f22d823f', 'request_id': 'req-f491e884-7a34-412f-b55f-b21285f363c6'} {{(pid=92056) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:13:11.226196 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-f491e884-7a34-412f-b55f-b21285f363c6 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'f7c3ade76fb9432d813a5288f22d823f', 'request_id': 'req-f491e884-7a34-412f-b55f-b21285f363c6'} {{(pid=92056) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:13:11.233495 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-f491e884-7a34-412f-b55f-b21285f363c6 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'f7c3ade76fb9432d813a5288f22d823f', 'request_id': 'req-f491e884-7a34-412f-b55f-b21285f363c6'} {{(pid=92056) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:13:11.244424 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-f491e884-7a34-412f-b55f-b21285f363c6 admin admin] The token cache is being invalidate because project f7c3ade76fb9432d813a5288f22d823f was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=92056) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} May 23 13:13:11.244734 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-f491e884-7a34-412f-b55f-b21285f363c6 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=92056) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:13:11.245416 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-f491e884-7a34-412f-b55f-b21285f363c6 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=92056) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:13:11.247015 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 1415/2843] 10.222.0.30 () {58 vars in 1232 bytes} [Mon May 23 13:13:11 2022] DELETE /identity/v3/projects/f7c3ade76fb9432d813a5288f22d823f => generated 0 bytes in 219 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) May 23 13:13:20.606002 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=56,keystone:DELETE=10,keystone:INSERT=1 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 13:13:21.241151 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=72,keystone:DELETE=7,keystone:INSERT=1 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 13:14:04.531004 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b18b7d21-7a05-4ee1-bf17-9ae223330e8a service neutron] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:14:04.532442 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.fernet_utils [None req-b18b7d21-7a05-4ee1-bf17-9ae223330e8a service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92057) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:14:04.625507 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b18b7d21-7a05-4ee1-bf17-9ae223330e8a service neutron] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:14:04.684849 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7aef73ae-aeed-4def-9fb7-3e035baf2872 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd5fb51f8d0fa46439cb29666970b885b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9c389ae65aaa4542bb5a5c4fd0bacf5b', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'service', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:14:04.685501 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7aef73ae-aeed-4def-9fb7-3e035baf2872 service placement] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:14:04.685787 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7aef73ae-aeed-4def-9fb7-3e035baf2872 service placement] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:14:04.686105 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7aef73ae-aeed-4def-9fb7-3e035baf2872 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:14:04.687878 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.fernet_utils [None req-7aef73ae-aeed-4def-9fb7-3e035baf2872 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92057) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:14:04.700085 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7aef73ae-aeed-4def-9fb7-3e035baf2872 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:14:04.701677 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7aef73ae-aeed-4def-9fb7-3e035baf2872 service placement] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:14:04.711607 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 1429/2844] 10.222.0.30 () {60 vars in 1365 bytes} [Mon May 23 13:14:04 2022] GET /identity/v3/auth/tokens => generated 3298 bytes in 181 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) May 23 13:14:14.707302 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=67 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 13:14:55.528596 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce9fb3b0-71a4-4c19-ab70-4386545ee0db None None] REQUEST_METHOD: `POST` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:14:55.529127 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce9fb3b0-71a4-4c19-ab70-4386545ee0db None None] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:14:55.529127 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce9fb3b0-71a4-4c19-ab70-4386545ee0db None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:14:55.587847 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.auth.core [None req-ce9fb3b0-71a4-4c19-ab70-4386545ee0db None None] MFA Rules not processed for user `6d21a56126fd4ccd837c33ccc08ab609`. Rule list: `[]` (Enabled: `True`). {{(pid=92056) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 13:14:55.589036 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.fernet_utils [None req-ce9fb3b0-71a4-4c19-ab70-4386545ee0db None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92056) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:14:55.659815 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 1416/2845] 10.222.0.30 () {58 vars in 967 bytes} [Mon May 23 13:14:55 2022] POST /identity/v3/auth/tokens => generated 3185 bytes in 132 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) May 23 13:14:55.700148 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7aef73ae-aeed-4def-9fb7-3e035baf2872 service placement] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:14:55.701617 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.fernet_utils [None req-7aef73ae-aeed-4def-9fb7-3e035baf2872 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92057) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:14:55.790811 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7aef73ae-aeed-4def-9fb7-3e035baf2872 service placement] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:14:55.795779 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-861fe4d4-6e0e-41d2-8830-c02b1c57ef47 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '138d91edb99f450e846765f9eef00d62', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9c389ae65aaa4542bb5a5c4fd0bacf5b', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'service', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:14:55.796422 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-861fe4d4-6e0e-41d2-8830-c02b1c57ef47 service swift] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:14:55.796710 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-861fe4d4-6e0e-41d2-8830-c02b1c57ef47 service swift] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:14:55.796992 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-861fe4d4-6e0e-41d2-8830-c02b1c57ef47 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:14:55.805705 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-861fe4d4-6e0e-41d2-8830-c02b1c57ef47 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:14:55.807335 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-861fe4d4-6e0e-41d2-8830-c02b1c57ef47 service swift] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:14:55.816046 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 1430/2846] 10.222.0.30 () {60 vars in 1384 bytes} [Mon May 23 13:14:55 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 116 msecs (HTTP/1.1 200) 6 headers in 379 bytes (2 switches on core 0) May 23 13:15:03.652576 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce9fb3b0-71a4-4c19-ab70-4386545ee0db None None] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:15:03.654611 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.fernet_utils [None req-ce9fb3b0-71a4-4c19-ab70-4386545ee0db None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92056) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:15:03.731533 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce9fb3b0-71a4-4c19-ab70-4386545ee0db None None] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:15:03.736157 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-063e1f6c-f1f1-43d2-bf7d-b24f6519edbb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:15:03.737110 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-063e1f6c-f1f1-43d2-bf7d-b24f6519edbb admin admin] REQUEST_METHOD: `DELETE` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:15:03.737392 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-063e1f6c-f1f1-43d2-bf7d-b24f6519edbb admin admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:15:03.737664 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-063e1f6c-f1f1-43d2-bf7d-b24f6519edbb admin admin] PATH_INFO: `/v3/users/20d9973f32f847a6a54a60405beabdbf` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:15:03.738949 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-063e1f6c-f1f1-43d2-bf7d-b24f6519edbb admin admin] RBAC: Authorizing `identity:delete_user(user_id=20d9973f32f847a6a54a60405beabdbf)` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:15:03.740880 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-063e1f6c-f1f1-43d2-bf7d-b24f6519edbb admin admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:15:03.929515 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-063e1f6c-f1f1-43d2-bf7d-b24f6519edbb admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '20d9973f32f847a6a54a60405beabdbf'} {{(pid=92056) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:15:03.930229 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-063e1f6c-f1f1-43d2-bf7d-b24f6519edbb admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '20d9973f32f847a6a54a60405beabdbf'} {{(pid=92056) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:15:04.006797 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-063e1f6c-f1f1-43d2-bf7d-b24f6519edbb admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '20d9973f32f847a6a54a60405beabdbf'} {{(pid=92056) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:15:04.011793 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=91,keystone:DELETE=8,keystone:INSERT=1 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 13:15:04.021329 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-063e1f6c-f1f1-43d2-bf7d-b24f6519edbb admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '20d9973f32f847a6a54a60405beabdbf'} {{(pid=92056) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:15:04.028202 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-063e1f6c-f1f1-43d2-bf7d-b24f6519edbb admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '20d9973f32f847a6a54a60405beabdbf'} {{(pid=92056) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:15:04.034730 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 1417/2847] 10.222.0.30 () {58 vars in 1223 bytes} [Mon May 23 13:15:03 2022] DELETE /identity/v3/users/20d9973f32f847a6a54a60405beabdbf => generated 0 bytes in 383 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) May 23 13:15:04.311574 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-861fe4d4-6e0e-41d2-8830-c02b1c57ef47 service swift] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:15:04.322850 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-861fe4d4-6e0e-41d2-8830-c02b1c57ef47 service swift] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:15:04.382325 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9aaccec-d5b6-4fad-8ec6-5ecfd0d6a2f4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:15:04.383185 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9aaccec-d5b6-4fad-8ec6-5ecfd0d6a2f4 admin admin] REQUEST_METHOD: `DELETE` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:15:04.383464 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9aaccec-d5b6-4fad-8ec6-5ecfd0d6a2f4 admin admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:15:04.383754 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9aaccec-d5b6-4fad-8ec6-5ecfd0d6a2f4 admin admin] PATH_INFO: `/v3/projects/1f2fed134f274a6ebe5ffe183205e269` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:15:04.384977 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9aaccec-d5b6-4fad-8ec6-5ecfd0d6a2f4 admin admin] RBAC: Authorizing `identity:delete_project(project_id=1f2fed134f274a6ebe5ffe183205e269)` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:15:04.386920 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9aaccec-d5b6-4fad-8ec6-5ecfd0d6a2f4 admin admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:15:04.490657 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [None req-d9aaccec-d5b6-4fad-8ec6-5ecfd0d6a2f4 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '1f2fed134f274a6ebe5ffe183205e269', 'request_id': 'req-d9aaccec-d5b6-4fad-8ec6-5ecfd0d6a2f4'} {{(pid=92057) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:15:04.496848 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [None req-d9aaccec-d5b6-4fad-8ec6-5ecfd0d6a2f4 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '1f2fed134f274a6ebe5ffe183205e269', 'request_id': 'req-d9aaccec-d5b6-4fad-8ec6-5ecfd0d6a2f4'} {{(pid=92057) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:15:04.566958 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [None req-d9aaccec-d5b6-4fad-8ec6-5ecfd0d6a2f4 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '1f2fed134f274a6ebe5ffe183205e269', 'request_id': 'req-d9aaccec-d5b6-4fad-8ec6-5ecfd0d6a2f4'} {{(pid=92057) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:15:04.575141 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 13:15:04.578803 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [None req-d9aaccec-d5b6-4fad-8ec6-5ecfd0d6a2f4 admin admin] The token cache is being invalidate because project 1f2fed134f274a6ebe5ffe183205e269 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=92057) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} May 23 13:15:04.579090 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [None req-d9aaccec-d5b6-4fad-8ec6-5ecfd0d6a2f4 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=92057) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:15:04.579700 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [None req-d9aaccec-d5b6-4fad-8ec6-5ecfd0d6a2f4 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=92057) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:15:04.583132 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 1431/2848] 10.222.0.30 () {58 vars in 1232 bytes} [Mon May 23 13:15:04 2022] DELETE /identity/v3/projects/1f2fed134f274a6ebe5ffe183205e269 => generated 0 bytes in 272 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) May 23 13:15:04.587218 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-063e1f6c-f1f1-43d2-bf7d-b24f6519edbb admin admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:15:04.588762 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.fernet_utils [None req-063e1f6c-f1f1-43d2-bf7d-b24f6519edbb admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92056) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:15:04.603441 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-063e1f6c-f1f1-43d2-bf7d-b24f6519edbb admin admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:15:04.660264 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5fbafef6-c5ed-42bf-bfbf-dc607d5ab88e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:15:04.661127 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fbafef6-c5ed-42bf-bfbf-dc607d5ab88e admin admin] REQUEST_METHOD: `DELETE` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:15:04.661410 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fbafef6-c5ed-42bf-bfbf-dc607d5ab88e admin admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:15:04.661685 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fbafef6-c5ed-42bf-bfbf-dc607d5ab88e admin admin] PATH_INFO: `/v3/users/03422526f5b843d49823f4f2316f7f56` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:15:04.663148 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5fbafef6-c5ed-42bf-bfbf-dc607d5ab88e admin admin] RBAC: Authorizing `identity:delete_user(user_id=03422526f5b843d49823f4f2316f7f56)` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:15:04.665083 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5fbafef6-c5ed-42bf-bfbf-dc607d5ab88e admin admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:15:04.860978 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-5fbafef6-c5ed-42bf-bfbf-dc607d5ab88e admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '03422526f5b843d49823f4f2316f7f56'} {{(pid=92056) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:15:04.861653 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-5fbafef6-c5ed-42bf-bfbf-dc607d5ab88e admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '03422526f5b843d49823f4f2316f7f56'} {{(pid=92056) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:15:04.926557 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-5fbafef6-c5ed-42bf-bfbf-dc607d5ab88e admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '03422526f5b843d49823f4f2316f7f56'} {{(pid=92056) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:15:04.939335 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-5fbafef6-c5ed-42bf-bfbf-dc607d5ab88e admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '03422526f5b843d49823f4f2316f7f56'} {{(pid=92056) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:15:04.946827 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-5fbafef6-c5ed-42bf-bfbf-dc607d5ab88e admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '03422526f5b843d49823f4f2316f7f56'} {{(pid=92056) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:15:04.952743 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 1418/2849] 10.222.0.30 () {58 vars in 1223 bytes} [Mon May 23 13:15:04 2022] DELETE /identity/v3/users/03422526f5b843d49823f4f2316f7f56 => generated 0 bytes in 366 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) May 23 13:15:05.167471 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9aaccec-d5b6-4fad-8ec6-5ecfd0d6a2f4 admin admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:15:05.168928 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.fernet_utils [None req-d9aaccec-d5b6-4fad-8ec6-5ecfd0d6a2f4 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92057) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:15:05.182525 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9aaccec-d5b6-4fad-8ec6-5ecfd0d6a2f4 admin admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:15:05.241001 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3af68d95-c8cd-48be-9ce7-53a30c3e22e6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:15:05.241858 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3af68d95-c8cd-48be-9ce7-53a30c3e22e6 admin admin] REQUEST_METHOD: `DELETE` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:15:05.242166 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3af68d95-c8cd-48be-9ce7-53a30c3e22e6 admin admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:15:05.242443 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3af68d95-c8cd-48be-9ce7-53a30c3e22e6 admin admin] PATH_INFO: `/v3/projects/5ec3d4c508314b668b685921169854d6` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:15:05.243639 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3af68d95-c8cd-48be-9ce7-53a30c3e22e6 admin admin] RBAC: Authorizing `identity:delete_project(project_id=5ec3d4c508314b668b685921169854d6)` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:15:05.245542 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3af68d95-c8cd-48be-9ce7-53a30c3e22e6 admin admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:15:05.356099 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [None req-3af68d95-c8cd-48be-9ce7-53a30c3e22e6 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '5ec3d4c508314b668b685921169854d6', 'request_id': 'req-3af68d95-c8cd-48be-9ce7-53a30c3e22e6'} {{(pid=92057) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:15:05.361581 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [None req-3af68d95-c8cd-48be-9ce7-53a30c3e22e6 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '5ec3d4c508314b668b685921169854d6', 'request_id': 'req-3af68d95-c8cd-48be-9ce7-53a30c3e22e6'} {{(pid=92057) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:15:05.446577 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [None req-3af68d95-c8cd-48be-9ce7-53a30c3e22e6 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '5ec3d4c508314b668b685921169854d6', 'request_id': 'req-3af68d95-c8cd-48be-9ce7-53a30c3e22e6'} {{(pid=92057) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:15:05.459021 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [None req-3af68d95-c8cd-48be-9ce7-53a30c3e22e6 admin admin] The token cache is being invalidate because project 5ec3d4c508314b668b685921169854d6 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=92057) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} May 23 13:15:05.459308 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [None req-3af68d95-c8cd-48be-9ce7-53a30c3e22e6 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=92057) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:15:05.459927 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [None req-3af68d95-c8cd-48be-9ce7-53a30c3e22e6 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=92057) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:15:05.461462 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 1432/2850] 10.222.0.30 () {58 vars in 1232 bytes} [Mon May 23 13:15:05 2022] DELETE /identity/v3/projects/5ec3d4c508314b668b685921169854d6 => generated 0 bytes in 295 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) May 23 13:15:14.949325 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:DELETE=12,keystone:SELECT=62,keystone:INSERT=1 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 13:15:15.455027 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=51,keystone:DELETE=7,keystone:INSERT=1 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 13:15:34.328086 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5fbafef6-c5ed-42bf-bfbf-dc607d5ab88e admin admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:15:34.329394 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.fernet_utils [None req-5fbafef6-c5ed-42bf-bfbf-dc607d5ab88e admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92056) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:15:34.387977 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5fbafef6-c5ed-42bf-bfbf-dc607d5ab88e admin admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:15:34.436352 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dabad3bd-3944-420c-8f65-ecbbd56295e1 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1666097f8ed545c6b0e4809c93e4775a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9c389ae65aaa4542bb5a5c4fd0bacf5b', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:15:34.436990 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dabad3bd-3944-420c-8f65-ecbbd56295e1 service cinder] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:15:34.437269 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dabad3bd-3944-420c-8f65-ecbbd56295e1 service cinder] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:15:34.437544 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dabad3bd-3944-420c-8f65-ecbbd56295e1 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:15:34.439406 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.fernet_utils [None req-dabad3bd-3944-420c-8f65-ecbbd56295e1 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92056) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:15:34.521723 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dabad3bd-3944-420c-8f65-ecbbd56295e1 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:15:34.523181 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dabad3bd-3944-420c-8f65-ecbbd56295e1 service cinder] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:15:34.544379 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 1419/2851] 10.222.0.30 () {60 vars in 1365 bytes} [Mon May 23 13:15:34 2022] GET /identity/v3/auth/tokens => generated 3362 bytes in 217 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) May 23 13:15:37.375327 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3af68d95-c8cd-48be-9ce7-53a30c3e22e6 admin admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:15:37.376716 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.fernet_utils [None req-3af68d95-c8cd-48be-9ce7-53a30c3e22e6 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92057) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:15:37.435337 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3af68d95-c8cd-48be-9ce7-53a30c3e22e6 admin admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:15:37.438939 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-494eb614-fdb2-48dd-84e3-9167e7446817 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2eb83ca38e79449095f3a2f947bbb06a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9c389ae65aaa4542bb5a5c4fd0bacf5b', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:15:37.439575 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-494eb614-fdb2-48dd-84e3-9167e7446817 service neutron] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:15:37.439858 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-494eb614-fdb2-48dd-84e3-9167e7446817 service neutron] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:15:37.440136 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-494eb614-fdb2-48dd-84e3-9167e7446817 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:15:37.441895 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.fernet_utils [None req-494eb614-fdb2-48dd-84e3-9167e7446817 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92057) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:15:37.456052 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-494eb614-fdb2-48dd-84e3-9167e7446817 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:15:37.457479 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-494eb614-fdb2-48dd-84e3-9167e7446817 service neutron] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:15:37.468840 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 1433/2852] 10.222.0.30 () {60 vars in 1365 bytes} [Mon May 23 13:15:37 2022] GET /identity/v3/auth/tokens => generated 3362 bytes in 94 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) May 23 13:15:44.541696 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=83 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 13:15:47.464054 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=26 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 13:16:03.547552 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dabad3bd-3944-420c-8f65-ecbbd56295e1 service cinder] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:16:03.557067 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dabad3bd-3944-420c-8f65-ecbbd56295e1 service cinder] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:16:03.560684 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-907309fd-b93b-4fd5-ae2e-3745d7509197 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2eb83ca38e79449095f3a2f947bbb06a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9c389ae65aaa4542bb5a5c4fd0bacf5b', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:16:03.561345 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-907309fd-b93b-4fd5-ae2e-3745d7509197 service neutron] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:16:03.561627 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-907309fd-b93b-4fd5-ae2e-3745d7509197 service neutron] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:16:03.561938 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-907309fd-b93b-4fd5-ae2e-3745d7509197 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:16:03.563688 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.fernet_utils [None req-907309fd-b93b-4fd5-ae2e-3745d7509197 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92056) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:16:03.574069 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-907309fd-b93b-4fd5-ae2e-3745d7509197 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:16:03.575489 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-907309fd-b93b-4fd5-ae2e-3745d7509197 service neutron] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:16:03.583461 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 1420/2853] 10.222.0.30 () {60 vars in 1365 bytes} [Mon May 23 13:16:03 2022] GET /identity/v3/auth/tokens => generated 3111 bytes in 36 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) May 23 13:16:13.579890 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=6 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 13:16:56.463792 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-494eb614-fdb2-48dd-84e3-9167e7446817 service neutron] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:16:56.465534 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.fernet_utils [None req-494eb614-fdb2-48dd-84e3-9167e7446817 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92057) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:16:56.479177 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-494eb614-fdb2-48dd-84e3-9167e7446817 service neutron] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:16:56.484838 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0a3026d-109f-4f7f-8c2d-65045498e3c3 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cbe9931d94d44cc7ace0e83a7e5015fc', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9c389ae65aaa4542bb5a5c4fd0bacf5b', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'member', 'reader', 'service', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:16:56.485628 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0a3026d-109f-4f7f-8c2d-65045498e3c3 service nova] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:16:56.485938 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0a3026d-109f-4f7f-8c2d-65045498e3c3 service nova] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:16:56.486219 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0a3026d-109f-4f7f-8c2d-65045498e3c3 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:16:56.487979 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.fernet_utils [None req-f0a3026d-109f-4f7f-8c2d-65045498e3c3 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92057) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:16:56.501812 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f0a3026d-109f-4f7f-8c2d-65045498e3c3 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:16:56.503420 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f0a3026d-109f-4f7f-8c2d-65045498e3c3 service nova] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:16:56.516396 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 1434/2854] 10.222.0.30 () {60 vars in 1365 bytes} [Mon May 23 13:16:56 2022] GET /identity/v3/auth/tokens => generated 3362 bytes in 53 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) May 23 13:17:01.072718 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e1a330c-6a24-490e-9e97-4def61591fe2 None None] REQUEST_METHOD: `POST` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:17:01.073249 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e1a330c-6a24-490e-9e97-4def61591fe2 None None] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:17:01.073359 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e1a330c-6a24-490e-9e97-4def61591fe2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:17:01.091433 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.auth.core [None req-2e1a330c-6a24-490e-9e97-4def61591fe2 None None] MFA Rules not processed for user `fe5a3e6e6a0b4509b92cdcd54da0f4fe`. Rule list: `[]` (Enabled: `True`). {{(pid=92056) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 13:17:01.092649 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.fernet_utils [None req-2e1a330c-6a24-490e-9e97-4def61591fe2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92056) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:17:01.147666 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 1421/2855] 10.222.0.30 () {58 vars in 957 bytes} [Mon May 23 13:17:01 2022] POST /identity/v3/auth/tokens => generated 3244 bytes in 76 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) May 23 13:17:01.820380 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0a3026d-109f-4f7f-8c2d-65045498e3c3 service nova] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:17:01.821848 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.fernet_utils [None req-f0a3026d-109f-4f7f-8c2d-65045498e3c3 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92057) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:17:01.900004 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0a3026d-109f-4f7f-8c2d-65045498e3c3 service nova] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:17:01.905768 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3d5b528-b077-4c07-a008-8fc66d002c00 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd5fb51f8d0fa46439cb29666970b885b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9c389ae65aaa4542bb5a5c4fd0bacf5b', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'service', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:17:01.906443 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3d5b528-b077-4c07-a008-8fc66d002c00 service placement] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:17:01.906730 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3d5b528-b077-4c07-a008-8fc66d002c00 service placement] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:17:01.907012 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3d5b528-b077-4c07-a008-8fc66d002c00 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:17:01.908790 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.fernet_utils [None req-f3d5b528-b077-4c07-a008-8fc66d002c00 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92057) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:17:01.921342 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3d5b528-b077-4c07-a008-8fc66d002c00 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:17:01.922993 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3d5b528-b077-4c07-a008-8fc66d002c00 service placement] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:17:01.933004 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 1435/2856] 10.222.0.30 () {60 vars in 1365 bytes} [Mon May 23 13:17:01 2022] GET /identity/v3/auth/tokens => generated 3298 bytes in 113 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) May 23 13:17:11.140535 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=25 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 13:17:11.184288 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e1a330c-6a24-490e-9e97-4def61591fe2 None None] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:17:11.192176 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.fernet_utils [None req-2e1a330c-6a24-490e-9e97-4def61591fe2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92056) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:17:11.201550 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e1a330c-6a24-490e-9e97-4def61591fe2 None None] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:17:11.207061 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41642f46-730e-4b3d-ae1b-9bd45ef866ad service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cbe9931d94d44cc7ace0e83a7e5015fc', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9c389ae65aaa4542bb5a5c4fd0bacf5b', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'member', 'reader', 'service', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:17:11.210401 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41642f46-730e-4b3d-ae1b-9bd45ef866ad service nova] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:17:11.210401 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41642f46-730e-4b3d-ae1b-9bd45ef866ad service nova] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:17:11.210401 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41642f46-730e-4b3d-ae1b-9bd45ef866ad service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:17:11.210401 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.fernet_utils [None req-41642f46-730e-4b3d-ae1b-9bd45ef866ad service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92056) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:17:11.222251 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-41642f46-730e-4b3d-ae1b-9bd45ef866ad service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:17:11.223854 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-41642f46-730e-4b3d-ae1b-9bd45ef866ad service nova] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:17:11.235043 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 1422/2857] 10.222.0.30 () {60 vars in 1365 bytes} [Mon May 23 13:17:11 2022] GET /identity/v3/auth/tokens => generated 3362 bytes in 52 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) May 23 13:17:11.928750 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=38 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 13:17:21.229939 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=6 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 13:18:20.386467 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3d5b528-b077-4c07-a008-8fc66d002c00 service placement] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:18:20.399350 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3d5b528-b077-4c07-a008-8fc66d002c00 service placement] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:18:20.404828 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-016d03fc-5983-46f7-956f-5cd31ccde305 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cbe9931d94d44cc7ace0e83a7e5015fc', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9c389ae65aaa4542bb5a5c4fd0bacf5b', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'member', 'reader', 'service', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:18:20.405465 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-016d03fc-5983-46f7-956f-5cd31ccde305 service nova] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:18:20.405746 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-016d03fc-5983-46f7-956f-5cd31ccde305 service nova] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:18:20.406076 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-016d03fc-5983-46f7-956f-5cd31ccde305 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:18:20.407819 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.fernet_utils [None req-016d03fc-5983-46f7-956f-5cd31ccde305 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92057) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:18:20.418215 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-016d03fc-5983-46f7-956f-5cd31ccde305 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:18:20.419829 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-016d03fc-5983-46f7-956f-5cd31ccde305 service nova] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:18:20.429457 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 1436/2858] 10.222.0.30 () {60 vars in 1365 bytes} [Mon May 23 13:18:20 2022] GET /identity/v3/auth/tokens => generated 3244 bytes in 43 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) May 23 13:18:30.424633 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=6 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 13:18:56.583661 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41642f46-730e-4b3d-ae1b-9bd45ef866ad service nova] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:18:56.585124 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.fernet_utils [None req-41642f46-730e-4b3d-ae1b-9bd45ef866ad service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92056) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:18:56.596884 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41642f46-730e-4b3d-ae1b-9bd45ef866ad service nova] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:18:56.600496 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6434a890-3b1d-4435-86a3-ad748b5e165f service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2eb83ca38e79449095f3a2f947bbb06a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9c389ae65aaa4542bb5a5c4fd0bacf5b', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:18:56.601192 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6434a890-3b1d-4435-86a3-ad748b5e165f service neutron] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:18:56.601476 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6434a890-3b1d-4435-86a3-ad748b5e165f service neutron] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:18:56.601801 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6434a890-3b1d-4435-86a3-ad748b5e165f service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:18:56.603585 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.fernet_utils [None req-6434a890-3b1d-4435-86a3-ad748b5e165f service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92056) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:18:56.666563 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6434a890-3b1d-4435-86a3-ad748b5e165f service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:18:56.668002 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6434a890-3b1d-4435-86a3-ad748b5e165f service neutron] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:18:56.677502 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 1423/2859] 10.222.0.30 () {60 vars in 1365 bytes} [Mon May 23 13:18:56 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 94 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) May 23 13:19:00.069904 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-016d03fc-5983-46f7-956f-5cd31ccde305 service nova] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:19:00.071422 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.fernet_utils [None req-016d03fc-5983-46f7-956f-5cd31ccde305 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92057) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:19:00.086460 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-016d03fc-5983-46f7-956f-5cd31ccde305 service nova] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:19:00.092104 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3a67beb-3556-445f-a3b8-6c8a086e7a63 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd5fb51f8d0fa46439cb29666970b885b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9c389ae65aaa4542bb5a5c4fd0bacf5b', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'service', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:19:00.092409 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3a67beb-3556-445f-a3b8-6c8a086e7a63 service placement] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:19:00.092694 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3a67beb-3556-445f-a3b8-6c8a086e7a63 service placement] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:19:00.092967 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3a67beb-3556-445f-a3b8-6c8a086e7a63 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:19:00.094893 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.fernet_utils [None req-c3a67beb-3556-445f-a3b8-6c8a086e7a63 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92057) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:19:00.108451 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c3a67beb-3556-445f-a3b8-6c8a086e7a63 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:19:00.110139 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c3a67beb-3556-445f-a3b8-6c8a086e7a63 service placement] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:19:00.121545 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 1437/2860] 10.222.0.30 () {60 vars in 1365 bytes} [Mon May 23 13:19:00 2022] GET /identity/v3/auth/tokens => generated 3298 bytes in 52 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) May 23 13:19:00.170785 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6434a890-3b1d-4435-86a3-ad748b5e165f service neutron] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:19:00.176522 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.fernet_utils [None req-6434a890-3b1d-4435-86a3-ad748b5e165f service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92056) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:19:00.184607 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6434a890-3b1d-4435-86a3-ad748b5e165f service neutron] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:19:00.189215 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4489778-4453-4640-ad43-b533401d6e74 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:19:00.190157 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4489778-4453-4640-ad43-b533401d6e74 admin admin] REQUEST_METHOD: `DELETE` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:19:00.190442 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4489778-4453-4640-ad43-b533401d6e74 admin admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:19:00.190720 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4489778-4453-4640-ad43-b533401d6e74 admin admin] PATH_INFO: `/v3/users/fe5a3e6e6a0b4509b92cdcd54da0f4fe` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:19:00.191959 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4489778-4453-4640-ad43-b533401d6e74 admin admin] RBAC: Authorizing `identity:delete_user(user_id=fe5a3e6e6a0b4509b92cdcd54da0f4fe)` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:19:00.193921 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4489778-4453-4640-ad43-b533401d6e74 admin admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:19:00.360614 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-e4489778-4453-4640-ad43-b533401d6e74 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'fe5a3e6e6a0b4509b92cdcd54da0f4fe'} {{(pid=92056) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:19:00.361282 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-e4489778-4453-4640-ad43-b533401d6e74 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'fe5a3e6e6a0b4509b92cdcd54da0f4fe'} {{(pid=92056) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:19:00.432160 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-e4489778-4453-4640-ad43-b533401d6e74 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'fe5a3e6e6a0b4509b92cdcd54da0f4fe'} {{(pid=92056) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:19:00.445528 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-e4489778-4453-4640-ad43-b533401d6e74 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'fe5a3e6e6a0b4509b92cdcd54da0f4fe'} {{(pid=92056) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:19:00.452478 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-e4489778-4453-4640-ad43-b533401d6e74 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'fe5a3e6e6a0b4509b92cdcd54da0f4fe'} {{(pid=92056) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:19:00.458731 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 1424/2861] 10.222.0.30 () {58 vars in 1223 bytes} [Mon May 23 13:19:00 2022] DELETE /identity/v3/users/fe5a3e6e6a0b4509b92cdcd54da0f4fe => generated 0 bytes in 289 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) May 23 13:19:00.742732 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3a67beb-3556-445f-a3b8-6c8a086e7a63 service placement] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:19:00.755011 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3a67beb-3556-445f-a3b8-6c8a086e7a63 service placement] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:19:00.818303 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-311ea265-c8d3-4472-bd64-a2034015e412 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:19:00.819190 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-311ea265-c8d3-4472-bd64-a2034015e412 admin admin] REQUEST_METHOD: `DELETE` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:19:00.819478 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-311ea265-c8d3-4472-bd64-a2034015e412 admin admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:19:00.819755 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-311ea265-c8d3-4472-bd64-a2034015e412 admin admin] PATH_INFO: `/v3/projects/11bdfcfe4dbb42d3bb07585d85bff52a` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:19:00.821432 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-311ea265-c8d3-4472-bd64-a2034015e412 admin admin] RBAC: Authorizing `identity:delete_project(project_id=11bdfcfe4dbb42d3bb07585d85bff52a)` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:19:00.823573 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-311ea265-c8d3-4472-bd64-a2034015e412 admin admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:19:00.938780 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [None req-311ea265-c8d3-4472-bd64-a2034015e412 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '11bdfcfe4dbb42d3bb07585d85bff52a', 'request_id': 'req-311ea265-c8d3-4472-bd64-a2034015e412'} {{(pid=92057) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:19:00.944756 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [None req-311ea265-c8d3-4472-bd64-a2034015e412 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '11bdfcfe4dbb42d3bb07585d85bff52a', 'request_id': 'req-311ea265-c8d3-4472-bd64-a2034015e412'} {{(pid=92057) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:19:01.010436 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [None req-311ea265-c8d3-4472-bd64-a2034015e412 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '11bdfcfe4dbb42d3bb07585d85bff52a', 'request_id': 'req-311ea265-c8d3-4472-bd64-a2034015e412'} {{(pid=92057) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:19:01.022558 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [None req-311ea265-c8d3-4472-bd64-a2034015e412 admin admin] The token cache is being invalidate because project 11bdfcfe4dbb42d3bb07585d85bff52a was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=92057) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} May 23 13:19:01.022849 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [None req-311ea265-c8d3-4472-bd64-a2034015e412 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=92057) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:19:01.023486 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [None req-311ea265-c8d3-4472-bd64-a2034015e412 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=92057) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:19:01.025186 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 1438/2862] 10.222.0.30 () {58 vars in 1232 bytes} [Mon May 23 13:19:00 2022] DELETE /identity/v3/projects/11bdfcfe4dbb42d3bb07585d85bff52a => generated 0 bytes in 283 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) May 23 13:19:01.030274 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43a2c6e3-a1a2-43f3-ad09-a2578e99c586 None None] REQUEST_METHOD: `POST` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:19:01.030573 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43a2c6e3-a1a2-43f3-ad09-a2578e99c586 None None] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:19:01.030955 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43a2c6e3-a1a2-43f3-ad09-a2578e99c586 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:19:01.048667 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.auth.core [None req-43a2c6e3-a1a2-43f3-ad09-a2578e99c586 None None] MFA Rules not processed for user `0f7119f8fa5c4445977671140ded91a4`. Rule list: `[]` (Enabled: `True`). {{(pid=92056) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 13:19:01.049870 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.fernet_utils [None req-43a2c6e3-a1a2-43f3-ad09-a2578e99c586 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92056) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:19:01.109165 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 1425/2863] 10.222.0.30 () {58 vars in 957 bytes} [Mon May 23 13:19:01 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 80 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) May 23 13:19:01.117081 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bc66a67-5db5-48a6-ae77-f2934fbda027 None None] REQUEST_METHOD: `POST` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:19:01.118105 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bc66a67-5db5-48a6-ae77-f2934fbda027 None None] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:19:01.118199 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bc66a67-5db5-48a6-ae77-f2934fbda027 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:19:01.136689 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.auth.core [None req-3bc66a67-5db5-48a6-ae77-f2934fbda027 None None] MFA Rules not processed for user `0f7119f8fa5c4445977671140ded91a4`. Rule list: `[]` (Enabled: `True`). {{(pid=92057) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 13:19:01.137913 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.fernet_utils [None req-3bc66a67-5db5-48a6-ae77-f2934fbda027 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92057) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:19:01.146009 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 1439/2864] 10.222.0.30 () {58 vars in 957 bytes} [Mon May 23 13:19:01 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 30 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) May 23 13:19:01.150313 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43a2c6e3-a1a2-43f3-ad09-a2578e99c586 None None] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:19:01.160040 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43a2c6e3-a1a2-43f3-ad09-a2578e99c586 None None] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:19:01.164710 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f948ea22-bce3-4adc-82c2-50cd6c2b7197 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:19:01.165349 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f948ea22-bce3-4adc-82c2-50cd6c2b7197 admin admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:19:01.165629 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f948ea22-bce3-4adc-82c2-50cd6c2b7197 admin admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:19:01.165930 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f948ea22-bce3-4adc-82c2-50cd6c2b7197 admin admin] PATH_INFO: `/v3/domains` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:19:01.166646 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f948ea22-bce3-4adc-82c2-50cd6c2b7197 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=92056) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} May 23 13:19:01.166949 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f948ea22-bce3-4adc-82c2-50cd6c2b7197 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:19:01.168543 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f948ea22-bce3-4adc-82c2-50cd6c2b7197 admin admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:19:01.170068 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 13:19:01.181856 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 1426/2865] 10.222.0.30 () {58 vars in 1152 bytes} [Mon May 23 13:19:01 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) May 23 13:19:01.188058 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3bc66a67-5db5-48a6-ae77-f2934fbda027 None None] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:19:01.197775 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3bc66a67-5db5-48a6-ae77-f2934fbda027 None None] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:19:01.202444 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2abbee29-90d8-432a-a41f-2b3c9c32a113 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:19:01.203161 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2abbee29-90d8-432a-a41f-2b3c9c32a113 admin admin] REQUEST_METHOD: `POST` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:19:01.203373 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2abbee29-90d8-432a-a41f-2b3c9c32a113 admin admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:19:01.203652 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2abbee29-90d8-432a-a41f-2b3c9c32a113 admin admin] PATH_INFO: `/v3/projects` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:19:01.204383 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2abbee29-90d8-432a-a41f-2b3c9c32a113 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:19:01.206346 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2abbee29-90d8-432a-a41f-2b3c9c32a113 admin admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:19:01.319126 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 1440/2866] 10.222.0.30 () {60 vars in 1152 bytes} [Mon May 23 13:19:01 2022] POST /identity/v3/projects => generated 375 bytes in 131 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) May 23 13:19:01.323652 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f948ea22-bce3-4adc-82c2-50cd6c2b7197 admin admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:19:01.333348 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f948ea22-bce3-4adc-82c2-50cd6c2b7197 admin admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:19:01.389003 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ff59ba5-584e-4904-9825-c788176b8cc6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:19:01.389661 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ff59ba5-584e-4904-9825-c788176b8cc6 admin admin] REQUEST_METHOD: `POST` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:19:01.389971 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ff59ba5-584e-4904-9825-c788176b8cc6 admin admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:19:01.390248 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ff59ba5-584e-4904-9825-c788176b8cc6 admin admin] PATH_INFO: `/v3/users` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:19:01.390939 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ff59ba5-584e-4904-9825-c788176b8cc6 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:19:01.393118 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ff59ba5-584e-4904-9825-c788176b8cc6 admin admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:19:01.472486 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 1427/2867] 10.222.0.30 () {60 vars in 1143 bytes} [Mon May 23 13:19:01 2022] POST /identity/v3/users => generated 288 bytes in 149 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) May 23 13:19:01.476453 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2abbee29-90d8-432a-a41f-2b3c9c32a113 admin admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:19:01.487195 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2abbee29-90d8-432a-a41f-2b3c9c32a113 admin admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:19:01.491805 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c6165d6-fecf-4702-9e20-7e9e43f6077e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:19:01.493067 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c6165d6-fecf-4702-9e20-7e9e43f6077e admin admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:19:01.493355 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c6165d6-fecf-4702-9e20-7e9e43f6077e admin admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:19:01.493635 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c6165d6-fecf-4702-9e20-7e9e43f6077e admin admin] PATH_INFO: `/v3/roles` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:19:01.494398 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c6165d6-fecf-4702-9e20-7e9e43f6077e admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:19:01.495984 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c6165d6-fecf-4702-9e20-7e9e43f6077e admin admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:19:01.505775 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 1441/2868] 10.222.0.30 () {58 vars in 1121 bytes} [Mon May 23 13:19:01 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) May 23 13:19:01.509704 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ff59ba5-584e-4904-9825-c788176b8cc6 admin admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:19:01.519836 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ff59ba5-584e-4904-9825-c788176b8cc6 admin admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:19:01.524423 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-606456bf-d76f-4f33-b73c-ff2475097914 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:19:01.525065 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-606456bf-d76f-4f33-b73c-ff2475097914 admin admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:19:01.525624 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-606456bf-d76f-4f33-b73c-ff2475097914 admin admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:19:01.525936 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-606456bf-d76f-4f33-b73c-ff2475097914 admin admin] PATH_INFO: `/v3/roles` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:19:01.526666 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-606456bf-d76f-4f33-b73c-ff2475097914 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:19:01.528265 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-606456bf-d76f-4f33-b73c-ff2475097914 admin admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:19:01.538096 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 1428/2869] 10.222.0.30 () {58 vars in 1121 bytes} [Mon May 23 13:19:01 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) May 23 13:19:01.542802 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c6165d6-fecf-4702-9e20-7e9e43f6077e admin admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:19:01.552498 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c6165d6-fecf-4702-9e20-7e9e43f6077e admin admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:19:01.557920 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13e5bccb-71cf-432c-8e05-a0a01b50723c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:19:01.558787 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13e5bccb-71cf-432c-8e05-a0a01b50723c admin admin] REQUEST_METHOD: `PUT` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:19:01.559069 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13e5bccb-71cf-432c-8e05-a0a01b50723c admin admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:19:01.559343 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13e5bccb-71cf-432c-8e05-a0a01b50723c admin admin] PATH_INFO: `/v3/projects/e9089605f3dc40408ce2fa54cf02fa9a/users/31e82249616a42c0973dcf3c0c065cd9/roles/80fb22507c9a4ff09d44d8f12f6fff1a` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:19:01.573872 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13e5bccb-71cf-432c-8e05-a0a01b50723c admin admin] RBAC: Authorizing `identity:create_grant(project_id=e9089605f3dc40408ce2fa54cf02fa9a, user_id=31e82249616a42c0973dcf3c0c065cd9, role_id=80fb22507c9a4ff09d44d8f12f6fff1a)` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:19:01.576518 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13e5bccb-71cf-432c-8e05-a0a01b50723c admin admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:19:01.644440 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 1442/2870] 10.222.0.30 () {60 vars in 1482 bytes} [Mon May 23 13:19:01 2022] PUT /identity/v3/projects/e9089605f3dc40408ce2fa54cf02fa9a/users/31e82249616a42c0973dcf3c0c065cd9/roles/80fb22507c9a4ff09d44d8f12f6fff1a => generated 0 bytes in 102 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) May 23 13:19:01.655613 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e1bf289-55dc-4780-99e5-04168ea67134 None None] REQUEST_METHOD: `POST` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:19:01.655937 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e1bf289-55dc-4780-99e5-04168ea67134 None None] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:19:01.656225 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e1bf289-55dc-4780-99e5-04168ea67134 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:19:01.690259 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.auth.core [None req-9e1bf289-55dc-4780-99e5-04168ea67134 None None] MFA Rules not processed for user `31e82249616a42c0973dcf3c0c065cd9`. Rule list: `[]` (Enabled: `True`). {{(pid=92056) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 13:19:01.691490 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.fernet_utils [None req-9e1bf289-55dc-4780-99e5-04168ea67134 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92056) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:19:01.755923 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 1429/2871] 10.222.0.30 () {58 vars in 957 bytes} [Mon May 23 13:19:01 2022] POST /identity/v3/auth/tokens => generated 3242 bytes in 101 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) May 23 13:19:01.767061 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13e5bccb-71cf-432c-8e05-a0a01b50723c admin admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:19:01.769125 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.fernet_utils [None req-13e5bccb-71cf-432c-8e05-a0a01b50723c admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92057) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:19:01.776657 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 13:19:01.843069 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13e5bccb-71cf-432c-8e05-a0a01b50723c admin admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:19:01.848730 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79f7d0a3-aa9d-42a6-b42c-d5aa5183daad service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cbe9931d94d44cc7ace0e83a7e5015fc', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9c389ae65aaa4542bb5a5c4fd0bacf5b', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'member', 'reader', 'service', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:19:01.849513 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79f7d0a3-aa9d-42a6-b42c-d5aa5183daad service nova] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:19:01.849797 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79f7d0a3-aa9d-42a6-b42c-d5aa5183daad service nova] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:19:01.850098 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79f7d0a3-aa9d-42a6-b42c-d5aa5183daad service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:19:01.858481 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79f7d0a3-aa9d-42a6-b42c-d5aa5183daad service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:19:01.860186 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79f7d0a3-aa9d-42a6-b42c-d5aa5183daad service nova] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:19:01.869735 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 1443/2872] 10.222.0.30 () {60 vars in 1365 bytes} [Mon May 23 13:19:01 2022] GET /identity/v3/auth/tokens => generated 3242 bytes in 103 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) May 23 13:19:02.084200 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e1bf289-55dc-4780-99e5-04168ea67134 None None] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:19:02.149363 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e1bf289-55dc-4780-99e5-04168ea67134 None None] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:19:02.156302 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff27f03f-c263-437e-8b08-e4c99926be9b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:19:02.157331 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff27f03f-c263-437e-8b08-e4c99926be9b admin admin] REQUEST_METHOD: `DELETE` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:19:02.157597 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff27f03f-c263-437e-8b08-e4c99926be9b admin admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:19:02.157872 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff27f03f-c263-437e-8b08-e4c99926be9b admin admin] PATH_INFO: `/v3/users/31e82249616a42c0973dcf3c0c065cd9` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:19:02.159079 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff27f03f-c263-437e-8b08-e4c99926be9b admin admin] RBAC: Authorizing `identity:delete_user(user_id=31e82249616a42c0973dcf3c0c065cd9)` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:19:02.161030 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff27f03f-c263-437e-8b08-e4c99926be9b admin admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:19:02.163537 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 13:19:02.336800 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-ff27f03f-c263-437e-8b08-e4c99926be9b admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '31e82249616a42c0973dcf3c0c065cd9'} {{(pid=92056) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:19:02.338013 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-ff27f03f-c263-437e-8b08-e4c99926be9b admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '31e82249616a42c0973dcf3c0c065cd9'} {{(pid=92056) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:19:02.398117 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-ff27f03f-c263-437e-8b08-e4c99926be9b admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '31e82249616a42c0973dcf3c0c065cd9'} {{(pid=92056) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:19:02.411920 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-ff27f03f-c263-437e-8b08-e4c99926be9b admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '31e82249616a42c0973dcf3c0c065cd9'} {{(pid=92056) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:19:02.419623 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-ff27f03f-c263-437e-8b08-e4c99926be9b admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '31e82249616a42c0973dcf3c0c065cd9'} {{(pid=92056) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:19:02.425808 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 1430/2873] 10.222.0.30 () {58 vars in 1223 bytes} [Mon May 23 13:19:02 2022] DELETE /identity/v3/users/31e82249616a42c0973dcf3c0c065cd9 => generated 0 bytes in 342 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) May 23 13:19:02.430413 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90636651-a5b1-40b4-aec9-2ffb5f4d663d None None] REQUEST_METHOD: `POST` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:19:02.430929 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90636651-a5b1-40b4-aec9-2ffb5f4d663d None None] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:19:02.431047 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90636651-a5b1-40b4-aec9-2ffb5f4d663d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:19:02.448137 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.auth.core [None req-90636651-a5b1-40b4-aec9-2ffb5f4d663d None None] MFA Rules not processed for user `0f7119f8fa5c4445977671140ded91a4`. Rule list: `[]` (Enabled: `True`). {{(pid=92057) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 13:19:02.449334 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.fernet_utils [None req-90636651-a5b1-40b4-aec9-2ffb5f4d663d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92057) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:19:02.507841 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 1444/2874] 10.222.0.30 () {58 vars in 957 bytes} [Mon May 23 13:19:02 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 78 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) May 23 13:19:02.517858 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff27f03f-c263-437e-8b08-e4c99926be9b admin admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:19:02.519266 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.fernet_utils [None req-ff27f03f-c263-437e-8b08-e4c99926be9b admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92056) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:19:02.575047 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff27f03f-c263-437e-8b08-e4c99926be9b admin admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:19:02.578696 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a22e77f-c3d5-4d4f-81f6-1178102e2915 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2eb83ca38e79449095f3a2f947bbb06a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9c389ae65aaa4542bb5a5c4fd0bacf5b', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:19:02.579328 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a22e77f-c3d5-4d4f-81f6-1178102e2915 service neutron] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:19:02.579607 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a22e77f-c3d5-4d4f-81f6-1178102e2915 service neutron] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:19:02.579878 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a22e77f-c3d5-4d4f-81f6-1178102e2915 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:19:02.589520 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a22e77f-c3d5-4d4f-81f6-1178102e2915 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:19:02.591007 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a22e77f-c3d5-4d4f-81f6-1178102e2915 service neutron] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:19:02.600561 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 1431/2875] 10.222.0.30 () {60 vars in 1365 bytes} [Mon May 23 13:19:02 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 84 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) May 23 13:19:02.988033 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90636651-a5b1-40b4-aec9-2ffb5f4d663d None None] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:19:02.989457 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.fernet_utils [None req-90636651-a5b1-40b4-aec9-2ffb5f4d663d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92057) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:19:03.001455 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90636651-a5b1-40b4-aec9-2ffb5f4d663d None None] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:19:03.006432 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1da34133-07f9-40e9-b0d9-b885cb6c4f90 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:19:03.007292 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1da34133-07f9-40e9-b0d9-b885cb6c4f90 admin admin] REQUEST_METHOD: `DELETE` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:19:03.007576 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1da34133-07f9-40e9-b0d9-b885cb6c4f90 admin admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:19:03.007850 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1da34133-07f9-40e9-b0d9-b885cb6c4f90 admin admin] PATH_INFO: `/v3/projects/e9089605f3dc40408ce2fa54cf02fa9a` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:19:03.009515 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1da34133-07f9-40e9-b0d9-b885cb6c4f90 admin admin] RBAC: Authorizing `identity:delete_project(project_id=e9089605f3dc40408ce2fa54cf02fa9a)` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:19:03.011903 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1da34133-07f9-40e9-b0d9-b885cb6c4f90 admin admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:19:03.100665 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [None req-1da34133-07f9-40e9-b0d9-b885cb6c4f90 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'e9089605f3dc40408ce2fa54cf02fa9a', 'request_id': 'req-1da34133-07f9-40e9-b0d9-b885cb6c4f90'} {{(pid=92057) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:19:03.106048 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [None req-1da34133-07f9-40e9-b0d9-b885cb6c4f90 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'e9089605f3dc40408ce2fa54cf02fa9a', 'request_id': 'req-1da34133-07f9-40e9-b0d9-b885cb6c4f90'} {{(pid=92057) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:19:03.169493 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [None req-1da34133-07f9-40e9-b0d9-b885cb6c4f90 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'e9089605f3dc40408ce2fa54cf02fa9a', 'request_id': 'req-1da34133-07f9-40e9-b0d9-b885cb6c4f90'} {{(pid=92057) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:19:03.181366 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [None req-1da34133-07f9-40e9-b0d9-b885cb6c4f90 admin admin] The token cache is being invalidate because project e9089605f3dc40408ce2fa54cf02fa9a was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=92057) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} May 23 13:19:03.181659 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [None req-1da34133-07f9-40e9-b0d9-b885cb6c4f90 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=92057) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:19:03.182487 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [None req-1da34133-07f9-40e9-b0d9-b885cb6c4f90 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=92057) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:19:03.184031 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 1445/2876] 10.222.0.30 () {58 vars in 1232 bytes} [Mon May 23 13:19:02 2022] DELETE /identity/v3/projects/e9089605f3dc40408ce2fa54cf02fa9a => generated 0 bytes in 197 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) May 23 13:19:03.191049 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dea0c80f-d4fb-4ba0-a15d-e16b05da96db None None] REQUEST_METHOD: `POST` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:19:03.191342 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dea0c80f-d4fb-4ba0-a15d-e16b05da96db None None] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:19:03.191742 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dea0c80f-d4fb-4ba0-a15d-e16b05da96db None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:19:03.209654 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.auth.core [None req-dea0c80f-d4fb-4ba0-a15d-e16b05da96db None None] MFA Rules not processed for user `0f7119f8fa5c4445977671140ded91a4`. Rule list: `[]` (Enabled: `True`). {{(pid=92056) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 13:19:03.210880 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.fernet_utils [None req-dea0c80f-d4fb-4ba0-a15d-e16b05da96db None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92056) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:19:03.269126 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 1432/2877] 10.222.0.30 () {58 vars in 957 bytes} [Mon May 23 13:19:03 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 79 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) May 23 13:19:03.277088 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdbc9379-cf37-41c5-bdb4-009beadb8c88 None None] REQUEST_METHOD: `POST` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:19:03.277380 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdbc9379-cf37-41c5-bdb4-009beadb8c88 None None] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:19:03.277662 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdbc9379-cf37-41c5-bdb4-009beadb8c88 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:19:03.295546 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.auth.core [None req-fdbc9379-cf37-41c5-bdb4-009beadb8c88 None None] MFA Rules not processed for user `0f7119f8fa5c4445977671140ded91a4`. Rule list: `[]` (Enabled: `True`). {{(pid=92057) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 13:19:03.296729 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.fernet_utils [None req-fdbc9379-cf37-41c5-bdb4-009beadb8c88 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92057) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:19:03.304772 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 1446/2878] 10.222.0.30 () {58 vars in 957 bytes} [Mon May 23 13:19:03 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 29 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) May 23 13:19:03.309536 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dea0c80f-d4fb-4ba0-a15d-e16b05da96db None None] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:19:03.318994 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dea0c80f-d4fb-4ba0-a15d-e16b05da96db None None] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:19:03.323709 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10acc13b-668a-48d6-9379-916023c6eabb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:19:03.324349 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10acc13b-668a-48d6-9379-916023c6eabb admin admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:19:03.324630 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10acc13b-668a-48d6-9379-916023c6eabb admin admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:19:03.324905 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10acc13b-668a-48d6-9379-916023c6eabb admin admin] PATH_INFO: `/v3/domains` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:19:03.325616 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10acc13b-668a-48d6-9379-916023c6eabb admin admin] RBAC: Adding query filter params (name=Default) {{(pid=92056) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} May 23 13:19:03.325953 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10acc13b-668a-48d6-9379-916023c6eabb admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:19:03.327543 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10acc13b-668a-48d6-9379-916023c6eabb admin admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:19:03.339259 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 1433/2879] 10.222.0.30 () {58 vars in 1152 bytes} [Mon May 23 13:19:03 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) May 23 13:19:03.343427 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fdbc9379-cf37-41c5-bdb4-009beadb8c88 None None] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:19:03.352613 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fdbc9379-cf37-41c5-bdb4-009beadb8c88 None None] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:19:03.357343 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-849f6d34-b1c9-4492-90ab-0f1767d5144b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:19:03.358015 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-849f6d34-b1c9-4492-90ab-0f1767d5144b admin admin] REQUEST_METHOD: `POST` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:19:03.358301 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-849f6d34-b1c9-4492-90ab-0f1767d5144b admin admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:19:03.358579 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-849f6d34-b1c9-4492-90ab-0f1767d5144b admin admin] PATH_INFO: `/v3/projects` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:19:03.359303 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-849f6d34-b1c9-4492-90ab-0f1767d5144b admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:19:03.361182 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-849f6d34-b1c9-4492-90ab-0f1767d5144b admin admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:19:03.364376 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 13:19:03.496838 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 1447/2880] 10.222.0.30 () {60 vars in 1152 bytes} [Mon May 23 13:19:03 2022] POST /identity/v3/projects => generated 373 bytes in 154 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) May 23 13:19:03.500897 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10acc13b-668a-48d6-9379-916023c6eabb admin admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:19:03.511610 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10acc13b-668a-48d6-9379-916023c6eabb admin admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:19:03.523116 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 13:19:03.567667 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d7f98fe-3e9d-4517-b1f2-37b65880a662 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:19:03.568346 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d7f98fe-3e9d-4517-b1f2-37b65880a662 admin admin] REQUEST_METHOD: `POST` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:19:03.568632 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d7f98fe-3e9d-4517-b1f2-37b65880a662 admin admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:19:03.568902 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d7f98fe-3e9d-4517-b1f2-37b65880a662 admin admin] PATH_INFO: `/v3/users` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:19:03.569589 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d7f98fe-3e9d-4517-b1f2-37b65880a662 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:19:03.572078 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d7f98fe-3e9d-4517-b1f2-37b65880a662 admin admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:19:03.656292 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 1434/2881] 10.222.0.30 () {60 vars in 1143 bytes} [Mon May 23 13:19:03 2022] POST /identity/v3/users => generated 287 bytes in 156 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) May 23 13:19:03.660441 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-849f6d34-b1c9-4492-90ab-0f1767d5144b admin admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:19:03.671622 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-849f6d34-b1c9-4492-90ab-0f1767d5144b admin admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:19:03.676378 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6610790a-885e-491c-9747-3a0900f60fb8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:19:03.677095 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6610790a-885e-491c-9747-3a0900f60fb8 admin admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:19:03.677381 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6610790a-885e-491c-9747-3a0900f60fb8 admin admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:19:03.677657 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6610790a-885e-491c-9747-3a0900f60fb8 admin admin] PATH_INFO: `/v3/roles` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:19:03.678424 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6610790a-885e-491c-9747-3a0900f60fb8 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:19:03.680031 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6610790a-885e-491c-9747-3a0900f60fb8 admin admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:19:03.691388 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 1448/2882] 10.222.0.30 () {58 vars in 1121 bytes} [Mon May 23 13:19:03 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) May 23 13:19:03.694915 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d7f98fe-3e9d-4517-b1f2-37b65880a662 admin admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:19:03.706117 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d7f98fe-3e9d-4517-b1f2-37b65880a662 admin admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:19:03.710952 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbd1637a-0479-48aa-bc17-acbfe9397282 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:19:03.711605 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbd1637a-0479-48aa-bc17-acbfe9397282 admin admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:19:03.711886 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbd1637a-0479-48aa-bc17-acbfe9397282 admin admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:19:03.712818 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbd1637a-0479-48aa-bc17-acbfe9397282 admin admin] PATH_INFO: `/v3/roles` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:19:03.713545 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bbd1637a-0479-48aa-bc17-acbfe9397282 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:19:03.715168 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bbd1637a-0479-48aa-bc17-acbfe9397282 admin admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:19:03.725470 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 1435/2883] 10.222.0.30 () {58 vars in 1121 bytes} [Mon May 23 13:19:03 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) May 23 13:19:03.729605 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6610790a-885e-491c-9747-3a0900f60fb8 admin admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:19:03.740102 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6610790a-885e-491c-9747-3a0900f60fb8 admin admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:19:03.745173 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5be0874e-173c-478b-b70d-1d6717e71ad2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:19:03.746079 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5be0874e-173c-478b-b70d-1d6717e71ad2 admin admin] REQUEST_METHOD: `PUT` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:19:03.746367 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5be0874e-173c-478b-b70d-1d6717e71ad2 admin admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:19:03.746645 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5be0874e-173c-478b-b70d-1d6717e71ad2 admin admin] PATH_INFO: `/v3/projects/0ad540e0075e486aba19b0a124e366be/users/0cfd4082ce714d1892f79179297a99d7/roles/80fb22507c9a4ff09d44d8f12f6fff1a` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:19:03.761813 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5be0874e-173c-478b-b70d-1d6717e71ad2 admin admin] RBAC: Authorizing `identity:create_grant(project_id=0ad540e0075e486aba19b0a124e366be, user_id=0cfd4082ce714d1892f79179297a99d7, role_id=80fb22507c9a4ff09d44d8f12f6fff1a)` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:19:03.764484 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5be0874e-173c-478b-b70d-1d6717e71ad2 admin admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:19:03.855698 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 1449/2884] 10.222.0.30 () {60 vars in 1482 bytes} [Mon May 23 13:19:03 2022] PUT /identity/v3/projects/0ad540e0075e486aba19b0a124e366be/users/0cfd4082ce714d1892f79179297a99d7/roles/80fb22507c9a4ff09d44d8f12f6fff1a => generated 0 bytes in 127 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) May 23 13:19:03.866046 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-601e612b-1172-4b54-8139-cec9f70fdac4 None None] REQUEST_METHOD: `POST` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:19:03.866341 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-601e612b-1172-4b54-8139-cec9f70fdac4 None None] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:19:03.866624 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-601e612b-1172-4b54-8139-cec9f70fdac4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:19:03.893734 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.auth.core [None req-601e612b-1172-4b54-8139-cec9f70fdac4 None None] MFA Rules not processed for user `0cfd4082ce714d1892f79179297a99d7`. Rule list: `[]` (Enabled: `True`). {{(pid=92056) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 13:19:03.894994 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.fernet_utils [None req-601e612b-1172-4b54-8139-cec9f70fdac4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92056) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:19:03.963373 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 1436/2885] 10.222.0.30 () {58 vars in 957 bytes} [Mon May 23 13:19:03 2022] POST /identity/v3/auth/tokens => generated 3240 bytes in 98 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) May 23 13:19:03.974956 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5be0874e-173c-478b-b70d-1d6717e71ad2 admin admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:19:03.976686 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.fernet_utils [None req-5be0874e-173c-478b-b70d-1d6717e71ad2 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92057) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:19:04.040241 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5be0874e-173c-478b-b70d-1d6717e71ad2 admin admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:19:04.044995 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8362315-5eb7-40aa-8ece-4900be94bd16 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1666097f8ed545c6b0e4809c93e4775a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9c389ae65aaa4542bb5a5c4fd0bacf5b', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:19:04.045653 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8362315-5eb7-40aa-8ece-4900be94bd16 service cinder] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:19:04.045978 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8362315-5eb7-40aa-8ece-4900be94bd16 service cinder] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:19:04.046255 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8362315-5eb7-40aa-8ece-4900be94bd16 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:19:04.054945 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d8362315-5eb7-40aa-8ece-4900be94bd16 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:19:04.056515 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d8362315-5eb7-40aa-8ece-4900be94bd16 service cinder] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:19:04.066364 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 1450/2886] 10.222.0.30 () {60 vars in 1365 bytes} [Mon May 23 13:19:03 2022] GET /identity/v3/auth/tokens => generated 3240 bytes in 92 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) May 23 13:19:09.880435 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-601e612b-1172-4b54-8139-cec9f70fdac4 None None] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:19:09.881846 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.fernet_utils [None req-601e612b-1172-4b54-8139-cec9f70fdac4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92056) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:19:09.951873 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-601e612b-1172-4b54-8139-cec9f70fdac4 None None] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:19:09.957062 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b18937ea-cee6-4ae5-8220-9f5532f203c6 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd5fb51f8d0fa46439cb29666970b885b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9c389ae65aaa4542bb5a5c4fd0bacf5b', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'service', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:19:09.957712 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b18937ea-cee6-4ae5-8220-9f5532f203c6 service placement] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:19:09.958030 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b18937ea-cee6-4ae5-8220-9f5532f203c6 service placement] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:19:09.958313 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b18937ea-cee6-4ae5-8220-9f5532f203c6 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:19:09.960097 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.fernet_utils [None req-b18937ea-cee6-4ae5-8220-9f5532f203c6 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92056) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:19:09.972419 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b18937ea-cee6-4ae5-8220-9f5532f203c6 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:19:09.974021 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b18937ea-cee6-4ae5-8220-9f5532f203c6 service placement] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:19:09.983762 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 1437/2887] 10.222.0.30 () {60 vars in 1365 bytes} [Mon May 23 13:19:09 2022] GET /identity/v3/auth/tokens => generated 3298 bytes in 104 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) May 23 13:19:14.061986 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=47,keystone:INSERT=2 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 13:19:19.979606 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=93,keystone:INSERT=3 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 13:19:29.876834 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8362315-5eb7-40aa-8ece-4900be94bd16 service cinder] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:19:29.940710 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8362315-5eb7-40aa-8ece-4900be94bd16 service cinder] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:19:29.945257 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73446ec3-b555-410d-a815-f386a8c612bf admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:19:29.946094 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73446ec3-b555-410d-a815-f386a8c612bf admin admin] REQUEST_METHOD: `DELETE` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:19:29.946374 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73446ec3-b555-410d-a815-f386a8c612bf admin admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:19:29.946644 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73446ec3-b555-410d-a815-f386a8c612bf admin admin] PATH_INFO: `/v3/users/0cfd4082ce714d1892f79179297a99d7` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:19:29.947800 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-73446ec3-b555-410d-a815-f386a8c612bf admin admin] RBAC: Authorizing `identity:delete_user(user_id=0cfd4082ce714d1892f79179297a99d7)` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:19:29.949782 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-73446ec3-b555-410d-a815-f386a8c612bf admin admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:19:30.130628 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [None req-73446ec3-b555-410d-a815-f386a8c612bf admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '0cfd4082ce714d1892f79179297a99d7'} {{(pid=92057) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:19:30.173920 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [None req-73446ec3-b555-410d-a815-f386a8c612bf admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '0cfd4082ce714d1892f79179297a99d7'} {{(pid=92057) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:19:30.180988 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [None req-73446ec3-b555-410d-a815-f386a8c612bf admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '0cfd4082ce714d1892f79179297a99d7'} {{(pid=92057) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:19:30.181677 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [None req-73446ec3-b555-410d-a815-f386a8c612bf admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '0cfd4082ce714d1892f79179297a99d7'} {{(pid=92057) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:19:30.195065 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [None req-73446ec3-b555-410d-a815-f386a8c612bf admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '0cfd4082ce714d1892f79179297a99d7'} {{(pid=92057) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:19:30.201658 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 1451/2888] 10.222.0.30 () {58 vars in 1223 bytes} [Mon May 23 13:19:29 2022] DELETE /identity/v3/users/0cfd4082ce714d1892f79179297a99d7 => generated 0 bytes in 325 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) May 23 13:19:30.206150 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8777c38-68b9-453c-8363-4b8a0ee1c0c9 None None] REQUEST_METHOD: `POST` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:19:30.209197 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8777c38-68b9-453c-8363-4b8a0ee1c0c9 None None] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:19:30.209197 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8777c38-68b9-453c-8363-4b8a0ee1c0c9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:19:30.224550 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.auth.core [None req-d8777c38-68b9-453c-8363-4b8a0ee1c0c9 None None] MFA Rules not processed for user `0f7119f8fa5c4445977671140ded91a4`. Rule list: `[]` (Enabled: `True`). {{(pid=92056) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 13:19:30.226430 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.fernet_utils [None req-d8777c38-68b9-453c-8363-4b8a0ee1c0c9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92056) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:19:30.286726 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 1438/2889] 10.222.0.30 () {58 vars in 957 bytes} [Mon May 23 13:19:30 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 82 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) May 23 13:19:30.296946 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73446ec3-b555-410d-a815-f386a8c612bf admin admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:19:30.298538 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.fernet_utils [None req-73446ec3-b555-410d-a815-f386a8c612bf admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92057) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:19:30.356123 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73446ec3-b555-410d-a815-f386a8c612bf admin admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:19:30.360754 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-129772cf-2a05-4062-9de0-6919b074c7d8 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2eb83ca38e79449095f3a2f947bbb06a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9c389ae65aaa4542bb5a5c4fd0bacf5b', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:19:30.361399 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-129772cf-2a05-4062-9de0-6919b074c7d8 service neutron] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:19:30.361680 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-129772cf-2a05-4062-9de0-6919b074c7d8 service neutron] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:19:30.361979 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-129772cf-2a05-4062-9de0-6919b074c7d8 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:19:30.371026 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-129772cf-2a05-4062-9de0-6919b074c7d8 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:19:30.372503 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-129772cf-2a05-4062-9de0-6919b074c7d8 service neutron] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:19:30.382075 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 1452/2890] 10.222.0.30 () {60 vars in 1365 bytes} [Mon May 23 13:19:30 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 86 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) May 23 13:19:30.788031 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8777c38-68b9-453c-8363-4b8a0ee1c0c9 None None] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:19:30.791153 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.fernet_utils [None req-d8777c38-68b9-453c-8363-4b8a0ee1c0c9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92056) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:19:30.803642 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8777c38-68b9-453c-8363-4b8a0ee1c0c9 None None] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:19:30.808592 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0b0bf72-63d4-45e1-ae64-9a56c0cf8b36 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:19:30.809471 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0b0bf72-63d4-45e1-ae64-9a56c0cf8b36 admin admin] REQUEST_METHOD: `DELETE` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:19:30.809753 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0b0bf72-63d4-45e1-ae64-9a56c0cf8b36 admin admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:19:30.810065 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0b0bf72-63d4-45e1-ae64-9a56c0cf8b36 admin admin] PATH_INFO: `/v3/projects/0ad540e0075e486aba19b0a124e366be` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:19:30.811556 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0b0bf72-63d4-45e1-ae64-9a56c0cf8b36 admin admin] RBAC: Authorizing `identity:delete_project(project_id=0ad540e0075e486aba19b0a124e366be)` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:19:30.813450 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0b0bf72-63d4-45e1-ae64-9a56c0cf8b36 admin admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:19:30.935653 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-b0b0bf72-63d4-45e1-ae64-9a56c0cf8b36 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '0ad540e0075e486aba19b0a124e366be', 'request_id': 'req-b0b0bf72-63d4-45e1-ae64-9a56c0cf8b36'} {{(pid=92056) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:19:31.002605 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-b0b0bf72-63d4-45e1-ae64-9a56c0cf8b36 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '0ad540e0075e486aba19b0a124e366be', 'request_id': 'req-b0b0bf72-63d4-45e1-ae64-9a56c0cf8b36'} {{(pid=92056) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:19:31.009367 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-b0b0bf72-63d4-45e1-ae64-9a56c0cf8b36 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '0ad540e0075e486aba19b0a124e366be', 'request_id': 'req-b0b0bf72-63d4-45e1-ae64-9a56c0cf8b36'} {{(pid=92056) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:19:31.020543 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-b0b0bf72-63d4-45e1-ae64-9a56c0cf8b36 admin admin] The token cache is being invalidate because project 0ad540e0075e486aba19b0a124e366be was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=92056) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} May 23 13:19:31.020831 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-b0b0bf72-63d4-45e1-ae64-9a56c0cf8b36 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=92056) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:19:31.021465 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-b0b0bf72-63d4-45e1-ae64-9a56c0cf8b36 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=92056) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:19:31.023050 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 1439/2891] 10.222.0.30 () {58 vars in 1232 bytes} [Mon May 23 13:19:30 2022] DELETE /identity/v3/projects/0ad540e0075e486aba19b0a124e366be => generated 0 bytes in 236 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) May 23 13:19:31.029811 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-adee4b7b-64c9-456d-b977-48d3fd5cd77c None None] REQUEST_METHOD: `POST` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:19:31.031280 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-adee4b7b-64c9-456d-b977-48d3fd5cd77c None None] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:19:31.031280 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-adee4b7b-64c9-456d-b977-48d3fd5cd77c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:19:31.047891 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.auth.core [None req-adee4b7b-64c9-456d-b977-48d3fd5cd77c None None] MFA Rules not processed for user `0f7119f8fa5c4445977671140ded91a4`. Rule list: `[]` (Enabled: `True`). {{(pid=92057) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 13:19:31.054593 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.fernet_utils [None req-adee4b7b-64c9-456d-b977-48d3fd5cd77c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92057) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:19:31.066308 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 13:19:31.110679 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 1453/2892] 10.222.0.30 () {58 vars in 957 bytes} [Mon May 23 13:19:31 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 83 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) May 23 13:19:31.120011 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d08967e-6785-495f-b08b-8ea9bf1d9282 None None] REQUEST_METHOD: `POST` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:19:31.120300 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d08967e-6785-495f-b08b-8ea9bf1d9282 None None] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:19:31.120577 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d08967e-6785-495f-b08b-8ea9bf1d9282 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:19:31.139518 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.auth.core [None req-2d08967e-6785-495f-b08b-8ea9bf1d9282 None None] MFA Rules not processed for user `0f7119f8fa5c4445977671140ded91a4`. Rule list: `[]` (Enabled: `True`). {{(pid=92056) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 13:19:31.140730 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.fernet_utils [None req-2d08967e-6785-495f-b08b-8ea9bf1d9282 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92056) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:19:31.148903 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 1440/2893] 10.222.0.30 () {58 vars in 957 bytes} [Mon May 23 13:19:31 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 30 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) May 23 13:19:31.153103 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-adee4b7b-64c9-456d-b977-48d3fd5cd77c None None] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:19:31.162741 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-adee4b7b-64c9-456d-b977-48d3fd5cd77c None None] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:19:31.167477 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7c771c5-d244-4503-a1f3-5004fd79b5d4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:19:31.168153 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7c771c5-d244-4503-a1f3-5004fd79b5d4 admin admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:19:31.168440 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7c771c5-d244-4503-a1f3-5004fd79b5d4 admin admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:19:31.168720 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7c771c5-d244-4503-a1f3-5004fd79b5d4 admin admin] PATH_INFO: `/v3/domains` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:19:31.169443 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d7c771c5-d244-4503-a1f3-5004fd79b5d4 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=92057) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} May 23 13:19:31.169758 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d7c771c5-d244-4503-a1f3-5004fd79b5d4 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:19:31.171438 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d7c771c5-d244-4503-a1f3-5004fd79b5d4 admin admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:19:31.182815 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 1454/2894] 10.222.0.30 () {58 vars in 1152 bytes} [Mon May 23 13:19:31 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) May 23 13:19:31.187422 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d08967e-6785-495f-b08b-8ea9bf1d9282 None None] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:19:31.196521 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d08967e-6785-495f-b08b-8ea9bf1d9282 None None] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:19:31.201081 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f510cce0-a011-461d-921d-a30a499ec939 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:19:31.202478 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f510cce0-a011-461d-921d-a30a499ec939 admin admin] REQUEST_METHOD: `POST` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:19:31.202767 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f510cce0-a011-461d-921d-a30a499ec939 admin admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:19:31.203040 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f510cce0-a011-461d-921d-a30a499ec939 admin admin] PATH_INFO: `/v3/projects` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:19:31.203721 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f510cce0-a011-461d-921d-a30a499ec939 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:19:31.205635 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f510cce0-a011-461d-921d-a30a499ec939 admin admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:19:31.330462 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 1441/2895] 10.222.0.30 () {60 vars in 1152 bytes} [Mon May 23 13:19:31 2022] POST /identity/v3/projects => generated 373 bytes in 143 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) May 23 13:19:31.334493 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7c771c5-d244-4503-a1f3-5004fd79b5d4 admin admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:19:31.344589 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7c771c5-d244-4503-a1f3-5004fd79b5d4 admin admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:19:31.403563 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03a2288e-5ed2-4aba-a748-f7b97fd0ca6c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:19:31.404232 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03a2288e-5ed2-4aba-a748-f7b97fd0ca6c admin admin] REQUEST_METHOD: `POST` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:19:31.404521 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03a2288e-5ed2-4aba-a748-f7b97fd0ca6c admin admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:19:31.404803 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03a2288e-5ed2-4aba-a748-f7b97fd0ca6c admin admin] PATH_INFO: `/v3/users` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:19:31.405483 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03a2288e-5ed2-4aba-a748-f7b97fd0ca6c admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:19:31.407439 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03a2288e-5ed2-4aba-a748-f7b97fd0ca6c admin admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:19:31.482383 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 1455/2896] 10.222.0.30 () {60 vars in 1143 bytes} [Mon May 23 13:19:31 2022] POST /identity/v3/users => generated 287 bytes in 148 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) May 23 13:19:31.486999 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f510cce0-a011-461d-921d-a30a499ec939 admin admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:19:31.497104 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f510cce0-a011-461d-921d-a30a499ec939 admin admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:19:31.502653 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0761d3cb-35cf-48f1-8687-6536a26b5b13 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:19:31.503336 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0761d3cb-35cf-48f1-8687-6536a26b5b13 admin admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:19:31.503621 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0761d3cb-35cf-48f1-8687-6536a26b5b13 admin admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:19:31.503897 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0761d3cb-35cf-48f1-8687-6536a26b5b13 admin admin] PATH_INFO: `/v3/roles` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:19:31.510125 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0761d3cb-35cf-48f1-8687-6536a26b5b13 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:19:31.510125 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0761d3cb-35cf-48f1-8687-6536a26b5b13 admin admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:19:31.517003 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 1442/2897] 10.222.0.30 () {58 vars in 1121 bytes} [Mon May 23 13:19:31 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) May 23 13:19:31.521181 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03a2288e-5ed2-4aba-a748-f7b97fd0ca6c admin admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:19:31.531423 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03a2288e-5ed2-4aba-a748-f7b97fd0ca6c admin admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:19:31.536140 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc38def3-9647-441f-adf6-bf9675f3d3f1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:19:31.536792 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc38def3-9647-441f-adf6-bf9675f3d3f1 admin admin] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:19:31.537076 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc38def3-9647-441f-adf6-bf9675f3d3f1 admin admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:19:31.537352 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc38def3-9647-441f-adf6-bf9675f3d3f1 admin admin] PATH_INFO: `/v3/roles` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:19:31.538152 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc38def3-9647-441f-adf6-bf9675f3d3f1 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:19:31.539734 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc38def3-9647-441f-adf6-bf9675f3d3f1 admin admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:19:31.550165 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 1456/2898] 10.222.0.30 () {58 vars in 1121 bytes} [Mon May 23 13:19:31 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) May 23 13:19:31.554146 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0761d3cb-35cf-48f1-8687-6536a26b5b13 admin admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:19:31.564395 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0761d3cb-35cf-48f1-8687-6536a26b5b13 admin admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:19:31.568990 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b51b64cb-d284-4690-9651-618e533dce81 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:19:31.569857 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b51b64cb-d284-4690-9651-618e533dce81 admin admin] REQUEST_METHOD: `PUT` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:19:31.570169 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b51b64cb-d284-4690-9651-618e533dce81 admin admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:19:31.570447 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b51b64cb-d284-4690-9651-618e533dce81 admin admin] PATH_INFO: `/v3/projects/c39739f8806b49638e73137c2403bae6/users/cf9dcdf4eb1643dc94c5d0e32f933205/roles/80fb22507c9a4ff09d44d8f12f6fff1a` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:19:31.584663 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b51b64cb-d284-4690-9651-618e533dce81 admin admin] RBAC: Authorizing `identity:create_grant(project_id=c39739f8806b49638e73137c2403bae6, user_id=cf9dcdf4eb1643dc94c5d0e32f933205, role_id=80fb22507c9a4ff09d44d8f12f6fff1a)` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:19:31.587289 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b51b64cb-d284-4690-9651-618e533dce81 admin admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:19:31.665764 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 1443/2899] 10.222.0.30 () {60 vars in 1482 bytes} [Mon May 23 13:19:31 2022] PUT /identity/v3/projects/c39739f8806b49638e73137c2403bae6/users/cf9dcdf4eb1643dc94c5d0e32f933205/roles/80fb22507c9a4ff09d44d8f12f6fff1a => generated 0 bytes in 112 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) May 23 13:19:31.673288 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f94d8a11-55c7-44b3-a58c-51043fa96dec None None] REQUEST_METHOD: `POST` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:19:31.675374 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f94d8a11-55c7-44b3-a58c-51043fa96dec None None] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:19:31.675374 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f94d8a11-55c7-44b3-a58c-51043fa96dec None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:19:31.697679 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.auth.core [None req-f94d8a11-55c7-44b3-a58c-51043fa96dec None None] MFA Rules not processed for user `0f7119f8fa5c4445977671140ded91a4`. Rule list: `[]` (Enabled: `True`). {{(pid=92057) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 13:19:31.699199 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.fernet_utils [None req-f94d8a11-55c7-44b3-a58c-51043fa96dec None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92057) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:19:31.756557 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 1457/2900] 10.222.0.30 () {58 vars in 957 bytes} [Mon May 23 13:19:31 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 84 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) May 23 13:19:31.766277 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b51b64cb-d284-4690-9651-618e533dce81 admin admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:19:31.768239 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.fernet_utils [None req-b51b64cb-d284-4690-9651-618e533dce81 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92056) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:19:31.789934 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 13:19:31.824792 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b51b64cb-d284-4690-9651-618e533dce81 admin admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:19:31.829206 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e20aa34-9e6a-4e5d-9875-1b621e227365 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2eb83ca38e79449095f3a2f947bbb06a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9c389ae65aaa4542bb5a5c4fd0bacf5b', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:19:31.829829 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e20aa34-9e6a-4e5d-9875-1b621e227365 service neutron] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:19:31.830146 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e20aa34-9e6a-4e5d-9875-1b621e227365 service neutron] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:19:31.830426 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e20aa34-9e6a-4e5d-9875-1b621e227365 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:19:31.839494 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e20aa34-9e6a-4e5d-9875-1b621e227365 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:19:31.840869 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e20aa34-9e6a-4e5d-9875-1b621e227365 service neutron] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:19:31.851057 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 1444/2901] 10.222.0.30 () {60 vars in 1365 bytes} [Mon May 23 13:19:31 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 85 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) May 23 13:19:36.853631 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cee03dd9-ff2c-47ea-bfe3-7c2ed1cd7080 None None] REQUEST_METHOD: `POST` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:19:36.854205 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cee03dd9-ff2c-47ea-bfe3-7c2ed1cd7080 None None] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:19:36.854317 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cee03dd9-ff2c-47ea-bfe3-7c2ed1cd7080 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:19:36.880598 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.auth.core [None req-cee03dd9-ff2c-47ea-bfe3-7c2ed1cd7080 None None] MFA Rules not processed for user `cf9dcdf4eb1643dc94c5d0e32f933205`. Rule list: `[]` (Enabled: `True`). {{(pid=92057) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 13:19:36.881782 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.fernet_utils [None req-cee03dd9-ff2c-47ea-bfe3-7c2ed1cd7080 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92057) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:19:36.907983 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 13:19:36.971212 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 1458/2902] 10.222.0.30 () {58 vars in 957 bytes} [Mon May 23 13:19:36 2022] POST /identity/v3/auth/tokens => generated 3240 bytes in 118 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) May 23 13:19:36.972152 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e20aa34-9e6a-4e5d-9875-1b621e227365 service neutron] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:19:36.987148 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e20aa34-9e6a-4e5d-9875-1b621e227365 service neutron] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:19:36.992212 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2ced84c-1525-4dff-a5ca-c39cb2095765 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:19:36.992816 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2ced84c-1525-4dff-a5ca-c39cb2095765 admin admin] REQUEST_METHOD: `POST` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:19:36.993121 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2ced84c-1525-4dff-a5ca-c39cb2095765 admin admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:19:36.993402 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2ced84c-1525-4dff-a5ca-c39cb2095765 admin admin] PATH_INFO: `/v3/projects` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:19:36.994144 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f2ced84c-1525-4dff-a5ca-c39cb2095765 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:19:36.996417 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f2ced84c-1525-4dff-a5ca-c39cb2095765 admin admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:19:37.085774 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 1445/2903] 10.222.0.30 () {60 vars in 1152 bytes} [Mon May 23 13:19:36 2022] POST /identity/v3/projects => generated 373 bytes in 114 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) May 23 13:19:37.089779 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cee03dd9-ff2c-47ea-bfe3-7c2ed1cd7080 None None] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:19:37.102404 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cee03dd9-ff2c-47ea-bfe3-7c2ed1cd7080 None None] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:19:37.168940 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-473328da-4dd3-42cb-9b28-9515984b56b0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:19:37.169722 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-473328da-4dd3-42cb-9b28-9515984b56b0 admin admin] REQUEST_METHOD: `POST` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:19:37.170102 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-473328da-4dd3-42cb-9b28-9515984b56b0 admin admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:19:37.170379 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-473328da-4dd3-42cb-9b28-9515984b56b0 admin admin] PATH_INFO: `/v3/users` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:19:37.171791 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-473328da-4dd3-42cb-9b28-9515984b56b0 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:19:37.173725 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-473328da-4dd3-42cb-9b28-9515984b56b0 admin admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:19:37.231859 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 1459/2904] 10.222.0.30 () {60 vars in 1143 bytes} [Mon May 23 13:19:37 2022] POST /identity/v3/users => generated 293 bytes in 143 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) May 23 13:19:37.236544 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2ced84c-1525-4dff-a5ca-c39cb2095765 admin admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:19:37.248124 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2ced84c-1525-4dff-a5ca-c39cb2095765 admin admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:19:37.254190 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dcba5af2-fe55-4d71-bbc6-0350a5f539ac admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:19:37.254867 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dcba5af2-fe55-4d71-bbc6-0350a5f539ac admin admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:19:37.255148 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dcba5af2-fe55-4d71-bbc6-0350a5f539ac admin admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:19:37.255424 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dcba5af2-fe55-4d71-bbc6-0350a5f539ac admin admin] PATH_INFO: `/v3/roles` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:19:37.256222 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dcba5af2-fe55-4d71-bbc6-0350a5f539ac admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:19:37.258001 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dcba5af2-fe55-4d71-bbc6-0350a5f539ac admin admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:19:37.270902 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 1446/2905] 10.222.0.30 () {58 vars in 1121 bytes} [Mon May 23 13:19:37 2022] GET /identity/v3/roles => generated 1420 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) May 23 13:19:37.280767 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-473328da-4dd3-42cb-9b28-9515984b56b0 admin admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:19:37.291600 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-473328da-4dd3-42cb-9b28-9515984b56b0 admin admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:19:37.297177 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64834535-83e1-4cf7-ac94-08124456c823 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:19:37.298122 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64834535-83e1-4cf7-ac94-08124456c823 admin admin] REQUEST_METHOD: `PUT` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:19:37.298407 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64834535-83e1-4cf7-ac94-08124456c823 admin admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:19:37.298780 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64834535-83e1-4cf7-ac94-08124456c823 admin admin] PATH_INFO: `/v3/projects/637c160cdcab4c07af8fa7a009691458/users/099d84617cd24f5b8b926be37d6a22a4/roles/bc3c3c9d654940cbaa5177a6be23d1bc` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:19:37.321269 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-64834535-83e1-4cf7-ac94-08124456c823 admin admin] RBAC: Authorizing `identity:create_grant(project_id=637c160cdcab4c07af8fa7a009691458, user_id=099d84617cd24f5b8b926be37d6a22a4, role_id=bc3c3c9d654940cbaa5177a6be23d1bc)` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:19:37.328768 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-64834535-83e1-4cf7-ac94-08124456c823 admin admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:19:37.399162 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 1460/2906] 10.222.0.30 () {60 vars in 1482 bytes} [Mon May 23 13:19:37 2022] PUT /identity/v3/projects/637c160cdcab4c07af8fa7a009691458/users/099d84617cd24f5b8b926be37d6a22a4/roles/bc3c3c9d654940cbaa5177a6be23d1bc => generated 0 bytes in 119 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) May 23 13:19:37.403071 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dcba5af2-fe55-4d71-bbc6-0350a5f539ac admin admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:19:37.413483 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dcba5af2-fe55-4d71-bbc6-0350a5f539ac admin admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:19:37.475974 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f1d6579-021a-471f-8dc2-99ba07c49ab0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:19:37.476649 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f1d6579-021a-471f-8dc2-99ba07c49ab0 admin admin] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:19:37.476930 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f1d6579-021a-471f-8dc2-99ba07c49ab0 admin admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:19:37.477202 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f1d6579-021a-471f-8dc2-99ba07c49ab0 admin admin] PATH_INFO: `/v3/roles` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:19:37.477964 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f1d6579-021a-471f-8dc2-99ba07c49ab0 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:19:37.479585 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f1d6579-021a-471f-8dc2-99ba07c49ab0 admin admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:19:37.490924 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 1447/2907] 10.222.0.30 () {58 vars in 1121 bytes} [Mon May 23 13:19:37 2022] GET /identity/v3/roles => generated 1420 bytes in 88 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) May 23 13:19:37.495058 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64834535-83e1-4cf7-ac94-08124456c823 admin admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:19:37.505946 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64834535-83e1-4cf7-ac94-08124456c823 admin admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:19:37.510707 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8598d82-03f9-4a67-8b44-8244d9781184 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:19:37.511742 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8598d82-03f9-4a67-8b44-8244d9781184 admin admin] REQUEST_METHOD: `PUT` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:19:37.512018 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8598d82-03f9-4a67-8b44-8244d9781184 admin admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:19:37.512289 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8598d82-03f9-4a67-8b44-8244d9781184 admin admin] PATH_INFO: `/v3/projects/637c160cdcab4c07af8fa7a009691458/users/099d84617cd24f5b8b926be37d6a22a4/roles/bc3c3c9d654940cbaa5177a6be23d1bc` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:19:37.514343 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8598d82-03f9-4a67-8b44-8244d9781184 admin admin] RBAC: Authorizing `identity:create_grant(project_id=637c160cdcab4c07af8fa7a009691458, user_id=099d84617cd24f5b8b926be37d6a22a4, role_id=bc3c3c9d654940cbaa5177a6be23d1bc)` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:19:37.517839 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8598d82-03f9-4a67-8b44-8244d9781184 admin admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:19:37.528852 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 1461/2908] 10.222.0.30 () {60 vars in 1482 bytes} [Mon May 23 13:19:37 2022] PUT /identity/v3/projects/637c160cdcab4c07af8fa7a009691458/users/099d84617cd24f5b8b926be37d6a22a4/roles/bc3c3c9d654940cbaa5177a6be23d1bc => generated 0 bytes in 34 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) May 23 13:19:42.978138 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4523cbbe-fae5-4853-a71d-8bd45330bdf9 None None] REQUEST_METHOD: `POST` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:19:42.978665 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4523cbbe-fae5-4853-a71d-8bd45330bdf9 None None] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:19:42.978787 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4523cbbe-fae5-4853-a71d-8bd45330bdf9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:19:43.007270 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.auth.core [None req-4523cbbe-fae5-4853-a71d-8bd45330bdf9 None None] MFA Rules not processed for user `099d84617cd24f5b8b926be37d6a22a4`. Rule list: `[]` (Enabled: `True`). {{(pid=92056) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 13:19:43.008492 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.fernet_utils [None req-4523cbbe-fae5-4853-a71d-8bd45330bdf9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92056) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:19:43.082901 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 1448/2909] 10.222.0.30 () {58 vars in 957 bytes} [Mon May 23 13:19:42 2022] POST /identity/v3/auth/tokens => generated 3307 bytes in 106 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) May 23 13:19:43.099108 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8598d82-03f9-4a67-8b44-8244d9781184 admin admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:19:43.101021 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.fernet_utils [None req-b8598d82-03f9-4a67-8b44-8244d9781184 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92057) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:19:43.180343 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8598d82-03f9-4a67-8b44-8244d9781184 admin admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:19:43.186163 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1359a3d-4c09-404a-9a99-828b5bfa1998 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cbe9931d94d44cc7ace0e83a7e5015fc', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9c389ae65aaa4542bb5a5c4fd0bacf5b', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'member', 'reader', 'service', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:19:43.186815 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1359a3d-4c09-404a-9a99-828b5bfa1998 service nova] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:19:43.187135 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1359a3d-4c09-404a-9a99-828b5bfa1998 service nova] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:19:43.187427 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1359a3d-4c09-404a-9a99-828b5bfa1998 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:19:43.192896 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 13:19:43.254052 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1359a3d-4c09-404a-9a99-828b5bfa1998 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:19:43.254504 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1359a3d-4c09-404a-9a99-828b5bfa1998 service nova] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:19:43.264387 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 1462/2910] 10.222.0.30 () {60 vars in 1365 bytes} [Mon May 23 13:19:43 2022] GET /identity/v3/auth/tokens => generated 3240 bytes in 166 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) May 23 13:19:46.554317 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d187402-8298-408d-a31b-fee2c2ebf3b4 None None] REQUEST_METHOD: `POST` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:19:46.554790 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d187402-8298-408d-a31b-fee2c2ebf3b4 None None] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:19:46.554790 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d187402-8298-408d-a31b-fee2c2ebf3b4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:19:46.580882 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.auth.core [None req-5d187402-8298-408d-a31b-fee2c2ebf3b4 None None] MFA Rules not processed for user `cf9dcdf4eb1643dc94c5d0e32f933205`. Rule list: `[]` (Enabled: `True`). {{(pid=92056) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} May 23 13:19:46.585080 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.fernet_utils [None req-5d187402-8298-408d-a31b-fee2c2ebf3b4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92056) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:19:46.594631 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 1449/2911] 10.222.0.30 () {58 vars in 957 bytes} [Mon May 23 13:19:46 2022] POST /identity/v3/auth/tokens => generated 3240 bytes in 41 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) May 23 13:19:53.293703 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=24 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 13:19:56.569918 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=97,keystone:INSERT=1 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 13:20:07.613532 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1359a3d-4c09-404a-9a99-828b5bfa1998 service nova] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:20:07.615084 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.fernet_utils [None req-f1359a3d-4c09-404a-9a99-828b5bfa1998 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92057) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:20:07.671476 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1359a3d-4c09-404a-9a99-828b5bfa1998 service nova] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:20:07.675332 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e1f0e85-e27d-4809-a34a-ac4fda13f5f1 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2eb83ca38e79449095f3a2f947bbb06a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9c389ae65aaa4542bb5a5c4fd0bacf5b', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:20:07.675969 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e1f0e85-e27d-4809-a34a-ac4fda13f5f1 service neutron] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:20:07.676256 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e1f0e85-e27d-4809-a34a-ac4fda13f5f1 service neutron] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:20:07.676540 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e1f0e85-e27d-4809-a34a-ac4fda13f5f1 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:20:07.682587 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.fernet_utils [None req-8e1f0e85-e27d-4809-a34a-ac4fda13f5f1 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92057) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:20:07.688125 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e1f0e85-e27d-4809-a34a-ac4fda13f5f1 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:20:07.689576 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e1f0e85-e27d-4809-a34a-ac4fda13f5f1 service neutron] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:20:07.698665 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 1463/2912] 10.222.0.30 () {60 vars in 1365 bytes} [Mon May 23 13:20:07 2022] GET /identity/v3/auth/tokens => generated 3111 bytes in 86 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) May 23 13:20:17.694926 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=26 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 13:20:25.196523 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d187402-8298-408d-a31b-fee2c2ebf3b4 None None] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:20:25.206048 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d187402-8298-408d-a31b-fee2c2ebf3b4 None None] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:20:25.209624 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f17344e-201a-4502-813b-ee410d6bae38 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2eb83ca38e79449095f3a2f947bbb06a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9c389ae65aaa4542bb5a5c4fd0bacf5b', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:20:25.210297 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f17344e-201a-4502-813b-ee410d6bae38 service neutron] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:20:25.210580 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f17344e-201a-4502-813b-ee410d6bae38 service neutron] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:20:25.210855 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f17344e-201a-4502-813b-ee410d6bae38 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:20:25.219740 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f17344e-201a-4502-813b-ee410d6bae38 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:20:25.221184 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f17344e-201a-4502-813b-ee410d6bae38 service neutron] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:20:25.231049 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 1450/2913] 10.222.0.30 () {60 vars in 1365 bytes} [Mon May 23 13:20:25 2022] GET /identity/v3/auth/tokens => generated 3307 bytes in 35 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) May 23 13:20:27.350706 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e1f0e85-e27d-4809-a34a-ac4fda13f5f1 service neutron] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:20:27.365853 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e1f0e85-e27d-4809-a34a-ac4fda13f5f1 service neutron] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:20:27.371617 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf5e206b-2974-40fa-b0c7-4861c9d0720e service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cbe9931d94d44cc7ace0e83a7e5015fc', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9c389ae65aaa4542bb5a5c4fd0bacf5b', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'member', 'reader', 'service', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:20:27.372266 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf5e206b-2974-40fa-b0c7-4861c9d0720e service nova] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:20:27.372578 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf5e206b-2974-40fa-b0c7-4861c9d0720e service nova] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:20:27.372858 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf5e206b-2974-40fa-b0c7-4861c9d0720e service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:20:27.374676 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.fernet_utils [None req-cf5e206b-2974-40fa-b0c7-4861c9d0720e service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92057) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:20:27.387715 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf5e206b-2974-40fa-b0c7-4861c9d0720e service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:20:27.389305 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf5e206b-2974-40fa-b0c7-4861c9d0720e service nova] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:20:27.401513 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 1464/2914] 10.222.0.30 () {60 vars in 1365 bytes} [Mon May 23 13:20:27 2022] GET /identity/v3/auth/tokens => generated 3362 bytes in 52 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) May 23 13:20:35.226678 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=6 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 13:20:37.395727 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=6 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 13:21:06.519226 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f17344e-201a-4502-813b-ee410d6bae38 service neutron] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:21:06.528748 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f17344e-201a-4502-813b-ee410d6bae38 service neutron] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:21:06.532582 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8e62af9-37f9-42de-9b14-6161cd1a5538 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2eb83ca38e79449095f3a2f947bbb06a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9c389ae65aaa4542bb5a5c4fd0bacf5b', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:21:06.533218 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8e62af9-37f9-42de-9b14-6161cd1a5538 service neutron] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:21:06.533505 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8e62af9-37f9-42de-9b14-6161cd1a5538 service neutron] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:21:06.533785 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8e62af9-37f9-42de-9b14-6161cd1a5538 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:21:06.535606 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.fernet_utils [None req-b8e62af9-37f9-42de-9b14-6161cd1a5538 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92056) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:21:06.546004 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8e62af9-37f9-42de-9b14-6161cd1a5538 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:21:06.547464 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8e62af9-37f9-42de-9b14-6161cd1a5538 service neutron] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:21:06.555712 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 1451/2915] 10.222.0.30 () {60 vars in 1365 bytes} [Mon May 23 13:21:06 2022] GET /identity/v3/auth/tokens => generated 3111 bytes in 37 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) May 23 13:21:16.551973 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=6 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 13:22:41.605300 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf5e206b-2974-40fa-b0c7-4861c9d0720e service nova] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:22:41.644860 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf5e206b-2974-40fa-b0c7-4861c9d0720e service nova] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:22:41.648616 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cba1c7ad-27cd-4b8b-9d79-91061f55d37f service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2eb83ca38e79449095f3a2f947bbb06a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9c389ae65aaa4542bb5a5c4fd0bacf5b', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:22:41.649256 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cba1c7ad-27cd-4b8b-9d79-91061f55d37f service neutron] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:22:41.649536 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cba1c7ad-27cd-4b8b-9d79-91061f55d37f service neutron] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:22:41.649809 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cba1c7ad-27cd-4b8b-9d79-91061f55d37f service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:22:41.651811 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.fernet_utils [None req-cba1c7ad-27cd-4b8b-9d79-91061f55d37f service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92057) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:22:41.697727 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cba1c7ad-27cd-4b8b-9d79-91061f55d37f service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:22:41.699245 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cba1c7ad-27cd-4b8b-9d79-91061f55d37f service neutron] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:22:41.710705 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 1465/2916] 10.222.0.30 () {60 vars in 1365 bytes} [Mon May 23 13:22:41 2022] GET /identity/v3/auth/tokens => generated 3362 bytes in 106 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) May 23 13:22:42.883712 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8e62af9-37f9-42de-9b14-6161cd1a5538 service neutron] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:22:42.920561 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8e62af9-37f9-42de-9b14-6161cd1a5538 service neutron] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:22:42.924225 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73eb91f7-4b05-4cc6-b549-754e39170199 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2eb83ca38e79449095f3a2f947bbb06a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9c389ae65aaa4542bb5a5c4fd0bacf5b', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:22:42.924857 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73eb91f7-4b05-4cc6-b549-754e39170199 service neutron] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:22:42.925143 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73eb91f7-4b05-4cc6-b549-754e39170199 service neutron] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:22:42.925419 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73eb91f7-4b05-4cc6-b549-754e39170199 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:22:42.927221 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.fernet_utils [None req-73eb91f7-4b05-4cc6-b549-754e39170199 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92056) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:22:42.941055 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-73eb91f7-4b05-4cc6-b549-754e39170199 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:22:42.942518 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-73eb91f7-4b05-4cc6-b549-754e39170199 service neutron] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:22:42.953524 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 1452/2917] 10.222.0.30 () {60 vars in 1365 bytes} [Mon May 23 13:22:42 2022] GET /identity/v3/auth/tokens => generated 3362 bytes in 70 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) May 23 13:22:51.705530 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=31 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 13:22:52.948593 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=18 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 13:22:56.252531 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cba1c7ad-27cd-4b8b-9d79-91061f55d37f service neutron] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:22:56.261175 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cba1c7ad-27cd-4b8b-9d79-91061f55d37f service neutron] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:22:56.265944 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-287f8068-aa34-4ae5-8ecc-a929d9c8f819 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2eb83ca38e79449095f3a2f947bbb06a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9c389ae65aaa4542bb5a5c4fd0bacf5b', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:22:56.266584 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-287f8068-aa34-4ae5-8ecc-a929d9c8f819 service neutron] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:22:56.266869 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-287f8068-aa34-4ae5-8ecc-a929d9c8f819 service neutron] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:22:56.267143 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-287f8068-aa34-4ae5-8ecc-a929d9c8f819 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:22:56.268923 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.fernet_utils [None req-287f8068-aa34-4ae5-8ecc-a929d9c8f819 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92057) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:22:56.292492 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-287f8068-aa34-4ae5-8ecc-a929d9c8f819 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:22:56.294115 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-287f8068-aa34-4ae5-8ecc-a929d9c8f819 service neutron] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:22:56.303437 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 1466/2918] 10.222.0.30 () {60 vars in 1365 bytes} [Mon May 23 13:22:56 2022] GET /identity/v3/auth/tokens => generated 3111 bytes in 51 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) May 23 13:22:56.397793 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73eb91f7-4b05-4cc6-b549-754e39170199 service neutron] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:22:56.406860 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73eb91f7-4b05-4cc6-b549-754e39170199 service neutron] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:22:56.410857 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ecba6ad-7148-48d6-8ed5-875c9b392715 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2eb83ca38e79449095f3a2f947bbb06a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9c389ae65aaa4542bb5a5c4fd0bacf5b', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:22:56.411512 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ecba6ad-7148-48d6-8ed5-875c9b392715 service neutron] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:22:56.411782 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ecba6ad-7148-48d6-8ed5-875c9b392715 service neutron] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:22:56.412084 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ecba6ad-7148-48d6-8ed5-875c9b392715 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:22:56.414253 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.fernet_utils [None req-9ecba6ad-7148-48d6-8ed5-875c9b392715 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92056) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:22:56.421562 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-287f8068-aa34-4ae5-8ecc-a929d9c8f819 service neutron] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:22:56.424645 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.fernet_utils [None req-287f8068-aa34-4ae5-8ecc-a929d9c8f819 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92057) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:22:56.429098 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ecba6ad-7148-48d6-8ed5-875c9b392715 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:22:56.430584 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ecba6ad-7148-48d6-8ed5-875c9b392715 service neutron] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:22:56.442483 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 1453/2919] 10.222.0.30 () {60 vars in 1365 bytes} [Mon May 23 13:22:56 2022] GET /identity/v3/auth/tokens => generated 3362 bytes in 45 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) May 23 13:22:56.447064 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-287f8068-aa34-4ae5-8ecc-a929d9c8f819 service neutron] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:22:56.449647 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1bafdc3-5f55-4f40-bad0-f304f6bba693 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cbe9931d94d44cc7ace0e83a7e5015fc', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9c389ae65aaa4542bb5a5c4fd0bacf5b', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'member', 'reader', 'service', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:22:56.450401 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1bafdc3-5f55-4f40-bad0-f304f6bba693 service nova] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:22:56.450759 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1bafdc3-5f55-4f40-bad0-f304f6bba693 service nova] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:22:56.451129 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1bafdc3-5f55-4f40-bad0-f304f6bba693 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:22:56.452909 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.fernet_utils [None req-b1bafdc3-5f55-4f40-bad0-f304f6bba693 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92057) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:22:56.465533 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1bafdc3-5f55-4f40-bad0-f304f6bba693 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:22:56.467169 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1bafdc3-5f55-4f40-bad0-f304f6bba693 service nova] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:22:56.478333 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 1467/2920] 10.222.0.30 () {60 vars in 1365 bytes} [Mon May 23 13:22:56 2022] GET /identity/v3/auth/tokens => generated 3362 bytes in 57 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) May 23 13:22:56.507196 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ecba6ad-7148-48d6-8ed5-875c9b392715 service neutron] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:22:56.508919 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.fernet_utils [None req-9ecba6ad-7148-48d6-8ed5-875c9b392715 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92056) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:22:56.580764 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ecba6ad-7148-48d6-8ed5-875c9b392715 service neutron] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:22:56.585905 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9543c0d-8257-497c-90b4-d527bfb7ab3b service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd5fb51f8d0fa46439cb29666970b885b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9c389ae65aaa4542bb5a5c4fd0bacf5b', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'service', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:22:56.586583 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9543c0d-8257-497c-90b4-d527bfb7ab3b service placement] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:22:56.586873 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9543c0d-8257-497c-90b4-d527bfb7ab3b service placement] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:22:56.587157 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9543c0d-8257-497c-90b4-d527bfb7ab3b service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:22:56.588934 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.fernet_utils [None req-d9543c0d-8257-497c-90b4-d527bfb7ab3b service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92056) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:22:56.600883 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9543c0d-8257-497c-90b4-d527bfb7ab3b service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:22:56.602503 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9543c0d-8257-497c-90b4-d527bfb7ab3b service placement] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:22:56.612995 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 1454/2921] 10.222.0.30 () {60 vars in 1365 bytes} [Mon May 23 13:22:56 2022] GET /identity/v3/auth/tokens => generated 3298 bytes in 106 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) May 23 13:23:06.473429 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=17 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 13:23:06.608784 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=38 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 13:24:10.794591 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1bafdc3-5f55-4f40-bad0-f304f6bba693 service nova] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:24:10.797011 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.fernet_utils [None req-b1bafdc3-5f55-4f40-bad0-f304f6bba693 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92057) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:24:10.833526 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1bafdc3-5f55-4f40-bad0-f304f6bba693 service nova] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:24:10.839320 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-770f9b8c-5738-4527-be45-1152be292cc3 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd5fb51f8d0fa46439cb29666970b885b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9c389ae65aaa4542bb5a5c4fd0bacf5b', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'service', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:24:10.839976 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-770f9b8c-5738-4527-be45-1152be292cc3 service placement] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:24:10.840265 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-770f9b8c-5738-4527-be45-1152be292cc3 service placement] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:24:10.840544 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-770f9b8c-5738-4527-be45-1152be292cc3 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:24:10.843532 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.fernet_utils [None req-770f9b8c-5738-4527-be45-1152be292cc3 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92057) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:24:10.855526 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-770f9b8c-5738-4527-be45-1152be292cc3 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:24:10.857114 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-770f9b8c-5738-4527-be45-1152be292cc3 service placement] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:24:10.867905 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 1468/2922] 10.222.0.30 () {60 vars in 1365 bytes} [Mon May 23 13:24:10 2022] GET /identity/v3/auth/tokens => generated 3298 bytes in 74 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) May 23 13:24:20.863663 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=15 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 13:24:42.865852 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9543c0d-8257-497c-90b4-d527bfb7ab3b service placement] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:24:42.880487 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9543c0d-8257-497c-90b4-d527bfb7ab3b service placement] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:24:42.888354 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c600b3b9-4d4b-4d9b-b185-947349cfeb6a service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cbe9931d94d44cc7ace0e83a7e5015fc', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9c389ae65aaa4542bb5a5c4fd0bacf5b', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'member', 'reader', 'service', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:24:42.889305 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c600b3b9-4d4b-4d9b-b185-947349cfeb6a service nova] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:24:42.889443 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c600b3b9-4d4b-4d9b-b185-947349cfeb6a service nova] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:24:42.889702 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c600b3b9-4d4b-4d9b-b185-947349cfeb6a service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:24:42.899437 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c600b3b9-4d4b-4d9b-b185-947349cfeb6a service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:24:42.901020 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c600b3b9-4d4b-4d9b-b185-947349cfeb6a service nova] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:24:42.912018 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 1455/2923] 10.222.0.30 () {60 vars in 1365 bytes} [Mon May 23 13:24:42 2022] GET /identity/v3/auth/tokens => generated 3240 bytes in 47 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) May 23 13:24:45.554629 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-770f9b8c-5738-4527-be45-1152be292cc3 service placement] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:24:45.567429 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-770f9b8c-5738-4527-be45-1152be292cc3 service placement] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:24:45.572838 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c4ec95e-ed73-44dd-a8ac-145f79febe5d service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cbe9931d94d44cc7ace0e83a7e5015fc', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9c389ae65aaa4542bb5a5c4fd0bacf5b', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'member', 'reader', 'service', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:24:45.573476 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c4ec95e-ed73-44dd-a8ac-145f79febe5d service nova] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:24:45.573761 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c4ec95e-ed73-44dd-a8ac-145f79febe5d service nova] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:24:45.574081 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c4ec95e-ed73-44dd-a8ac-145f79febe5d service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:24:45.576343 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.fernet_utils [None req-5c4ec95e-ed73-44dd-a8ac-145f79febe5d service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92057) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:24:45.589437 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c4ec95e-ed73-44dd-a8ac-145f79febe5d service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:24:45.591136 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c4ec95e-ed73-44dd-a8ac-145f79febe5d service nova] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:24:45.602374 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 1469/2924] 10.222.0.30 () {60 vars in 1365 bytes} [Mon May 23 13:24:45 2022] GET /identity/v3/auth/tokens => generated 3362 bytes in 48 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) May 23 13:24:49.216866 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c600b3b9-4d4b-4d9b-b185-947349cfeb6a service nova] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:24:49.227555 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c600b3b9-4d4b-4d9b-b185-947349cfeb6a service nova] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:24:49.231254 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-716a1a04-2e83-4734-9746-7fcfda839c1a service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2eb83ca38e79449095f3a2f947bbb06a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9c389ae65aaa4542bb5a5c4fd0bacf5b', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:24:49.231930 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-716a1a04-2e83-4734-9746-7fcfda839c1a service neutron] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:24:49.232212 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-716a1a04-2e83-4734-9746-7fcfda839c1a service neutron] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:24:49.232487 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-716a1a04-2e83-4734-9746-7fcfda839c1a service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:24:49.302190 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-716a1a04-2e83-4734-9746-7fcfda839c1a service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:24:49.303626 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-716a1a04-2e83-4734-9746-7fcfda839c1a service neutron] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:24:49.321796 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 1456/2925] 10.222.0.30 () {60 vars in 1365 bytes} [Mon May 23 13:24:49 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 105 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) May 23 13:24:52.464243 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c4ec95e-ed73-44dd-a8ac-145f79febe5d service nova] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:24:52.490696 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c4ec95e-ed73-44dd-a8ac-145f79febe5d service nova] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:24:52.497094 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d00370cf-c335-4961-9699-e63a6355c81e service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd5fb51f8d0fa46439cb29666970b885b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9c389ae65aaa4542bb5a5c4fd0bacf5b', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'service', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:24:52.497827 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d00370cf-c335-4961-9699-e63a6355c81e service placement] REQUEST_METHOD: `GET` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:24:52.502333 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d00370cf-c335-4961-9699-e63a6355c81e service placement] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:24:52.502333 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d00370cf-c335-4961-9699-e63a6355c81e service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:24:52.502333 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.fernet_utils [None req-d00370cf-c335-4961-9699-e63a6355c81e service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92057) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:24:52.514199 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d00370cf-c335-4961-9699-e63a6355c81e service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:24:52.515952 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d00370cf-c335-4961-9699-e63a6355c81e service placement] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:24:52.526327 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 1470/2926] 10.222.0.30 () {60 vars in 1365 bytes} [Mon May 23 13:24:52 2022] GET /identity/v3/auth/tokens => generated 3298 bytes in 63 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) May 23 13:24:56.173630 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-716a1a04-2e83-4734-9746-7fcfda839c1a service neutron] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:24:56.183441 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-716a1a04-2e83-4734-9746-7fcfda839c1a service neutron] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:24:56.188521 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35031f6f-6aa0-4695-aab9-c91f437aed53 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:24:56.189476 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35031f6f-6aa0-4695-aab9-c91f437aed53 admin admin] REQUEST_METHOD: `DELETE` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:24:56.189783 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35031f6f-6aa0-4695-aab9-c91f437aed53 admin admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:24:56.190211 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35031f6f-6aa0-4695-aab9-c91f437aed53 admin admin] PATH_INFO: `/v3/users/cf9dcdf4eb1643dc94c5d0e32f933205` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:24:56.191864 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35031f6f-6aa0-4695-aab9-c91f437aed53 admin admin] RBAC: Authorizing `identity:delete_user(user_id=cf9dcdf4eb1643dc94c5d0e32f933205)` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:24:56.193930 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35031f6f-6aa0-4695-aab9-c91f437aed53 admin admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:24:56.416030 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-35031f6f-6aa0-4695-aab9-c91f437aed53 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'cf9dcdf4eb1643dc94c5d0e32f933205'} {{(pid=92056) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:24:56.416689 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-35031f6f-6aa0-4695-aab9-c91f437aed53 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'cf9dcdf4eb1643dc94c5d0e32f933205'} {{(pid=92056) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:24:56.493539 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-35031f6f-6aa0-4695-aab9-c91f437aed53 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'cf9dcdf4eb1643dc94c5d0e32f933205'} {{(pid=92056) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:24:56.505803 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-35031f6f-6aa0-4695-aab9-c91f437aed53 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'cf9dcdf4eb1643dc94c5d0e32f933205'} {{(pid=92056) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:24:56.513000 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-35031f6f-6aa0-4695-aab9-c91f437aed53 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'cf9dcdf4eb1643dc94c5d0e32f933205'} {{(pid=92056) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:24:56.520024 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 1457/2927] 10.222.0.30 () {58 vars in 1223 bytes} [Mon May 23 13:24:56 2022] DELETE /identity/v3/users/cf9dcdf4eb1643dc94c5d0e32f933205 => generated 0 bytes in 347 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) May 23 13:24:56.752378 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d00370cf-c335-4961-9699-e63a6355c81e service placement] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:24:56.763395 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d00370cf-c335-4961-9699-e63a6355c81e service placement] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:24:56.820994 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f58b8453-c9b5-485b-a39e-8e87597c1415 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:24:56.821846 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f58b8453-c9b5-485b-a39e-8e87597c1415 admin admin] REQUEST_METHOD: `DELETE` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:24:56.822173 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f58b8453-c9b5-485b-a39e-8e87597c1415 admin admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:24:56.822433 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f58b8453-c9b5-485b-a39e-8e87597c1415 admin admin] PATH_INFO: `/v3/projects/c39739f8806b49638e73137c2403bae6` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:24:56.823591 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f58b8453-c9b5-485b-a39e-8e87597c1415 admin admin] RBAC: Authorizing `identity:delete_project(project_id=c39739f8806b49638e73137c2403bae6)` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:24:56.825472 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f58b8453-c9b5-485b-a39e-8e87597c1415 admin admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:24:56.939354 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [None req-f58b8453-c9b5-485b-a39e-8e87597c1415 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'c39739f8806b49638e73137c2403bae6', 'request_id': 'req-f58b8453-c9b5-485b-a39e-8e87597c1415'} {{(pid=92057) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:24:56.944877 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [None req-f58b8453-c9b5-485b-a39e-8e87597c1415 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'c39739f8806b49638e73137c2403bae6', 'request_id': 'req-f58b8453-c9b5-485b-a39e-8e87597c1415'} {{(pid=92057) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:24:57.012571 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [None req-f58b8453-c9b5-485b-a39e-8e87597c1415 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'c39739f8806b49638e73137c2403bae6', 'request_id': 'req-f58b8453-c9b5-485b-a39e-8e87597c1415'} {{(pid=92057) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:24:57.036776 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [None req-f58b8453-c9b5-485b-a39e-8e87597c1415 admin admin] The token cache is being invalidate because project c39739f8806b49638e73137c2403bae6 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=92057) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} May 23 13:24:57.039067 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [None req-f58b8453-c9b5-485b-a39e-8e87597c1415 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=92057) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:24:57.039067 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [None req-f58b8453-c9b5-485b-a39e-8e87597c1415 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=92057) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:24:57.039563 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 1471/2928] 10.222.0.30 () {58 vars in 1232 bytes} [Mon May 23 13:24:56 2022] DELETE /identity/v3/projects/c39739f8806b49638e73137c2403bae6 => generated 0 bytes in 288 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) May 23 13:24:57.043345 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35031f6f-6aa0-4695-aab9-c91f437aed53 admin admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:24:57.044732 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.fernet_utils [None req-35031f6f-6aa0-4695-aab9-c91f437aed53 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92056) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:24:57.059049 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35031f6f-6aa0-4695-aab9-c91f437aed53 admin admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:24:57.093433 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 13:24:57.117868 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55df2001-5bf1-494b-8121-27ecddf38d19 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:24:57.118762 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55df2001-5bf1-494b-8121-27ecddf38d19 admin admin] REQUEST_METHOD: `DELETE` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:24:57.119046 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55df2001-5bf1-494b-8121-27ecddf38d19 admin admin] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:24:57.119320 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55df2001-5bf1-494b-8121-27ecddf38d19 admin admin] PATH_INFO: `/v3/users/099d84617cd24f5b8b926be37d6a22a4` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:24:57.120496 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55df2001-5bf1-494b-8121-27ecddf38d19 admin admin] RBAC: Authorizing `identity:delete_user(user_id=099d84617cd24f5b8b926be37d6a22a4)` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:24:57.122440 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55df2001-5bf1-494b-8121-27ecddf38d19 admin admin] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:24:57.285007 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-55df2001-5bf1-494b-8121-27ecddf38d19 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '099d84617cd24f5b8b926be37d6a22a4'} {{(pid=92056) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:24:57.285636 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-55df2001-5bf1-494b-8121-27ecddf38d19 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '099d84617cd24f5b8b926be37d6a22a4'} {{(pid=92056) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:24:57.356176 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-55df2001-5bf1-494b-8121-27ecddf38d19 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '099d84617cd24f5b8b926be37d6a22a4'} {{(pid=92056) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:24:57.369123 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-55df2001-5bf1-494b-8121-27ecddf38d19 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '099d84617cd24f5b8b926be37d6a22a4'} {{(pid=92056) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:24:57.376008 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.notifications [None req-55df2001-5bf1-494b-8121-27ecddf38d19 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '099d84617cd24f5b8b926be37d6a22a4'} {{(pid=92056) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:24:57.381827 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 1458/2929] 10.222.0.30 () {58 vars in 1223 bytes} [Mon May 23 13:24:57 2022] DELETE /identity/v3/users/099d84617cd24f5b8b926be37d6a22a4 => generated 0 bytes in 339 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) May 23 13:24:57.615367 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f58b8453-c9b5-485b-a39e-8e87597c1415 admin admin] Authenticating user token {{(pid=92057) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:24:57.616769 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.fernet_utils [None req-f58b8453-c9b5-485b-a39e-8e87597c1415 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92057) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:24:57.632973 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f58b8453-c9b5-485b-a39e-8e87597c1415 admin admin] Validating token access rules against request {{(pid=92057) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:24:57.678546 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 13:24:57.687245 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47b79344-1262-4eaf-ba46-e2f8ae2b7f57 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0f7119f8fa5c4445977671140ded91a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'af8b84d1e3e44efba851eab6bed63282', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92057) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:24:57.693532 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47b79344-1262-4eaf-ba46-e2f8ae2b7f57 admin admin] REQUEST_METHOD: `DELETE` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:24:57.693532 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47b79344-1262-4eaf-ba46-e2f8ae2b7f57 admin admin] SCRIPT_NAME: `/identity` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:24:57.693532 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47b79344-1262-4eaf-ba46-e2f8ae2b7f57 admin admin] PATH_INFO: `/v3/projects/637c160cdcab4c07af8fa7a009691458` {{(pid=92057) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:24:57.693532 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47b79344-1262-4eaf-ba46-e2f8ae2b7f57 admin admin] RBAC: Authorizing `identity:delete_project(project_id=637c160cdcab4c07af8fa7a009691458)` {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:24:57.693532 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47b79344-1262-4eaf-ba46-e2f8ae2b7f57 admin admin] RBAC: Authorization granted {{(pid=92057) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:24:57.818769 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [None req-47b79344-1262-4eaf-ba46-e2f8ae2b7f57 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '637c160cdcab4c07af8fa7a009691458', 'request_id': 'req-47b79344-1262-4eaf-ba46-e2f8ae2b7f57'} {{(pid=92057) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:24:57.824912 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [None req-47b79344-1262-4eaf-ba46-e2f8ae2b7f57 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '637c160cdcab4c07af8fa7a009691458', 'request_id': 'req-47b79344-1262-4eaf-ba46-e2f8ae2b7f57'} {{(pid=92057) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:24:57.910592 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [None req-47b79344-1262-4eaf-ba46-e2f8ae2b7f57 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '637c160cdcab4c07af8fa7a009691458', 'request_id': 'req-47b79344-1262-4eaf-ba46-e2f8ae2b7f57'} {{(pid=92057) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:24:57.923017 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [None req-47b79344-1262-4eaf-ba46-e2f8ae2b7f57 admin admin] The token cache is being invalidate because project 637c160cdcab4c07af8fa7a009691458 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=92057) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} May 23 13:24:57.923423 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [None req-47b79344-1262-4eaf-ba46-e2f8ae2b7f57 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=92057) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:24:57.924051 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG keystone.notifications [None req-47b79344-1262-4eaf-ba46-e2f8ae2b7f57 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=92057) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} May 23 13:24:57.925611 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: [pid: 92057|app: 0|req: 1472/2930] 10.222.0.30 () {58 vars in 1232 bytes} [Mon May 23 13:24:57 2022] DELETE /identity/v3/projects/637c160cdcab4c07af8fa7a009691458 => generated 0 bytes in 311 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) May 23 13:25:07.378440 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=37,keystone:DELETE=10,keystone:INSERT=1 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 13:25:07.919601 dv-cinder-master-3-iscsi devstack@keystone.service[92057]: DEBUG dbcounter [-] [92057] Writing DB stats keystone:SELECT=27,keystone:DELETE=7,keystone:INSERT=1 {{(pid=92057) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 23 13:29:13.731180 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55df2001-5bf1-494b-8121-27ecddf38d19 admin admin] Authenticating user token {{(pid=92056) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} May 23 13:29:13.732565 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.fernet_utils [None req-55df2001-5bf1-494b-8121-27ecddf38d19 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92056) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:29:13.801974 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55df2001-5bf1-494b-8121-27ecddf38d19 admin admin] Validating token access rules against request {{(pid=92056) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} May 23 13:29:13.860531 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e2d37ca-0c94-4847-a6c9-a159428fbc07 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd5fb51f8d0fa46439cb29666970b885b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9c389ae65aaa4542bb5a5c4fd0bacf5b', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'service', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92056) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} May 23 13:29:13.861174 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e2d37ca-0c94-4847-a6c9-a159428fbc07 service placement] REQUEST_METHOD: `GET` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} May 23 13:29:13.861453 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e2d37ca-0c94-4847-a6c9-a159428fbc07 service placement] SCRIPT_NAME: `/identity` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} May 23 13:29:13.861727 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e2d37ca-0c94-4847-a6c9-a159428fbc07 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=92056) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} May 23 13:29:13.863523 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.fernet_utils [None req-9e2d37ca-0c94-4847-a6c9-a159428fbc07 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92056) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} May 23 13:29:13.875034 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e2d37ca-0c94-4847-a6c9-a159428fbc07 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} May 23 13:29:13.876591 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e2d37ca-0c94-4847-a6c9-a159428fbc07 service placement] RBAC: Authorization granted {{(pid=92056) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} May 23 13:29:13.886469 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: [pid: 92056|app: 0|req: 1459/2931] 10.222.0.30 () {60 vars in 1365 bytes} [Mon May 23 13:29:13 2022] GET /identity/v3/auth/tokens => generated 3298 bytes in 156 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) May 23 13:29:23.882299 dv-cinder-master-3-iscsi devstack@keystone.service[92056]: DEBUG dbcounter [-] [92056] Writing DB stats keystone:SELECT=58 {{(pid=92056) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}}